Found a very interesting article on how XKeyscore actually works. Interesting to me that despite assurances we are not being spied on ( which is a lie) that Obama announced a comprehensive review of the NSA last night. Snowden has forced his hand. It is time to consider Snowden as a hero instead of a “hacker” or spy . The fact is it a duty to point out the wrongs of the Govt and not follow along like fish. Indeed if the Govt is acting in a illegal way it is a moral imperative to expose it. No Nazi who said ” we were just following orders ” had any success with that argument. By doing nothing you are as guilty as the people who ordered it.
Definition of Panopticon
A circular prison with cells arranged around a central well, from which prisoners could at all times be observed.
The National Security Agency’s (NSA) apparatus for spying on what passes over the Internet, phone lines, and airways has long been the stuff of legend, with the public catching only brief glimpses into its Leviathan nature. Thanks to the documents leaked by former NSA contractor Edward Snowden, we now have a much bigger picture.
When that picture is combined with federal contract data and other pieces of the public record—as well as information from other whistleblowers and investigators—it’s possible to deduce a great deal about what the NSA has built and what it can do.
We’ve already looked at the NSA’s basic capabilities of collecting, managing, and processing “big data.” But the recently released XKeyscore documents provide a much more complete picture of how the NSA feeds its big data monsters and how it gets “situational awareness” of what’s happening on the Internet. What follows is an analysis of how XKeyscore works and how the NSA’s network surveillance capabilities have evolved over the past decade.
After the attacks of September 11, 2001 and the subsequent passage of the USA PATRIOT Act, the NSA and other organizations within the federal intelligence, defense, and law enforcement communities rushed to up their game in Internet surveillance. The NSA had already developed a “signals intelligence” operation that spanned the globe. But it had not had a mandate for sweeping surveillance operations—let alone permission for it—since the Foreign Intelligence Surveillance Act (FISA) was passed in 1978. (Imagine what Richard Nixon could have done with Facebook monitoring.)
The Global War On Terror, or GWOT as it was known around DC’s beltway, opened up the purse strings for everything on the intelligence, surveillance, and reconnaissance (ISR) shopping list. The NSA’s budget is hidden within the larger National Intelligence Program (NIP) budget. But some estimates suggest that the NSA’s piece of that pie is between 17 and 20 percent—putting its cumulative budget from fiscal year 2006 through 2012, conservatively, at about $58 billion.
Early on, the NSA needed a quick fix. It got that by buying largely off-the-shelf systems for network monitoring, as evidenced by the installation of hardware from Boeing subsidiary Narus at network tap sites such as AT&T’s Folsom Street facility in San Francisco. In 2003, the NSA worked with AT&T to install a collection of networking and computing gear—including Narus’ Semantic Traffic Analyzer (STA) 6400—to monitor the peering links for AT&T’s WorldNet Internet service. Narus’ STA software, which evolved into the Intelligent Traffic Analyzer line, was also used by the FBI as a replacement for its Carnivore system during that time frame.
Catching packets like tuna (not dolphin-safe)
Narus’ system is broken into two parts. The first is a computing device in-line with the network that watches the metadata in the packets passing by for ones that match “key pairs,” which can be a specific IP address or a range of IP addresses, a keyword within a Web browser request, or a pattern identifying a certain type of traffic such as a VPN or Tor connection.
Packets that match those rules are thrown to the second part of Narus’ system—a collection of analytic processing systems—over a separate high-speed network backbone by way of messaging middleware similar to the transaction systems used in financial systems and commodity trading floors.
In the current generation of Narus’ system, the processing systems run on commodity Linux servers and re-assemble network sessions as they’re captured, mining them for metadata, file attachments, and other application data and then indexing and dumping that information to a searchable database.
There are a couple of trade-offs with Narus’ approach. For one thing, the number of rules loaded on the network-sensing machine directly impact how much traffic it can handle—the more rules, the more compute power burned and memory consumed per packet, and the fewer packets that can be handled simultaneously. When I interviewed Narus’ director of product management for cyber analytics Neil Harrington last year, he said that “with everything turned on” on a two-way, 10-gigabit Ethernet connection—that is, with all of the pre-configured filters turned on—”out of the possible 20 gigabits, we see about 12. If we turn off tag pairs that we’re not interested in, we can make it more efficient.”
This PDF contains slides that explain the system to some degree.
The Obama News.
Analysis — NSA leaker Edward Snowden’s revelations have forced President Barack Obama’s hand, leading the president to announce new reforms of the government’s classified surveillance programs.
After his administration issued repeated defenses of a National Security Agency monitoring program that collects Americans’ phone and Internet data, Obama announced during a press conference Friday afternoon that reforms to the system will make the collection activities more transparent and “give the American people additional confidence that there are additional safeguards against abuse.”
Obama said the changes will include changes to the Foreign Intelligence Surveillance Act (FISA) court system — which currently greenlights requests for data gathering — as well as the creation of both an internal NSA position devoted to privacy and an external working group to evaluate transparency in the program. Officials will also launch a new website next week that will serve as “a hub for further transparency” for interested members of the public.
“Given the history of abuse by governments, it’s right to ask questions about surveillance by governments, particularly as technology is reshaping every aspects of our lives,” he said.
Obama’s announcement comes even as Snowden — the defense-contractor-turned-fugitive who released information to reporters about the NSA’s monitoring programs — has been charged with theft of government property and two offenses under U.S. espionage law.
He continues to evade extradition to the United States under a temporary asylum granted by the Russian government – an agreement that prompted Obama to cancel a planned meeting with Russian President Vladimir Putin in protest.
Snowden has generated strong feelings in the wake of disclosures, with many lawmakers decrying him as treasonous for releasing the information while others have used the case to press their concerns about how the government is watching American citizens.
Obama bluntly rejected the idea Friday that Snowden’s actions were patriotic.
“No, I don’t think Mr. Snowden is a patriot,” he said, adding that he would have preferred a “lawful, orderly” debate over privacy concerns rather than finger-pointing in the wake of the leaks.