!! History Commons Alert, Exciting News
Profile: Total Information Awareness (TIA)
Total Information Awareness (TIA) was a participant or observer in the following events:
An illustration of the NIMD dataflow. [Source: LibertyThink.com] (click image to enlarge)Shortly after the 9/11 attacks, the NSA awards $64 million in research contracts for a program called Novel Intelligence from Massive Data (NIMD). [New York Times, 5/21/2003; National Journal, 1/20/2006] NIMD is one of several cutting-edge data mining technologies that not only has the capability of finding keywords among millions of electronically monitored communications, but can find hidden relationships among data points, and even critique the thinking and biases of a particular analyst and suggest alternative hypotheses differing from the human analysts’ conclusion. Like other data-mining technologies, the NSA will steadfastly refuse to discuss whether NIMD is used to analyze data from domestic surveillance operations. NIMD is designed as an preliminary sort program, to keep human analysts from becoming overwhelmed by raw data. In essence, NIMD is an early-warning system. “NIMD funds research to…help analysts deal with information-overload, detect early indicators of strategic surprise, and avoid analytic errors,” according to the “Call for 2005 Challenge Workshop Proposals” released by the Advanced Research and Development Activity (ARDA). ARDA was founded in 1998 to create, design, and field new technologies for US intelligence agencies, particularly the NSA. A selected few Congressional lawmakers (see January 18, 2006) were informed that the warrantless surveillance program authorized by President George W. Bush (see Early 2002) was designed to be an early-warning system for possible terrorist attacks or plans. Assistant Attorney General William Moschella will inform the top Democrats and Republicans on the House and Senate Intelligence committees in December 2002 that the “president determined that it was necessary following September 11 to create an early-warning detection system” to prevent more attacks. He will justify the use of programs such as NIMD by claiming, as NSA director Michael Hayden and other administration officials have repeatedly claimed, that the Foreign Intelligence Surveillance Act (FISA), which allows the government to obtain warrants to conduct domestic eavesdropping or wiretapping, “could not have provided the speed and agility required for the early-warning detection system.” Many experts outside of the Bush administration feel that NIMD and other programs do not have to operate outside of the Foreign Intelligence Surveillance Act (FISA) because of limitations in the law, but because of the fact that the programs cannot meet the law’s minimum requirements for surveillance. FISA requires that any such surveillance must have a probable cause that the target is a terrorist. NIMD has no such threshold. Steven Aftergood, an expert on intelligence and government secrecy with the Federation of American Scientists, will say in 2006, “Logistically speaking, the early-warning approach may involve a significant increase in the number of surveillance actions. It may be that neither the Justice Department nor the [Foreign Intelligence Surveillance Court, which approves wiretapping warrants] is prepared to prepare and process several thousand additional FISA applications per year, beyond the 1,700 or so approved in 2004.” [National Journal, 1/20/2006] Some experts will later express the opinion that NIMD is the controversial Total Information Awareness program in a slightly different form (see February 2003 and September 2002).
Entity Tags: Senate Intelligence Committee, US Department of Justice, Total Information Awareness, William E. Moschella, Tom Armour, Novel Intelligence from Massive Data, Steven Aftergood, Michael Hayden, National Security Agency, Advanced Capabilities for Intelligence Analysis, Advanced Research and Development Activity, John Poindexter, Foreign Intelligence Surveillance Act, George W. Bush, Federation of American Scientists (FAS), House Intelligence Committee
Timeline Tags: Civil Liberties
Vice Admiral John Poindexter testifying before Congress in the Iran Contra hearings in 1987. [Source: Associated Press]Vice Admiral John Poindexter begins running a shadowy new government agency called the Information Awareness Office. [New York Times, 2/13/2002; Federal Computer Week, 10/17/2002] Poindexter, formerly President Reagan’s National Security Adviser, is known for his five felony convictions of lying to Congress, destroying documents, and obstructing Congress in its investigation of his role in the mid-1980s Iran-Contra affair. Later his convictions were overturned on a technicality. [Los Angeles Times, 11/17/2002] Far from apologizing, Poindexter said it was his duty to lie to Congress. [Newsday, 12/1/2002] The New York Times notes that his new agency “is developing technologies to give federal officials instant access to vast new surveillance and information-analysis systems.” The new office is part of DARPA, the Defense Advanced Research Projects Agency. Poindexter was also known for his controversial role in shifting control of computer security to the military in the 1980s. Says Marc Rotenberg, former counsel for the Senate Judiciary Committee, “It took three administrations and both political parties over a decade to correct those mistakes.” [New York Times, 2/13/2002] Surprisingly, Poindexter’s appointment is little noticed until later in 2002 when the Total Information Awareness program is revealed (see March 2002; November 9, 2002). Incidentally, several others involved in the Iran-Contra affair also find jobs in the Bush Administration, including Elliott Abrams, John Negroponte, and Otto Reich. [Observer, 12/8/2002]
This partial image from a Total Information Awareness slide presentation shows types of data that will be collected. Note that
even “gait” - the way one walks, will be analyzed.
[Source: DARPA]The US military internally announces the creation of a new global data collection system called Total Information Awareness. The existence of this program is not reported until August 2002 [Wired News, 8/7/2002] , and not widely known until November 2002 (see November 9, 2002). Interestingly, the early accounts of this program suggest its budget is a “significant amount” of $96 million [Federal Computer Week, 10/17/2002] , and not the $10 million later reported. [Guardian, 11/23/2002] It is also reported that “parts” of the program “are already operational” whereas later it is said to be only in the conceptual stages of development. [Federal Computer Week, 10/17/2002]
The NSA’s enormous data mining program, Novel Intelligence from Massive Data (NIMD—see After September 11, 2001), closely resembles another data mining program, the Defense Department’s Total Information Awareness program (see March 2002). TIA, which will be suspended in 2003 after outcries from citizens and legal experts concerned over that program’s refusal to comply with fundamental Constitutional guarantees of privacy, was also designed as an early-warning system that not only compiled intelligence data, but mined through private financial databases for credit-card and other financial transactions. Six of the corporations and research institutions who win NIMD contracts also held contracts for the earlier TIA project. Much of their work with TIA duplicated the same aspects and protocols to be used by NIMD, including challenging analytic assumptions and building prototype data-mining devices. Both TIA and NIMD attempt to second-guess human analysts’ conclusions over a particular data schema by creating a database of what TIA creator John Poindexter once called “plausible futures,” or likely terrorism scenarios. NIMD is a creation of the Advanced Research and Development Activity agency (ARDA); another ARDA project, the Advanced Capabilities for Intelligence Analysis (ACIA) also envisions a similar database (see 2005). Though TIA focused more on counterterrorism than the more sweeping NIMD, the two projects coordinated closely with one another, according to former program manager Tom Armour, who worked in Poindexter’s office.
NIMD Survives In Other Agencies - Congress will eliminate funding for TIA and other Poindexter projects, but many of those projects, and related projects such as NIMD, do not disappear. Many are instead transferred to intelligence agencies such as the NSA. Although information about these projects is strictly classified, a former Army intelligence analyst familiar with the programs will confirm in 2006 that large elements of TIA were transferred to other agencies, where they will continue to be researched and implemented. It is highly likely that NIMD is an offshoot and outgrowth of TIA. Armour will say the two programs are specifically designed to analyze large amounts of phone and e-mail traffic: “That’s, in fact, what the interest is.” [National Journal, 1/20/2006]
Government 'Shell Game' - Steven Aftergood of the Federation of American Scientists says in 2004, “The whole congressional action looks like a shell game. There may be enough of a difference for them to claim TIA was terminated while for all practical purposes the identical work is continuing.” [Associated Press, 2/23/2004] Aftergood will note that NIMD has thrived in the shadows where TIA died of exposure: “Pursued with a minimal public profile and lacking a polarizing figure like Admiral Poindexter to galvanize opposition, NIMD has proceeded quietly even as TIA imploded.” [Defense Tech, 9/26/2003]
The original logo for the Total Information Awareness program. An eye from a Masonic pyramid appears to cast a beam over the world, with Muslim regions highlighted. [San Francisco Chronicle, 11/20/02, Guardian, 11/23/02] The motto, Scientia Est Potentia, means “knowledge is power.” The logo is later removed from the department’s website. [Baltimore Sun, 1/5/03]
[Source: DARPA]The New York Times exposes the existence of John Poindexter’s Total Information Awareness data collection program, begun in early 2002 (see Mid-January 2002; March 2002). [New York Times, 11/9/2002] Conservative columnist William Safire writes, “If the Homeland Security Act is not amended before passage, here is what will happen to you: Every purchase you make with a credit card, every magazine subscription you buy and medical prescription you fill, every Web site you visit and e-mail you send or receive, every academic grade you receive, every bank deposit you make, every trip you book and every event you attend—all these transactions and communications will go into what the Defense Department describes as ‘a virtual, centralized grand database.’ To this computerized dossier on your private life from commercial sources, add every piece of information that government has about you—passport application, driver’s license and bridge toll records, judicial and divorce records, complaints from nosy neighbors to the FBI, your lifetime paper trail plus the latest hidden camera surveillance—and you have the supersnoop’s dream: a ‘Total Information Awareness’ about every US citizen.” [New York Times, 11/14/2002] Poindexter says it will take years to realize his vision, but his office has already begun providing some technology to government agencies. [Washington Post, 11/12/2002] The existence of this program, and the fact that Poindexter is running it, causes concern for many on both the left and right. [USA Today, 1/16/2003] It is regularly called Orwellian, conjuring visions of 1984’s Big Brother, and even supporters admit it sounds Orwellian. [Newsweek, 11/15/2002; Los Angeles Times, 11/17/2002; Guardian, 11/23/2002; Newsday, 12/1/2002; New Yorker, 12/9/2002; BBC, 12/12/2002; Dallas Morning News, 12/16/2002; Baltimore Sun, 1/5/2003] The New York Times suggests, “Congress should shut down the program pending a thorough investigation.” [New York Times, 11/18/2002] Experts question not only its civil liberties implications, but also if it is even feasible. If it does work, would its database be swapped with errors that could not be removed? (see March 2002) [Mercury News (San Jose), 12/26/2002] However, many newspapers fail to report on the program at all, and ABC is the only network to report the story on prime time television. [ABC News, 11/16/2002; ABC News, 11/25/2002] Despite so many objections, the program is included in the Homeland Security bill (see November 25, 2002), and only later somewhat curbed by Congress (see January 23, 2003).
Veteran AT&T technician Mark Klein (see July 7, 2009) becomes convinced that the “secure facility” being constructed at an AT&T facility in San Francisco (see Summer 2002 and Fall 2002) has some connection to the Bush administration’s “Total Information Awareness” program (see Mid-January 2002 and March 2002). The press has recently begun reporting on the program (see November 9, 2002). In 2007, Klein will tell a reporter: “You might recall there [around this time] was a big blowup in the news about the Total Information Awareness [TIA] program, led by Admiral [John] Poindexter, which caused the big upsetness in Congress, because what Poindexter was proposing to do was draw in databases from everywhere… draw in Internet data, bank records, travel records, everything into one big conglomeration which could be searchable by the government so they could find out everything about what anybody’s doing at any time of day. And all this would be done without any warrants.” Klein and other AT&T employees begin speculating that the “secure facility” might have some connection to Poindexter’s TIA program. “[T]he Total Information Awareness program is involved with the NSA [National Security Agency] and with DARPA, which is the Defense [Advanced Research] Projects Agency,” he will tell the reporter. “So I began to connect the two, because it seemed quite logical at least that if they are looking for large amounts of Internet data to sift through and vacuum up, what would be a perfect place? It would be in the Internet room at a place like AT&T. And lo and behold, the NSA guy shows up. Then I started learning that it’s not only a new room; it’s a room that all the technicians cannot go into. Only the one guy—a management guy, no union people—a management field specialist with security clearance obviously given to him by the NSA, only he could go into this room, which was being built on the sixth floor, right next door to the phone switch room. So I got very worried about that. What does this mean? What are they doing there?” In 2009, Klein will write, “Gradually I started to connect the TIA program with the curiously simultaneous appearance of the NSA at our office, and the more I learned about what they were installing, and where, the more I was convinced the two were connected.” [PBS Frontline, 5/15/2007; Klein, 2009, pp. 25-26]
Congress imposes some limitations on the Total Information Awareness program (see March 2002; November 9, 2002). Research and development of the program would have to halt within 90 days of enactment of the bill unless the Defense Department submits a detailed report about the program. The research can also continue if Bush certifies that the report cannot be provided. Congress also okays use of the program internationally, but it cannot be used inside the US unless Congress passes new legislation specifically authorizing such use. [New York Times, 1/24/2003; Los Angeles Times, 2/19/2003] However, a bill to completely stop the program has yet to pass. [Mercury News (San Jose), 1/17/2003; Los Angeles Times, 2/19/2003] Several days earlier, Senator Charles Grassley (R-IA) alleged that the Justice Department and FBI are more extensively exploring the use of the Total Information Awareness program than they have previously acknowledged. [Associated Press, 1/21/2003; Washington Post, 1/22/2003] Contracts worth tens of millions of dollars have been signed with private companies to develop pieces of the program. [Associated Press, 2/12/2003] Salon also reports that the program “has now advanced to the point where it’s much more than a mere ‘research project’.” [Salon, 1/29/2003]
Steven Aftergood of the Federation of American Scientists says that he is not sure that Congress’s public termination of the Total Information Awareness (TIA) project (see January 23, 2003) was as real and outrage-driven as it seemed at the time. “The whole congressional action looks like a shell game,” Aftergood says. “There may be enough of a difference for them to claim TIA was terminated while for all practical purposes the identical work is continuing.” While Congress terminated TIA with visible indignation, it also quietly funded the “National Foreign Intelligence Program,” and never identified which intelligence agency would do the work—which was also kept from the public eye. Congress did say that none of the research would be used against US citizens. No one in Congress will discuss how many of Poindexter’s programs survived, but knowledgeable sources will confirm that some 18 data-mining programs known as Evidence Extraction and Link Discovery in Poindexter’s research were preserved after TIA’s termination. These programs may well include the sprawling data mining program known as Novel Intelligence from Massive Data (NIMD) (see After September 11, 2001), though this cannot be confirmed. Former TIA chief John Poindexter’s vision of the technology behind NIMD envisioned software that can quickly analyze “multiple petabytes” of data. A single petabyte would fill the Library of Congress space for 18 million books more than 50 times, or could hold 40 pages of text for each of the more than 6.2 billion humans on Earth. Poindexter and his colleagues envisioned the program as handling a petabyte or more of data a month. [Associated Press, 2/23/2004] Concerns about the privacy rights of US citizens being damaged by the program are rife. “If they were to stick to strictly military-related research and development, there is less of an issue, but these technologies have much broader social implications,” says Barbara Simons, a computer scientist who is past president of the Association of Computing Machinery, an organization that has expressed concerns about TIA. [New York Times, 5/21/2003] At least one Senator is uncomfortable with the apparent resurgence of TIA. Jay Rockefeller (D-WV) will write Vice President Dick Cheney in June 2003 after receiving a briefing on the various secret surveillance programs (see July 17, 2003). Rockefeller will write, “As I reflected on the meeting today, John Poindexter’s TIA project sprung to mind, exacerbating my concern regarding the direction the administration is moving with regard to security, technology, and surveillance.” [National Journal, 1/20/2006]
Senator John D. Rockefeller. [Source: ViewImages.com]John D. Rockefeller (D-WV), the ranking member of the Senate Intelligence Committee, learns of the secret NSA warrantless wiretapping program against US citizens (see Early 2002) in a secret briefing for himself, the chairman of the committee, and the chairman and ranking member of the House Intelligence Committee. Hours later, Rockefeller sends a handwritten letter to Vice President Cheney expressing his concerns about the potential illegality of the program, concerns he apparently expressed in the briefing as well. Rockefeller will not release the letter publicly until December 19, 2005, four days after the New York Times publishes an article revealing the program’s existence (see December 15, 2005). Disturbed both by the information he was given and the information that was obviously being withheld, Rockefeller writes in part: “Clearly the activities we discussed raise profound oversight issues.… Given the security restrictions associated with this information, and my inability to consult staff or counsel on my own [Cheney had prohibited Rockefeller and the three other lawmakers in the briefing from consulting with their staff experts], I feel unable to fully evaluate, much less endorse these activities. As I reflected on the meeting today, and the future we face, John Poindexter’s TIA [Total Information Awareness (see March 2002)] project sprung to mind, exacerbating my concern regarding the direction the administration is moving with regard to security, technology, and surveillance. Without more information and the ability to draw on any independent legal or technical expertise, I simply cannot satisfy lingering concerns raised by the briefing we received.” [Democratic Party, 12/19/2005; Savage, 2007, pp. 115] Rockefeller also notes that he is not at liberty to do anything about his concerns, since he is legally bound to obey the secrecy rules the White House has invoked, but he wants his concerns noted. [Savage, 2007, pp. 116] It is unclear whether Rockefeller ever receives a reply. Rockefeller is apparently unaware of evidence showing that domestic surveillance may have begun well before 9/11 (see Late 1999, February 27, 2000, December 2000, February 2001, February 2001, Spring 2001, July 2001, and Early 2002).
DARPA logo. [Source: Duke University]The computer and technology experts at Ars Technica, a well-regarded Web publication which describes itself as focusing on “the art of technology,” speculate on the technology behind the NSA warrantless wiretapping program recently revealed to the public (see December 15, 2005). The Ars Technica experts believe that Senator Jay Rockefeller (D-WV)‘s 2003 comparison between the program and the Total Information Awareness (TIA) project (see March 2002) is the most apt. They believe that the NSA wiretapping program may be built upon the foundation of a shadowy, highly classified surveillance program called Echelon. They write, “This system’s purpose would be to monitor communications and detect would-be terrorists and plots before they happen… This project is not interested in funding ‘evolutionary’ changes in technology, e.g., bit-step improvements to current data mining and storage techniques. Rather, the amount of data that the directors are anticipating (petabytes!) would require massive leaps in technology (and perhaps also some massive leaps in surveillance laws).” [Ars Technica, 12/20/2005; Ars Technica, 2007] Data storage measured in petabytes is a colossal capacity; a petabyte is 1,024 terabytes, and a single terabyte is 1,024 gigabytes, the usual measurement for hard drive capacity. [TechTerms, 2007] The Ars Technica experts continue, “According to DARPA, such data collection ‘increases information coverage by an order of magnitude,’ and ultimately ‘requires keeping track of individuals and understanding how they fit into models.’” They go on to note that the NSA wiretapping program was instituted shortly after the TIA project was quashed by Congress, and say they believe the NSA program is an extension and an outgrowth of TIA. They note that “the FBI requested the legal authorization to do very high-volume monitoring of digital calls” in 1995, that there is “no way for the judicial system to approve warrants for the number of calls that the FBI wanted to monitor,” and that the FBI “could never hire enough humans to be able to monitor that many calls simultaneously, which means that they’d have to use voice recognition technology to look for ‘hits’ that they could then follow up on with human wiretaps.” The Ars Technica experts believe the NSA is using “some kind of high-volume, automated voice recognition and pattern matching system,” employing a form of “smart filtering” that would weed through perhaps hundreds of thousands of computer-monitored calls and turning a fraction of those calls over to human analysts for evaluation: “[Y]es, this kind of real-time voice recognition, crude semantic parsing and pattern matching is doable with today’s technology, especially when you have a budget like the NSA.” In a follow-up, Ars Technica technology specialist and self-described conservative and “privacy nazi” Jon Stokes writes of his own concerns over the program, noting that the program is too wide-reaching and too blunt to actually catch many real terrorists, and that the program is a tremendous intrusion into Americans’ fundamental privacy: “The problem is not that such large-scale industrial fishing invariably catches a few dolphins along with the tuna, but that between 99.999 and 100 percent of what you’re going to get is dolphin.” Stokes also warns that such an intrusive surveillance program will not only violate privacy rights, but be quite ineffective: “As the TSA, with its strip-searching of people’s elderly grandparents, abundantly proves every holiday season, blunt instruments and scorched earth tactics are of dubious value in catching genuine bad actors. In fact, blunt instruments and wide nets are the easiest for professional bad guys to evade. All you need to beat such surveillance tools is patience and know-how.…Blunt instruments like airport facial recognition software and random subway bag searches produce much more noise than they do signal, and any engineer or computer scientist worth his or her salt will tell you that an intelligent, targeted, low-tech approach beats a brute-force high-tech approach every time. There is no high-tech substitute for human intelligence gathering. In fact…an overload of crudely processed information is actually more likely to lead an analyst astray than it is to produce any useful insight.…In the end, brute force security techniques are not only corrosive to democratic values but they’re also bad for national security. They waste massive resources that could be spent more effectively elsewhere, and they give governments and countries a false sense of security that a savvy enemy can exploit to devastating effect.…[I]t’s not just enough to have sound intelligence; you also need political leaders who have the wisdom to use that intelligence appropriately.” [Ars Technica, 12/20/2005]
Chart showing NSA surveillance network. [Source: NSA Watch] (click image to enlarge)The National Security Agency has built a far larger database of information collected from warrantless surveillance of telephone and Internet communications to and from US citizens than the NSA or the Bush administration has acknowledged (see October 2001). On December 15, the New York Times exposed the NSA’s program (see December 15, 2005), which was authorized by President Bush in early 2002 (see Early 2002), but which actually began far earlier (see Spring 2001). The NSA built its database with the cooperation of several major American telecommunications firms (see June 26, 2006), and much of the information was mined directly into the US telecommunications system’s major connections. Many law enforcement and judicial officials question the legality of the program (see May 12, 2006 and December 18, 2005), and many say the program goes beyond the bounds of the Foreign Intelligence Surveillance Act (see 1978). One question is whether the FISA Court, or FISC, can authorize monitoring of international communications that pass through US-based telephonic “switches,” which handle much of the US’s electronic communications traffic. “There was a lot of discussion about the switches” in conversations with FISC, says a Justice Department official. “You’re talking about access to such a vast amount of communications, and the question was, How do you minimize something that’s on a switch that’s carrying such large volumes of traffic? The court was very, very concerned about that.” While Bush and his officials have insisted that the warrantless wiretaps only target people with known links to al-Qaeda, they have not acknowledged that NSA technicials have not only eavesdropped on specific conversations between people with no known links to terrorism, but have combed through huge numbers of electronic communications in search of “patterns” that might point to terrorism suspects. Such “pattern analysis” usually requires court warrants before surveillance can begin, but in many cases, no such warrants have been obtained or even requested. Other, similar data-mining operations, such as the Total Information Awareness program, developed by the Defense Department to track terror suspects (see March 2002), and the Department of Homeland Security’s CAPPS program, which screened airline passengers (see (6:20 a.m.-7:48 a.m.) September 11, 2001), were subjected to intense public scrutiny and outrage, and were publicly scrapped. The Bush administration has insisted that it has no intention of scrapping the NSA’s warrantless wiretapping program, because, as its officials have said, it is necessary to identify and track terrorism suspects and foil terrorist plots before they can be hatched. Administration officials say that FISC is not quick enough to respond to its need to respond to potential terrorist acts. A former technology manager at a major telecommunications company says that after 9/11, the leading telecom firms have been storing information on calling patterns and giving it to the federal government to aid in tracking possible terrorists. “All that data is mined with the cooperation of the government and shared with them, and since 9/11, there’s been much more active involvement in that area,” says the former manager. “If they get content, that’s useful to them too, but the real plum is going to be the transaction data and the traffic analysis. Massive amounts of traffic analysis information—who is calling whom, who is in Osama Bin Laden’s circle of family and friends—is used to identify lines of communication that are then given closer scrutiny.” And, according to a government expert on communications privacy who used to work at the NSA, says that in the last few years, the government has quietly encouraged the telecom firms to rout more international traffic through its US-based switches so it can be monitored. Such traffic is not fully addressed by 1970s-era laws that were written before the onset of modern communications technology; neither does FISA adequately address the issues surrounding that technology. Computer engineer Phil Karn, who works for a major West Coast telecom firm, says access to those switches is critical: “If the government is gaining access to the switches like this, what you’re really talking about is the capability of an enormous vacuum operation to sweep up data.” [New York Times, 12/24/2005]
Entity Tags: US Department of Defense, US Department of Justice, Total Information Awareness, New York Times, US Department of Homeland Security, Computer Assisted Passenger Prescreening System, Bush administration (43), Foreign Intelligence Surveillance Act, Foreign Intelligence Surveillance Court, George W. Bush, National Security Agency, Phil Karn
Timeline Tags: Civil Liberties
Receive weekly email updates summarizing what contributors have added to the History Commons database
Developing and maintaining this site is very labor intensive. If you find it useful, please give us a hand and donate what you can.
If you would like to help us with this effort, please contact us. We need help with programming (Java, JDO, mysql, and xml), design, networking, and publicity. If you want to contribute information to this site, click the register link at the top of the page, and start contributing.