!! History Commons Alert, Exciting News
Profile: Novel Intelligence from Massive Data (NIMD)
Novel Intelligence from Massive Data (NIMD) was a participant or observer in the following events:
An illustration of the NIMD dataflow. [Source: LibertyThink.com] (click image to enlarge)Shortly after the 9/11 attacks, the NSA awards $64 million in research contracts for a program called Novel Intelligence from Massive Data (NIMD). [New York Times, 5/21/2003; National Journal, 1/20/2006] NIMD is one of several cutting-edge data mining technologies that not only has the capability of finding keywords among millions of electronically monitored communications, but can find hidden relationships among data points, and even critique the thinking and biases of a particular analyst and suggest alternative hypotheses differing from the human analysts’ conclusion. Like other data-mining technologies, the NSA will steadfastly refuse to discuss whether NIMD is used to analyze data from domestic surveillance operations. NIMD is designed as an preliminary sort program, to keep human analysts from becoming overwhelmed by raw data. In essence, NIMD is an early-warning system. “NIMD funds research to…help analysts deal with information-overload, detect early indicators of strategic surprise, and avoid analytic errors,” according to the “Call for 2005 Challenge Workshop Proposals” released by the Advanced Research and Development Activity (ARDA). ARDA was founded in 1998 to create, design, and field new technologies for US intelligence agencies, particularly the NSA. A selected few Congressional lawmakers (see January 18, 2006) were informed that the warrantless surveillance program authorized by President George W. Bush (see Early 2002) was designed to be an early-warning system for possible terrorist attacks or plans. Assistant Attorney General William Moschella will inform the top Democrats and Republicans on the House and Senate Intelligence committees in December 2002 that the “president determined that it was necessary following September 11 to create an early-warning detection system” to prevent more attacks. He will justify the use of programs such as NIMD by claiming, as NSA director Michael Hayden and other administration officials have repeatedly claimed, that the Foreign Intelligence Surveillance Act (FISA), which allows the government to obtain warrants to conduct domestic eavesdropping or wiretapping, “could not have provided the speed and agility required for the early-warning detection system.” Many experts outside of the Bush administration feel that NIMD and other programs do not have to operate outside of the Foreign Intelligence Surveillance Act (FISA) because of limitations in the law, but because of the fact that the programs cannot meet the law’s minimum requirements for surveillance. FISA requires that any such surveillance must have a probable cause that the target is a terrorist. NIMD has no such threshold. Steven Aftergood, an expert on intelligence and government secrecy with the Federation of American Scientists, will say in 2006, “Logistically speaking, the early-warning approach may involve a significant increase in the number of surveillance actions. It may be that neither the Justice Department nor the [Foreign Intelligence Surveillance Court, which approves wiretapping warrants] is prepared to prepare and process several thousand additional FISA applications per year, beyond the 1,700 or so approved in 2004.” [National Journal, 1/20/2006] Some experts will later express the opinion that NIMD is the controversial Total Information Awareness program in a slightly different form (see February 2003 and September 2002).
Entity Tags: Senate Intelligence Committee, US Department of Justice, Total Information Awareness, William E. Moschella, Tom Armour, Novel Intelligence from Massive Data, Steven Aftergood, Michael Hayden, National Security Agency, Advanced Capabilities for Intelligence Analysis, Advanced Research and Development Activity, John Poindexter, Foreign Intelligence Surveillance Act, George W. Bush, Federation of American Scientists (FAS), House Intelligence Committee
Timeline Tags: Civil Liberties
The NSA’s enormous data mining program, Novel Intelligence from Massive Data (NIMD—see After September 11, 2001), closely resembles another data mining program, the Defense Department’s Total Information Awareness program (see March 2002). TIA, which will be suspended in 2003 after outcries from citizens and legal experts concerned over that program’s refusal to comply with fundamental Constitutional guarantees of privacy, was also designed as an early-warning system that not only compiled intelligence data, but mined through private financial databases for credit-card and other financial transactions. Six of the corporations and research institutions who win NIMD contracts also held contracts for the earlier TIA project. Much of their work with TIA duplicated the same aspects and protocols to be used by NIMD, including challenging analytic assumptions and building prototype data-mining devices. Both TIA and NIMD attempt to second-guess human analysts’ conclusions over a particular data schema by creating a database of what TIA creator John Poindexter once called “plausible futures,” or likely terrorism scenarios. NIMD is a creation of the Advanced Research and Development Activity agency (ARDA); another ARDA project, the Advanced Capabilities for Intelligence Analysis (ACIA) also envisions a similar database (see 2005). Though TIA focused more on counterterrorism than the more sweeping NIMD, the two projects coordinated closely with one another, according to former program manager Tom Armour, who worked in Poindexter’s office.
NIMD Survives In Other Agencies - Congress will eliminate funding for TIA and other Poindexter projects, but many of those projects, and related projects such as NIMD, do not disappear. Many are instead transferred to intelligence agencies such as the NSA. Although information about these projects is strictly classified, a former Army intelligence analyst familiar with the programs will confirm in 2006 that large elements of TIA were transferred to other agencies, where they will continue to be researched and implemented. It is highly likely that NIMD is an offshoot and outgrowth of TIA. Armour will say the two programs are specifically designed to analyze large amounts of phone and e-mail traffic: “That’s, in fact, what the interest is.” [National Journal, 1/20/2006]
Government 'Shell Game' - Steven Aftergood of the Federation of American Scientists says in 2004, “The whole congressional action looks like a shell game. There may be enough of a difference for them to claim TIA was terminated while for all practical purposes the identical work is continuing.” [Associated Press, 2/23/2004] Aftergood will note that NIMD has thrived in the shadows where TIA died of exposure: “Pursued with a minimal public profile and lacking a polarizing figure like Admiral Poindexter to galvanize opposition, NIMD has proceeded quietly even as TIA imploded.” [Defense Tech, 9/26/2003]
Assistant Attorney General William Moschella informs the ranking members of the House and Senate Intelligence committees of the administration’s use of potentially unconstitutional data mining and electronic surveillance programs after the 9/11 attacks. Moschella tells the lawmakers, “The president determined that it was necessary following September 11 to create an early-warning detection system” to prevent more attacks. One such program is the Novel Intelligence from Massive Data (NIMD) initiative (see After September 11, 2001). Moschella echoes the claims of National Security Agency director Michael Hayden and other administration officials, saying that the Foreign Intelligence Surveillance Act (FISA), which allows the government to obtain warrants to conduct domestic eavesdropping or wiretapping, “could not have provided the speed and agility required for the early-warning detection system.” [National Journal, 1/20/2006]
Domestic Surveillance Began Before 9/11? - Though Bush officials eventually admit to beginning surveillance of US citizens only after the 9/11 attacks, that assertion is disputed by evidence suggesting that the domestic surveillance program began well before 9/11 (see Late 1999, February 27, 2000, December 2000, February 2001, February 2001, Spring 2001, July 2001, and Early 2002). Moschella informs the lawmakers of none of this.
Steven Aftergood of the Federation of American Scientists says that he is not sure that Congress’s public termination of the Total Information Awareness (TIA) project (see January 23, 2003) was as real and outrage-driven as it seemed at the time. “The whole congressional action looks like a shell game,” Aftergood says. “There may be enough of a difference for them to claim TIA was terminated while for all practical purposes the identical work is continuing.” While Congress terminated TIA with visible indignation, it also quietly funded the “National Foreign Intelligence Program,” and never identified which intelligence agency would do the work—which was also kept from the public eye. Congress did say that none of the research would be used against US citizens. No one in Congress will discuss how many of Poindexter’s programs survived, but knowledgeable sources will confirm that some 18 data-mining programs known as Evidence Extraction and Link Discovery in Poindexter’s research were preserved after TIA’s termination. These programs may well include the sprawling data mining program known as Novel Intelligence from Massive Data (NIMD) (see After September 11, 2001), though this cannot be confirmed. Former TIA chief John Poindexter’s vision of the technology behind NIMD envisioned software that can quickly analyze “multiple petabytes” of data. A single petabyte would fill the Library of Congress space for 18 million books more than 50 times, or could hold 40 pages of text for each of the more than 6.2 billion humans on Earth. Poindexter and his colleagues envisioned the program as handling a petabyte or more of data a month. [Associated Press, 2/23/2004] Concerns about the privacy rights of US citizens being damaged by the program are rife. “If they were to stick to strictly military-related research and development, there is less of an issue, but these technologies have much broader social implications,” says Barbara Simons, a computer scientist who is past president of the Association of Computing Machinery, an organization that has expressed concerns about TIA. [New York Times, 5/21/2003] At least one Senator is uncomfortable with the apparent resurgence of TIA. Jay Rockefeller (D-WV) will write Vice President Dick Cheney in June 2003 after receiving a briefing on the various secret surveillance programs (see July 17, 2003). Rockefeller will write, “As I reflected on the meeting today, John Poindexter’s TIA project sprung to mind, exacerbating my concern regarding the direction the administration is moving with regard to security, technology, and surveillance.” [National Journal, 1/20/2006]
The National Security Agency calls for proposals in regard to a new electronic surveillance program, the Advanced Capabilities for Intelligence Analysis (ACIA). Like its cousin, the Novel Intelligence from Massive Data (NIMD) project (see After September 11, 2001), ACIA will use a huge electronic database of information on US citizens and foreign nationals to track potential terrorists and terror plots. Like NIMD, ACIA will look for ways “to construct and use plausible futures in order to provide additional, novel interpretations for today’s collection” of intelligence information, according to the call for proposals. [National Journal, 1/20/2006]
Receive weekly email updates summarizing what contributors have added to the History Commons database
Developing and maintaining this site is very labor intensive. If you find it useful, please give us a hand and donate what you can.
If you would like to help us with this effort, please contact us. We need help with programming (Java, JDO, mysql, and xml), design, networking, and publicity. If you want to contribute information to this site, click the register link at the top of the page, and start contributing.