A recent article by Cheryl Gerber in the November 21 issue of Military Information Technology online on "Smart Searching"  provides a useful overview of issues and leading vendors dealing with large-scale issues of content search and discovery.  Some of the useful vendors covered in this article include:   Endeca Technologies, Basis Technology, Inxight Software, Insightful, Attensity, Convera, SRA International (NetOwl), ClearForest and BrightPlanet.

The focus of these efforts in the Defense Intelligence Agency is characterized by Gerber as:

The unique requirements of defense intelligence analysts are refining search technology down from mass production, with its vast and sometimes trivial outcomes, to more guided, dynamic navigation able to produce results that are both inclusive and relevant.

As one of the largest collectors of information on the planet, the Defense Intelligence Agency (DIA) is responsible for amassing and analyzing all sources of human intelligence in the field from all information types in a multitude of languages.

"This forces us to deal with huge volumes of data. It's an enormous challenge," said a senior DIA official.

The task is indeed a massive one. Sources of intelligence in the field include feeds from UAVs, intelligence, surveillance and reconnaissance data from a vast array of sensors and overhead platforms, signal intelligence, satellites, film and video, not to mention all the data from the open source world. "We need to manage all that data and make it available as quickly as possible to analysts," the DIA official said.

The intel community, as with others forming in the commercial sector, is also relying on community standards for metadata transfer and management.  In the case of the DIA, these standards are being provided by the Intelligence Community Metadata Working Group (ICMWG), which is charged with establishing standards for the tagging of all data used by DIA systems.

In the article, BrightPlanet’s Duncan Witte commented on the importance of having the abilities to "organize, manage and distribute the huge volume of information as well. You need various specialties that allow collaboration with teammates and effective distribution of information."

This article again affirms that the federal intelligence community continues to assume the lead in large-scale content discovery and evaluation.

As the article notes, the DIA maintains a steady push toward technology improvement. "We try to do the best we can with the volumes. In-house we have a lot of expertise on search algorithms and text analysis. But we need to do a better job of combing through the massive volumes of information to find that which is interesting and nontrivial in a way that leads to knowledge discovery. We need better information retrieval through machine understanding of the semantic meaning of text, regardless of language," the DIA official said.