Objectivity, Inc. Case Study: Objectivity Platform in THREADS

Click here to view, print or download this Case Study in Adobe Acrobat PDF format

Sector: Government Security & Intelligence

The Challenge:

Counter-terrorism analysts needed an application to compile and analyze HUMINT - intelligence that comes from human sources - with an end goal of generating actionable information. THREADS was the proposed application, and its developers set out to find the best database management solutions for the complex task.

The Objectivity/DB Solution:

We chose Objectivity/DB because we wanted proven technology that was highly reliable and offered a scalable repository for a wide range of complex data types

Having already used Objectivity/DB to create ASA, a similar intelligence tool, the developers knew from first-hand experience that Objectivity/DB allowed them faster development and proven scalability, among other advantages. Nonetheless, they created a performance study, pitting Objectivity/DB against a leading relational database management system (RDBMS).

The databases were tested for speed and accuracy in data insertion, messaging, EEIs and deep queries. In every test, Objectivity/DB matched or beat the RDBMS. In deep queries, Objectivity/DB was 25 times faster. Objectivity/DB was the clear choice for the THREADS project.

Northrop Grumman Mission Systems Application Assists U.S. Counter Terrorist and Force Protection Analysts in the Global War on Terrorism

The Objectivity/DB® based Threat HUMINT Reporting Evaluation Analysis and Display System (THREADS) application, developed by Northrop Grumman Mission Systems as a prototype, is now a deployed application operating in a variety of different environments. Objectivity/DB uniquely solves the complex data management requirements of THREADS, which assists Counter Terrorist and Force Protection analysts corroborate Human Intelligence (HUMINT) sources and correlate these "threads" of evidence with other sources leading to actionable intelligence. Objectivity/DB is the scalable "data fusion" repository storing, managing and associating large volumes of near real-time data with historical data and a priori knowledge models within THREADS.

Assisting with the Global War on Terrorism

The THREADS application, derived from a field proven analytical methodology, is being used in the Global War on Terrorism. Objectivity/DB enables THREADS to analyze and correlate large volumes of streaming event data that has complicated interrelationships with known attributes.

Objectivity/DB provides a reliable, scalable repository for the wide range of data types in THREADS and assists analysts in orchestrating multi-discipline reactions to defeat planned terrorist activities.

THREADS is largely dealing with heavy volumes of HUMINT data that is primarily in the form of unformatted text messages and reports. Analysts read hundreds of these per day to develop a "thread" of evidence that produces a case to be monitored for a potential response. THREADS tracks information from these incoming messages looking for people, facilities, locations, weapons, organizations, vehicles or similar identified attributes that are extracted from the messages. An analyst can select one of the messages and review the tree of extracted attributes correcting any errors or adding more information to it. As location information is identified THREADS automatically hyperlinks a map to the location reference in the message. Depending on the location references type, i.e. a street address or coordinates with good resolution versus a more general city name, THREADS displays the appropriate geographic area map. When an updated case is returned to the Objectivity/DB platform all of the future incoming data is monitored and analyzed with the new revisions generating alerts for analyst response. The Objectivity/DB based THREADS application makes connections and builds known relationships and associations enabling U.S. Counter Terrorist and Force Protection analysts develop and follow cases of terrorist activities for intelligence organizations.

A Proven Highly Reliable and Scalable Technology

"We chose Objectivity/DB because we wanted proven technology that was highly reliable and offered a scalable repository for a wide range of complex data types," said David Sopsick, Senior Software Engineer of Northrop Grumman Mission Systems. Within their tight time constraints the Objectivity/DB platform enabled the THREADS prototype to be developed faster than would have been possible with other technologies. Objectivity/DB's schema evolution provided advantage over other technologies that had a difficult time with change; adding new fields resulted in changes to the database schema, which Objectivity/DB handled with ease. Objectivity/DB was known to be more flexible, allowing evolution over time and could be deployed in a wide range of configurations. Additionally, Objectivity/DB is wholly developed and maintained in the USA with no outsourcing or incorporation of foreign-controlled components, which also was an important consideration.

The proven analytical methodology that forms the knowledge-base used to corroborate HUMINT reporting of threats in THREADS was field developed through the fusion of National Systems data to generate actionable intelligence. THREADS helps analysts make timely connections to largely unformatted data reporting related threat activities gathered from multiple intelligence domains. The THREADS application incorporates more traditional intelligence to back-up the HUMINT data using inherent mapping, imagery and geographic capabilities to determine the exact location and physical attributes of threat-related targets. This application enables a previously unavailable information management capability to an analyst's visibility into on-going counter terrorism threat analysis activities.

Technology Evaluation for Deployment

As part of the evaluation process for deployment a comparison of the capabilities of the Objectivity/DB ODBMS platform versus an RDBMS platform was conducted. A representative test case was designed where typical THREADS queries were executed on the two platforms. Evaluations of the configuration strategies were conducted to insure optimized performance by allowing experts in each technology helping to tune the implementations. Multiple container strategies were also considered and the same equipment was used throughout the study. The benchmark results of the test case clearly demonstrated the Objectivity/DB implementation was significantly better than the best performance metrics for the RDBMS implementation. As shown in the following graph in some cases the Objectivity/DB platform was performing more than twenty times as many operations as the RDBMS platform. Valuable lessons were learned including a disk cache anomaly that significantly improves query performance, putting items frequently queried in smaller objects, and; defragmenting the disk has considerable impact on performance.

In April, 2005 the first THREADS—MU multi-user client/server application that enables users to share databases and allows many users on the same database was installed. The multi-user version needed a different approach realizing network traffic can be significant for large queries when the system is distributed. The Objectivity/DB scalability and distributed platform facilitated the development of a query server solution. The solution executes remote queries on a multi-threaded Objectivity/DB based query server. The server executes queries and returns OIDs to the clients who are able to access objects directly using the returned OIDs. Additionally the application provides an audit trail for management.