Is the future of intelligence analysis imperiled by the very data it relies upon? We stand on the brink of a transformation in data integration. The Intelligence Community (IC) has several formidable tasks ahead. With data availability rising, budgets declining, and customer trust waning—closing the gap between unclassified and classified data to achieve accurate all-source analysis is vital. The question is, can the IC evolve fast enough to harness this potential?
Navigating the Data Deluge
Within the next year, we expect the datasphere to measure between 170 to 200 zettabytes. Consider this staggering fact: if you stored 175 zettabytes on DVDs, the stack would circle the Earth 222 times.
This number is expected to grow, and the explosion of data across the internet is reshaping how intelligence agencies gather and analyze information. In fact, the quality and abundance of open-source data now rival traditional national intelligence sources.
With the Internet of Things (IoT), we all have devices on us, and everything emanates in one way or another, generating massive amounts of telemetry data. This data is ridiculously available and open for sale, and anyone can have it to include the US and our adversaries. This democratization of data presents a dual challenge for the IC—how to leverage these vast sources for national security while preventing adversaries from doing the same.
The Complexity of Data
The projected size of the datasphere reflects more than growth; they underscore the complexity and variety of information the IC must manage. With such a vast sea of data, the risk is clear—analysts are drowning in irrelevant information or missing critical insights hidden in isolated data silos. Information that was once considered "secret" is no longer a secret. To some degree, open-source intelligence has broken the barrier to secret information, and the IC community must adapt to categorizing and classifying this sensitive data efficiently.
Accrete's Federal CTO, Brian Drake, suggests combining all the data sources the IC is pulling from and combining them to uncover deeper insights. According to Brian, “The IC is optimized to collect, protect, and understand secrets. In recent years, it has attempted to just collect data and figure out how secret the information might be. This just isn’t sustainable and distracts the IC from its core mission. It’s time for the IC to focus on its critical function of stealing secrets and processing open source data for the purpose of creating all-source intelligence.”
In his Executive Session at DoDIIS 2024, Brian stated that valuable information is in the open and classified data should be layered on top of that open-source information. This provides customers with a more complete picture and helps achieve more profound insights.
The reliance on incomplete or constrained data sets prevents the discovery of vital contrary or supportive intelligence, something the United States can't afford to have happen as adversaries close in. We must break down these barriers to maximize the potential of acquired, collected, or created data.
Endemic Challenges
The IC faces a significant hurdle with its current strategy. Despite mandates like ICD 501 aiming to enhance sharing and integration, the reality is stark. Data remains siloed due to mission-specific management, hindering the IC's ability to exploit its resources fully. The Intelligence Community Information Environment (IC IE) Data Strategy from 2017-2021 emphasized the need for data freedom—an aspiration yet fully realized. Analysts must access data independently, linking multiple databases and analytic environments to generate comprehensive intelligence, causing a "swivel chair" effect that is not efficient.
Budgets compound these challenges, as the IC receives only a portion of the National Defense Budget, which does not sufficiently match the data growth and requirements. Despite recognizing data as a strategic asset, the IC's deployment of AI-enabled capabilities lags behind in what's needed to maintain intelligence superiority.
Lastly, given the degree of misinformation polluting many data streams and the political environment, customers are going to trust the IC less. Ensuring that the best, more reliable data sources available are analyzed and delivered to customers in a timely manner is critical.
Pioneering Integrated Data Solutions
Achieving all-source analysis means truly encompassing all data sources; classified and unclassified. The path forward is clear, but not without hurdles. Fortunately, events like DoDIIS bring together industry, government officials, and academia to discuss these challenges and offer solutions.
We must adopt more strategic data processing methods to address fiscal constraints and unsustainable mass data ingestion into classified systems. As Brian, stated at DoDIIS, "We must process low and bring high." This means processing what we can in unclassified systems and integrating with classified data for a comprehensive snapshot. Furthermore, he suggests that only relevant, high-value intelligence is processed at the highest classification levels.
The IC must leverage adaptive technologies and innovative solutions to integrate data seamlessly. This includes harnessing AI for real-time data processing and adopting a modular approach to data sharing that transcends individual agency silos. By prioritizing data integration and removing unnecessary barriers, the IC can unlock the full potential of its intelligence capabilities.
The Future of Data Integration
The future of data integration in the intelligence community hinges on action. We must address the fiscal and strategic challenges that impede our progress. By fostering a culture of collaboration and trust, we can achieve the true all-source analysis imperative for decision advantage.
Join us in shaping this transformation—process low, bring high. It's time for the IC to redefine what "all sources" indeed mean and secure our intelligence future. At Accrete, we're here to support the all-source mission