Mind the Gap: Foundation Models and the Covert Proliferation of Military Intelligence, Surveillance, and Targeting (2410.14831v1)

Abstract: Discussions regarding the dual use of foundation models and the risks they pose have overwhelmingly focused on a narrow set of use cases and national security directives-in particular, how AI may enable the efficient construction of a class of systems referred to as CBRN: chemical, biological, radiological and nuclear weapons. The overwhelming focus on these hypothetical and narrow themes has occluded a much-needed conversation regarding present uses of AI for military systems, specifically ISTAR: intelligence, surveillance, target acquisition, and reconnaissance. These are the uses most grounded in actual deployments of AI that pose life-or-death stakes for civilians, where misuses and failures pose geopolitical consequences and military escalations. This is particularly underscored by novel proliferation risks specific to the widespread availability of commercial models and the lack of effective approaches that reliably prevent them from contributing to ISTAR capabilities. In this paper, we outline the significant national security concerns emanating from current and envisioned uses of commercial foundation models outside of CBRN contexts, and critique the narrowing of the policy debate that has resulted from a CBRN focus (e.g. compute thresholds, model weight release). We demonstrate that the inability to prevent personally identifiable information from contributing to ISTAR capabilities within commercial foundation models may lead to the use and proliferation of military AI technologies by adversaries. We also show how the usage of foundation models within military settings inherently expands the attack vectors of military systems and the defense infrastructures they interface with. We conclude that in order to secure military systems and limit the proliferation of AI armaments, it may be necessary to insulate military AI systems and personal data from commercial foundation models.

• The paper reveals how commercial foundation models can covertly bridge civilian data with military ISTAR capabilities, heightening security concerns.
• It demonstrates that current safeguards overlook the misuse of personal data and the expansion of attack vectors like model extraction and adversarial inputs.
• It calls for urgent policy reforms that enhance data traceability and protection to mitigate the dual-use risks inherent in AI technologies.

Mind the Gap: Foundation Models and Military ISTAR Proliferation

The paper, "Mind the Gap: Foundation Models and the Covert Proliferation of Military Intelligence, Surveillance, and Targeting" by Heidy Khlaaf, Sarah Myers West, and Meredith Whittaker, offers an incisive critique of the current discourse surrounding the dual-use nature of AI technologies in military contexts. The focus is on how commercial foundation models serve as a bridge between civilian data and military ISTAR (Intelligence, Surveillance, Target Acquisition, and Reconnaissance) capabilities, posing significant national security risks.

Core Arguments

The authors challenge the prevailing focus on CBRN (Chemical, Biological, Radiological, and Nuclear) scenarios, which, they argue, has restricted policy debates and interventions. Instead, the paper highlights present-day applications of AI in military systems, particularly ISTAR—where AI misuse or failure can lead to severe geopolitical and civilian repercussions. There is a critical examination of how commercial foundation models, such as those used for general AI purposes, could be repurposed to enhance ISTAR capabilities.

Key Insights and Risks

1. Data as a Risk Vector: The paper underscores the inadequacy of current interventions that fixate on compute thresholds and model weight restrictions. These measures do not address the exploitation of personal data within AI models. Personal data, which is increasingly integrated into commercial foundation models, poses a proliferation risk as it can be repurposed for military operations. This risk remains unmitigated by current governance strategies.
2. Expansion of Attack Vectors: The employment of foundation models in military contexts inherently broadens the attack surface, enabling adversaries to execute a variety of attacks such as model extraction, membership inference, and adversarial examples. These vulnerabilities persist regardless of whether models are open or closed source.
3. Efficacy of Policy and Governance: The authors call for a shift in policy focus towards the protection and traceability of data used in AI models to bolster national security. They suggest that data should be considered as critically as other components when designing interventions aimed at preventing AI-based military proliferation.

Implications for the Future

The findings have profound implications for AI governance, especially in military applications. The authors advocate for more stringent controls on personal data usage in AI models and emphasize the necessity of traceability in AI supply chains. They further propose that military-exclusive AI models, unburdened by commercial lineage, might provide a viable alternative. However, these solutions must overcome inherent limitations, such as the basic execution vulnerabilities of deep neural networks and the challenge of ensuring traceability.

Concluding Thoughts

The paper delivers a compelling argument for the need to reassess and broaden the scope of AI governance and nonproliferation strategies. Addressing the dual-use nature of AI and securing personal data against misuse are imperative steps in mitigating the potential escalation of military conflicts driven by AI advancements. The authors’ recommendations serve as a foundation for developing more nuanced policies capable of confronting the dual-use challenge posed by foundation models in ISTAR contexts. The future of AI in military applications requires rigorous scrutiny and governance to navigate the blurred lines between civilian and military use responsibly.