- The paper's main contribution is ThreMoLIA, which automates threat modeling for LLM-integrated applications using a novel LLM-based approach.
- It leverages Retrieval Augmented Generation and data aggregation to integrate diverse security data and tailor prompt strategies to varied expertise levels.
- The approach is evaluated through industrial case studies that benchmark its performance against traditional methods, highlighting improved efficiency in early threat detection.
LLMs are increasingly integrated into traditional software applications, creating what the authors call LLM-Integrated Applications (LIAs). While LIAs offer enhanced capabilities, they also expand the attack surface and introduce novel security threats, such as prompt injections. Threat modeling is a crucial practice for identifying and mitigating these threats early in the software development lifecycle. However, traditional threat modeling methods and tools are often manual, time-consuming, require significant security expertise, and are not specifically tailored to the dynamic and non-deterministic nature of LIAs and their unique threat landscape.
The paper proposes ThreMoLIA (2504.18369), a vision for an LLM-based approach designed to assist practitioners in the threat modeling of LIAs. The primary goals of ThreMoLIA are to provide a method for early-stage threat modeling of LIAs, develop a tool integrating existing threat models and knowledge sources, and ensure high-quality threat modeling outputs with reduced reliance on extensive security expertise. Developed in collaboration with an industry partner, the approach aims to accelerate the security assessment process and maintain up-to-date threat models throughout the application's lifecycle.
The ThreMoLIA approach is guided by established threat modeling principles, such as Shostack's Four Question Framework. It begins by representing the LIA architecture using Data-Flow Diagrams (DFDs) to create an abstraction of components and data flows. Threat identification and assessment leverage existing frameworks like the OWASP Top 10 for LLM applications and MITRE ATLAS, adapted to the LIA context, alongside traditional methods like STRIDE. Identified LLM-specific threats are then cataloged, potentially using MITRE ATLAS attack playbooks and tactics, with suggested mitigation techniques also drawn from sources like MITRE ATLAS.
The proposed architecture for ThreMoLIA includes several key components:
- Retrieval Augmented Generation (RAG): This component provides the LLM with necessary context from external data sources to generate accurate threat models. It follows a standard RAG workflow involving indexing, retrieval, and generation based on the user query (part of the stakeholder prompt). Vectorization is used to compare and retrieve relevant documents from a database.
- Data Aggregation: Collects and aggregates data from various sources to populate the RAG database. Sources include natural language documents (requirements, design specifications), architectural diagrams (DFDs), existing threat models from previous sessions or similar projects, and potentially monitoring data from deployed LIAs for continuous threat modeling.
- Prompting: Constructs the input prompt for the LLM. This includes a predefined system prompt tailored for threat modeling, the stakeholder's user prompt, a reasoning strategy (e.g., Chain of Thought), and context retrieved by the RAG component based on system documentation.
- Quality Assurance: Performs checks on the generated threat model. This includes verifying syntactical correctness (e.g., for formats like Open Threat Model), parsing the output to validate relationships, and using test suites based on metamorphic testing principles to assess the quality. A "health score" summarizes the quality for stakeholders, allowing them to refine their input. The approach considers extending metrics from the Machine Learning Security Maturity Model (MLSMM) (Jedrzejewski et al., 2023) for this purpose.
The authors identify several challenges in implementing ThreMoLIA, including:
- RAG: Vectorizing non-textual data like DFDs and effectively filtering relevant information without redundancy or token limits issues.
- Data Aggregation: Handling heterogeneous data quality and company-specific terminology, prioritizing data sources, and mitigating LLM source bias.
- Prompting: Designing prompt templates adaptable to users with varying security expertise and managing complex multi-user interactions across sessions.
- Quality Assurance: The current lack of standardized metrics for threat model quality, which are necessary for defining robust validation methods like metamorphic relationships and test case selection strategies.
The evaluation plan for ThreMoLIA involves creating a benchmark for LIA threat models by extracting and validating metrics from literature and expert feedback. This benchmark will be used in industrial case studies to compare ThreMoLIA's performance against traditional methods. A multiple case study is planned to observe its integration into existing security practitioner workflows.
Early investigations using ChatGPT-3.5 Turbo on a simple LIA architecture showed that while the LLM could refer to relevant frameworks like OWASP Top 10 LLM and MITRE ATLAS, initial vague prompts required clarification from the model, indicating the importance of the RAG and Prompting components for providing necessary context and structure. The paper concludes by presenting a preliminary list of evaluation metrics for threat models gathered from existing literature, which will be refined with expert input.
In summary, ThreMoLIA represents a forward-looking approach to address the evolving security challenges of LIAs by leveraging LLMs, RAG, and structured quality assurance to automate and enhance the threat modeling process, making it more accessible and efficient for industry practitioners.
