Crimson: Empowering Strategic Reasoning in Cybersecurity through Large Language Models (2403.00878v1)
Abstract: We introduces Crimson, a system that enhances the strategic reasoning capabilities of LLMs within the realm of cybersecurity. By correlating CVEs with MITRE ATT&CK techniques, Crimson advances threat anticipation and strategic defense efforts. Our approach includes defining and evaluating cybersecurity strategic tasks, alongside implementing a comprehensive human-in-the-loop data-synthetic workflow to develop the CVE-to-ATT&CK Mapping (CVEM) dataset. We further enhance LLMs' reasoning abilities through a novel Retrieval-Aware Training (RAT) process and its refined iteration, RAT-R. Our findings demonstrate that an LLM fine-tuned with our techniques, possessing 7 billion parameters, approaches the performance level of GPT-4, showing markedly lower rates of hallucination and errors, and surpassing other models in strategic reasoning tasks. Moreover, domain-specific fine-tuning of embedding models significantly improves performance within cybersecurity contexts, underscoring the efficacy of our methodology. By leveraging Crimson to convert raw vulnerability data into structured and actionable insights, we bolster proactive cybersecurity defenses.
- V.O. Poddubnyi and O. Severinov. Vulnerability management using a formalized description, 2020.
- Cvss-bert: Explainable natural language processing to determine the severity of a computer security vulnerability from its description, 2021.
- Looking beyond iocs: Automatically extracting attack patterns from external cti, 2023.
- Attackg: Constructing technique knowledge graph from cyber threat intelligence reports, 2021.
- Threatkg: A threat knowledge graph for automated open-source cyber threat intelligence gathering and management, 2022.
- Vox populi, vox chatgpt: Large language models, education and democracy, 2023.
- Chain-of-thought prompting elicits reasoning in large language models. In Neural Information Processing Systems, 2022.
- Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
- Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, 2023.
- Common vulnerability scoring system. IEEE Security & Privacy, 4(6):85–89, 2007.
- Dave Shackleford. Who ’ s using cyberthreat intelligence and how ? 2015.
- Leveraging bert’s power to classify ttp from unstructured text. In 2022 Workshop on Communication Networks and Power Systems (WCNPS), pages 1–7. IEEE, 2022.
- MITRE Engenuity. Mapping att&ck to cve for impact. https://mitre-engenuity.org/cybersecurity/center-for-threat-informed-defense/our-work/mapping-attck-to-cve-for-impact/, October 2021. Accessed: 2024-03-01.
- Can generalist foundation models outcompete special-purpose tuning? case study in medicine, 2023.
- Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021.
- Task-aware retrieval with instructions, 2022.
- Gorilla: Large language model connected with massive apis, 2023.
- Mistral 7b, 2023.
- Llama 2: Open foundation and fine-tuned chat models, 2023.
- C-pack: Packaged resources to advance general chinese embedding, 2023.
- Lora: Low-rank adaptation of large language models, 2021.