Papers
Topics
Authors
Recent
Search
2000 character limit reached

Semantic Ranking for Automated Adversarial Technique Annotation in Security Text

Published 25 Mar 2024 in cs.CR | (2403.17068v1)

Abstract: We introduce a new method for extracting structured threat behaviors from threat intelligence text. Our method is based on a multi-stage ranking architecture that allows jointly optimizing for efficiency and effectiveness. Therefore, we believe this problem formulation better aligns with the real-world nature of the task considering the large number of adversary techniques and the extensive body of threat intelligence created by security analysts. Our findings show that the proposed system yields state-of-the-art performance results for this task. Results show that our method has a top-3 recall performance of 81\% in identifying the relevant technique among 193 top-level techniques. Our tests also demonstrate that our system performs significantly better (+40\%) than the widely used LLMs when tested under a zero-shot setting.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. 2023. https://mitre-engenuity.org/cybersecurity/center-for-threat-informed-defense/our-work/threat-report-attck-mapper-tram/
  2. SMET: Semantic Mapping of CVE to ATT&CK and Its Application to Cybersecurity. In IFIP Annual Conference on Data and Applications Security and Privacy. Springer, 243–260.
  3. Looking Beyond IoCs: Automatically Extracting Attack Patterns from External CTI. arXiv preprint arXiv:2211.01753 (2022).
  4. Looking Beyond IoCs: Automatically Extracting Attack Patterns from External CTI. arXiv:2211.01753 [cs.CR]
  5. Nima Asadi and Jimmy Lin. 2013. Effectiveness/efficiency tradeoffs for candidate generation in multi-stage retrieval architectures. In Proceedings of the 36th international ACM SIGIR conference on Research and development in information retrieval. 997–1000.
  6. Symantec Enterprise Blogs. 2023. Threat Intelligence. https://symantec-enterprise-blogs.security.com/blogs/threat-intelligence Accessed: May 27, 2023.
  7. Broadcom. 2023. Protection Bulletin. https://www.broadcom.com/support/security-center/protection-bulletin Accessed: May 27, 2023.
  8. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  9. Assessing efficiency–effectiveness tradeoffs in multi-stage retrieval systems without using relevance judgments. Information Retrieval Journal 19 (2016), 351–377.
  10. MITRE Corporation. 2022. MITRE ATT&CK Enterprise Matrix. https://attack.mitre.org/versions/v12/matrices/enterprise/. Accessed: June 5, 2023.
  11. “Is this document relevant?… probably” a survey of probabilistic models in information retrieval. ACM Computing Surveys (CSUR) 30, 4 (1998), 528–552.
  12. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), Vol. 1. 4171–4186.
  13. A system for automated open-source threat intelligence gathering and management. In Proceedings of the 2021 International Conference on Management of Data. 2716–2720.
  14. Enabling efficient cyber threat hunting with cyber threat intelligence. In 2021 IEEE 37th International Conference on Data Engineering (ICDE). IEEE, 193–204.
  15. Embedding-based retrieval in facebook search. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2553–2561.
  16. TTPDrill: Automatic and Accurate Extraction of Threat Actions from Unstructured Text of CTI Sources. In Proceedings of the 33rd Annual Computer Security Applications Conference (Orlando, FL, USA) (ACSAC ’17). Association for Computing Machinery, New York, NY, USA, 103–115. https://doi.org/10.1145/3134600.3134646
  17. Ttpdrill: Automatic and accurate extraction of threat actions from unstructured text of cti sources. In Proceedings of the 33rd annual computer security applications conference. 103–115.
  18. A Knowledge Base Question Answering System for Cyber Threat Knowledge Acquisition. In 2022 IEEE 38th International Conference on Data Engineering (ICDE). IEEE, 3158–3161.
  19. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300 (2019).
  20. Automated retrieval of att&ck tactics and techniques for cyber threat reports. arXiv preprint arXiv:2004.14322 (2020).
  21. AttacKG: Constructing Technique Knowledge Graph From Cyber Threat Intelligence Reports. In Computer Security – ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part I (Copenhagen, Denmark). Springer-Verlag, Berlin, Heidelberg, 589–609. https://doi.org/10.1007/978-3-031-17140-6_29
  22. AttacKG: Constructing technique knowledge graph from cyber threat intelligence reports. In Computer Security–ESORICS 2022: 27th European Symposium on Research in Computer Security, Copenhagen, Denmark, September 26–30, 2022, Proceedings, Part I. Springer, 589–609.
  23. Acing the ioc game: Toward automatic discovery and analysis of open-source cyber threat intelligence. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 755–766.
  24. Pretrained transformers for text ranking: Bert and beyond. Springer Nature.
  25. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 (2019).
  26. Query driven algorithm selection in early stage retrieval. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. 396–404.
  27. High accuracy retrieval with multiple nested ranker. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval. 437–444.
  28. Understanding the reproducibility of crowd-reported security vulnerabilities. In 27th USENIX Security Symposium (USENIX Security 18). 919–936.
  29. Ms marco: A human-generated machine reading comprehension dataset. (2016).
  30. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020).
  31. Scientific claim verification with VerT5erini. arXiv preprint arXiv:2010.11930 (2020).
  32. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 1 (2020), 5485–5551.
  33. Cybert: Contextualized embeddings for the cybersecurity domain. In 2021 IEEE International Conference on Big Data (Big Data). IEEE, 3334–3342.
  34. Nils Reimers and Iryna Gurevych. 2019a. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084 (2019).
  35. Nils Reimers and Iryna Gurevych. 2019b. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. CoRR abs/1908.10084 (2019). arXiv:1908.10084 http://arxiv.org/abs/1908.10084
  36. CSKG4APT: A Cybersecurity Knowledge Graph for Advanced Persistent Threat Organization Attribution. IEEE Transactions on Knowledge and Data Engineering (2022).
  37. Okapi at TREC-3. Nist Special Publication Sp 109 (1995), 109.
  38. Extractor: Extracting attack behavior from threat reports. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 598–615.
  39. Extractor: Extracting Attack Behavior from Threat Reports. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE Computer Society, Vienna, Austria, 598–615. https://doi.org/10.1109/EuroSP51992.2021.00046
  40. Securelist. 2023. APT Reports Archives. https://securelist.com/category/apt-reports/ Accessed: May 27, 2023.
  41. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223 (2019).
  42. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136 (2019).
  43. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  44. CTI ANT: Hunting for Chinese threat intelligence. In 2020 IEEE International Conference on Big Data (Big Data). IEEE, 1847–1852.
  45. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944 (2023).
  46. WithSecure. 2023. Publications. https://labs.withsecure.com/publications Accessed: May 27, 2023.
  47. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244 (2023).
  48. A unified pretraining framework for passage ranking and expansion. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35. 4555–4563.
  49. Designing templates for eliciting commonsense knowledge from pretrained sequence-to-sequence models. In Proceedings of the 28th International Conference on Computational Linguistics. 3449–3453.
  50. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629 (2022).
  51. Pretrained transformers for text ranking: BERT and beyond. In Proceedings of the 14th ACM International Conference on web search and data mining. 1154–1156.
  52. TIM: threat context-enhanced TTP intelligence mining on unstructured threat data. Cybersecurity 5, 1 (2022), 3.
  53. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685 (2023).
  54. Ziyun Zhu and Tudor Dumitras. 2018. Chainsmith: Automatically learning the semantics of malicious campaigns by mining threat intelligence reports. In 2018 IEEE European symposium on security and privacy (EuroS&P). IEEE, 458–472.
  55. Pre-trained language model based ranking in Baidu search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining. 4014–4022.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free