Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

AI ATAC 1: An Evaluation of Prominent Commercial Malware Detectors (2308.14835v1)

Published 28 Aug 2023 in cs.CR

Abstract: This work presents an evaluation of six prominent commercial endpoint malware detectors, a network malware detector, and a file-conviction algorithm from a cyber technology vendor. The evaluation was administered as the first of the Artificial Intelligence Applications to Autonomous Cybersecurity (AI ATAC) prize challenges, funded by / completed in service of the US Navy. The experiment employed 100K files (50/50% benign/malicious) with a stratified distribution of file types, including ~1K zero-day program executables (increasing experiment size two orders of magnitude over previous work). We present an evaluation process of delivering a file to a fresh virtual machine donning the detection technology, waiting 90s to allow static detection, then executing the file and waiting another period for dynamic detection; this allows greater fidelity in the observational data than previous experiments, in particular, resource and time-to-detection statistics. To execute all 800K trials (100K files $\times$ 8 tools), a software framework is designed to choreographed the experiment into a completely automated, time-synced, and reproducible workflow with substantial parallelization. A cost-benefit model was configured to integrate the tools' recall, precision, time to detection, and resource requirements into a single comparable quantity by simulating costs of use. This provides a ranking methodology for cyber competitions and a lens through which to reason about the varied statistical viewpoints of the results. These statistical and cost-model results provide insights on state of commercial malware detection.

User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (14)
  1. Robert A. Bridges (34 papers)
  2. Brian Weber (26 papers)
  3. Justin M. Beaver (3 papers)
  4. Jared M. Smith (5 papers)
  5. Miki E. Verma (7 papers)
  6. Savannah Norem (3 papers)
  7. Kevin Spakes (2 papers)
  8. Cory Watson (4 papers)
  9. Jeff A. Nichols (3 papers)
  10. Brian Jewell (3 papers)
  11. Chelsey Dunivan Stahl (1 paper)
  12. Kelly M. T. Huffer (3 papers)
  13. T. Sean Oesch (1 paper)
  14. Michael. D. Iannacone (1 paper)

Summary

We haven't generated a summary for this paper yet.