Automated Detection Engineer (ADE)

• Automated Detection Engineer (ADE) is an autonomous system that generates, validates, and refines security detection rules using multi-agent strategies and LLMs.
• It employs a closed-loop process that benchmarks ADE-generated rules against human-crafted standards using metrics like detection accuracy, economic cost, and robustness.
• ADE’s iterative workflow ensures deployable, auditable, and economically efficient security logic, making it highly effective for rapid threat detection.

An Automated Detection Engineer (ADE) is an agentic system designed to autonomously generate, validate, and iteratively refine security detection rules—typically expressed in specialized query languages—for use in enterprise security infrastructures. ADEs leverage advanced LLMs, domain-specific knowledge, and multi-agent orchestrations to convert isolated threat artifacts (such as malicious emails) into production-quality detection logic. Although the concept has broad relevance to detection automation in cybersecurity, the archetypal ADE described in (Bertiger et al., 20 Sep 2025) is notable for its explicit use of open evaluation benchmarks and strict comparison against human-generated security content.

1. Agentic Architecture and Workflow

The ADE operates as a multi-agent system with an integrated knowledge base and a suite of validation tools. Its workflow incorporates the following canonical steps:

• Input Analysis: Upon receipt of a malicious artifact (e.g., a phishing email), ADE parses and extracts salient features by analyzing headers, attachments, sender attributes, and heuristic indicators.
• Knowledge Retrieval: The system consults detection engineering best practices, attacker tactics, techniques, and procedures (TTPs), and a corpus of detection rule archetypes from its internal knowledge base.
• Rule Generation: ADE synthesizes candidate Message Query Language (MQL) rules, mapping observed and inferred threat features to structured detection logic.
• Delegated Critique and Threat Hunting: Dedicated subagents evaluate candidate rules, provide critical feedback, and may conduct targeted threat hunts or hypothecation to test generalizability.
• Closed Feedback Loop: Through repeated validation—including syntactic correctness using an MQL validator and empirical evaluation on both the original sample and historical email corpora—the ADE iteratively refines output until it achieves acceptable detection accuracy and economic efficiency thresholds.
• Deployable Artifact Production: The final rule is delivered as a production-ready artifact with extensive annotations and justifications that facilitate downstream deployment and auditing.

This closed-loop, multi-agent process aligns with the requirements for robust and verifiable automation in security operations.

2. Benchmarking and Evaluation Methodology

The effectiveness of ADEs is assessed via a holdout set-based methodology as formalized in (Bertiger et al., 20 Sep 2025):

• Human-Baseline Comparison: Each ADE-generated rule is evaluated against a "gold standard" corpus of human-generated detection rules (e.g., from the Sublime Security Core Feed). In each test, one human rule is removed ("held out"), and ADE is tasked with producing a replacement rule based solely on a true positive message corresponding to the omitted human rule.
• Performance Metrics: The evaluation framework quantifies ADE output using three core metrics:

Metric Category	Measured By	Formula / Criteria
Detection Accuracy	True positives (TP), false positives (FP), unique TPs	Score = ½ × [ (#TP/(#TP+#FP)) + (#unique TP/(#TP+#FP)) ]
Economic Cost	Syntax validation attempts, cost per attempt, pass@1 rate	v(m,p)=Cₘ(p)/Rₘ(p); Total Cost = Cₘ(p) × k
Robustness (Brittleness)	Ratio of non-brittle (reward) vs brittle (penalty) patterns	B=100/(1+e^{k((R/P)-x₀)}); Robustness = 1 – (B/100)

Each metric emphasizes a distinct operational requirement:

• Precision, recall, and the ability to uniquely identify threats without excessive false positives.
• Actual monetary and computational costs accrued during rule synthesis and validation retries.
• Structural robustness, balancing fuzzy pattern generality against rigid, easily-defeated string matches.

3. Key Empirical Results and Profiling

Empirical testing on an expert-labeled corpus of approximately 45,000 emails revealed:

• Detection Scope: ADE-generated rules, crafted from single samples, exhibit narrower applicability than broadly constructed human rules, flagging fewer overall messages and unique true positives.
• Precision and False Positives: ADE rules tend toward lower false positive rates and, in some instances, marginally higher precision than their human counterparts. For the "EML w/ Javascript in SVG File" test case, ADE achieved a precision of 0.998 versus 0.996 for the human rule.
• Rule Structure and Commentary: ADE outputs feature more thorough in-line comments, suggesting an orientation toward explainable, auditable rule production.
• Robustness: Brittleness and robustness metrics are comparable between ADE and human rules, indicating ADE's proficiency in balancing detection coverage with resistance to trivial evasion.

This data suggests that while ADE may initially underperform in total threat coverage (due to sample-driven specificity), it consistently produces rules that are precise, robust to evasion, and operationally efficient. A plausible implication is that ADE excels in situations where rapid, sample-driven rule synthesis with high precision is needed, while broader coverage may require ensembling or more diverse training data.

4. Economic and Operational Efficiency

An important design constraint is the minimization of economic, computational, and operational overhead:

• Validation Pass and Cost Management: Each generated rule must pass a syntax validator before further use. Validation incurs tangible resource costs, captured by the cost-to-pass metric. ADE integrates retry loops and economic thresholds, ensuring that only cost-effective rules progress to empirical evaluation.
• Resource Transparency: The open-source evaluation pipeline outputs resource use at every step, making cost and throughput explicit and reproducible for practitioners scaling ADE-based systems.

5. Practitioner Integration and Open-Source Benchmarking

The ADE evaluation protocol is released as an open-source framework. Practitioners may:

• Substitute their own detection rule generators (LLM or otherwise), in any domain-specific language (e.g., MQL, YARA), and replicate the entire holdout benchmarking process.
• Adjust detection, cost, and robustness parameters to match organizational requirements or regulatory constraints.
• Iteratively enhance ADE rulesets through cycles of empirical feedback and modular subagent orchestration, ensuring both performance and resilience.

This systematic benchmarking infrastructure is intended to foster continuous performance enhancement and to facilitate quantitative comparisons between automated and manual detection engineering efforts.

6. Distinctions and Limitations Relative to Human Rule Writing

The analysis demonstrates:

• Scope and Generalization: ADE's tendency to generate narrowly scoped rules is a direct result of its prompt design (single sample input). Broader generalization, typical of the human detection engineering process, is not natively captured but may arise by training or ensemble strategies.
• Explainability and Documentation: Enhanced commentary in ADE output may aid auditability and ongoing maintainability.
• Iterative Feedback Integration: The ADE architectural bias toward feedback and iterative refinement closes the loop on both syntactic and semantic validation, arguably better formalizing the empirical and review-driven processes long found in human detection teams.

A plausible implication is that combinations of ADE- and human-generated rules may offer optimal trade-offs between precision, recall, and explainability in production settings.

7. Summary and Relevance to Detection Automation

In sum, the Automated Detection Engineer operationalizes agentic, LLM-based rule generation for security automation, demonstrating measurable efficacy and efficiency when objectively benchmarked against human-crafted rule corpora (Bertiger et al., 20 Sep 2025). Its design principles—multi-agent orchestration, knowledge-guided inference, closed-loop feedback, and openly benchmarked performance—establish both best practices for evaluation and clear avenues for further research and integration within operational cybersecurity workflows. The provided open-source framework ensures reproducibility and extensibility for ongoing deployments and research-driven enhancements.