---
title: 'AutoBnB-RAG: Retrieval-Augmented Incident Simulation'
url: https://www.emergentmind.com/topics/autobnb-rag
type: topic
---

# AutoBnB-RAG: Retrieval-Augmented Incident Simulation

Searching arXiv for the main paper and closely related RAG-adaptation papers to ground the article with current references.
AutoBnB-RAG is a multi-agent incident response simulation framework that extends AutoBnB by incorporating retrieval-augmented generation into the Backdoors & Breaches tabletop environment. It treats retrieval as a first-class capability during collaborative investigations: after failed or inconclusive procedure attempts, the incident captain can issue a concise query, a retrieval agent returns external evidence, and defenders use that material to guide subsequent procedure selection. The framework defines two retrieval settings—RAG-Wiki, grounded in curated technical documentation, and RAG-News, grounded in synthetic narrative-style incident reports—and evaluates them across eight organizational structures, including newly introduced argumentative teams intended to promote constructive critique and reduce groupthink [2508.13118].

## 1. Research setting and motivation

Incident response in this formulation is time-critical, multi-phased, and conducted under uncertainty. The motivating premise is that human-only tabletop exercises can be slow or inconsistent, whereas multi-agent LLM systems offer natural language understanding, planning, and communication capabilities that are suitable for simulating or supporting incident response teams responsible for selecting investigative procedures, sharing findings, and converging on action [2508.13118].

The central deficiency addressed by AutoBnB-RAG is that LLM-only incident response agents frequently hallucinate and often lack domain-specific or up-to-date threat knowledge. Retrieval augmentation is introduced as a grounding mechanism that mirrors actual responder practice: when investigations stall, responders consult wikis, MITRE ATT&CK entries, vendor blogs, and case reports. In this setting, retrieval is not an always-on context injector; it is activated when procedure failure or uncertainty indicates that internal reasoning is insufficient.

AutoBnB-RAG therefore occupies a specific point in the design space of LLM-based cyber operations research. It is not framed as a generic RAG benchmark, and it is not limited to single-agent decision support. Its core object of study is collaborative incident reconstruction under constrained turns, hidden adversary state, stochastic procedure resolution, and evolving shared context.

## 2. Architecture and tabletop instantiation

The system comprises four principal components. The **incident captain** sets the scenario by choosing hidden attack cards, orchestrates turns, prompts post-failure retrieval, crafts retrieval queries, and disseminates retrieved evidence. The **defender agents** are five LLM agents whose composition varies by team structure. The **retrieval agent** is a non-discursive utility agent integrated into the communication graph; it processes retrieval function calls and returns top-$k$ passages but does not otherwise participate in dialogue. The **group chat manager** coordinates speaker turns and message flow [2508.13118].

The environment is an instantiation of Backdoors & Breaches with four hidden attack stages: **Initial Compromise**, **Pivot and Escalate**, **C2 and Exfiltration**, and **Persistence**. The corresponding card sets span 13, 12, 7, and 14 cards respectively, together with 12 procedure cards; inject and consultant cards are omitted for control. The game runs for 10 turns. On each turn, defenders discuss and select one procedure card. Resolution uses a $d20$ roll, with success if the final roll is $\ge 11$; procedures marked **Established** receive a $+3$ modifier. A successful procedure reveals one matching unrevealed attack card, while failure reveals nothing. Victory requires revealing all four attack cards within 10 turns [2508.13118].

The turn-level loop is explicitly procedural. Hidden scenario generation occurs first. Defenders then discuss and propose a procedure, after which the captain finalizes the selection in centralized teams, or the group decides collectively in decentralized teams. The die roll determines success or failure. On success, one attack card is revealed and its implications are summarized. On failure, the captain constructs a retrieval query from scenario, procedure, or evidence keywords, invokes the retrieval agent, receives top-$k$ passages, and injects them into the group chat. Defenders then revise their reasoning in light of the external material, and the cycle repeats.

This architecture assigns retrieval control to the captain rather than distributing it symmetrically across all agents. That choice is consequential because it couples knowledge acquisition to an explicit coordination role and makes evidence dissemination part of command-and-control, not merely background augmentation.

## 3. Retrieval substrates and grounding mechanism

AutoBnB-RAG defines two complementary corpora. **RAG-Wiki** consists of 125 curated webpages focused on technical explanations, threat models, and defensive strategies aligned to Backdoors & Breaches cards. Its source distribution is: Wikipedia 67 (53.6%), MITRE ATT&CK 9 (7.2%), Microsoft Learn/Support 6 (4.8%), CISA/Gov 3 (2.4%), cybersecurity blogs/vendors 27 (21.6%), and other 13 (10.4%). **RAG-News** consists of 100 synthetic incident reports generated from a structured prompt to cover diverse attack/procedure combinations while minimizing overlap with evaluation scenarios; manual spot checks were performed for narrative quality and realistic investigative logic [2508.13118].

Indexing and retrieval use recursive character-based splitting in LangChain with overlapping chunks of 5,000 characters and 500-character overlap in the default simulations. Chroma serves as the vector database for dense retrieval. The default retrieval depth is top-3 passages per query. In the realistic breach reconstruction setting, the configuration shifts to 1,000-character chunks with overlap, again returning top-3 passages. The specific embedding model and any re-ranking method are not specified, and no BM25, tf-idf, or formal scoring equations are reported [2508.13118].

Grounding is implemented through post-failure narrative insertion. Retrieved passages are returned silently by the retrieval agent, then shared by the incident captain in the group chat so that defenders can use them in subsequent reasoning. Prompt templates and context window sizes are not detailed. The paper’s emphasis is therefore on the operational role of retrieval rather than on prompt engineering minutiae or retriever-stack optimization.

The two corpora support different forms of grounding. RAG-Wiki emphasizes procedural and technical factuality. RAG-News emphasizes multi-stage narrative coherence. This suggests that retrieval utility in incident response is not exhausted by canonical documentation; incident reports can function as structured analogical priors for sequencing investigative steps, provided they are injected at the right decision points.

## 4. Team structures and argumentative reasoning

The framework varies organizational form along two axes: **leadership** and **expertise**. Leadership modes are **centralized**, **decentralized**, and **hierarchical**. Expertise composition is either **homogeneous generalists** or **heterogeneous domain experts**. The heterogeneous role set comprises endpoint, network traffic analysis, logs/behavior, deception/containment, and incident response [2508.13118].

Beyond the six original structures formed by these axes, AutoBnB-RAG introduces two argumentative configurations. In the **homogeneous argumentative** structure, all generalists adopt constructive critique roles intended to reduce groupthink. In the **heterogeneous argumentative** structure, each domain expert adopts an argumentative variant of the corresponding role: argumentative endpoint security expert, argumentative network traffic analysis expert, argumentative log and behavioral analysis expert, argumentative deception and containment expert, and argumentative incident response expert. These agents explicitly question peer suggestions, propose alternatives, and probe the robustness of plans [2508.13118].

Coordination remains structure-dependent. In centralized teams, a leader integrates inputs and decides the procedure. In decentralized teams, consensus emerges through dialogue, with the group chat manager sequencing contributions. In hierarchical teams, senior agents guide and mentor juniors. Argumentative variants alter the style of deliberation rather than the game mechanics: critiques must still resolve into a single procedure choice per turn.

The significance of the argumentative extension lies in its interaction with evidence. The paper reports that argumentative roles foster critical review by forcing evidence use, including retrieved context, before commitment to procedures. A plausible implication is that retrieval becomes more valuable when team process is organized to contest unsupported hypotheses rather than merely accumulate suggestions.

## 5. Evaluation protocol, quantitative results, and ablations

Experiments use AutoGen for multi-agent orchestration and GPT-4o as the base LLM with temperature 0.7. Eight team structures are evaluated under three conditions: **Base** (AutoBnB without retrieval), **RAG-Wiki**, and **RAG-News**. For consistency, each team structure under each condition is run in 30 independent simulations. The primary metric is **win rate**, defined as the percentage of runs that achieve full reconstruction within 10 turns. No statistical significance tests are reported; results are presented as win-rate averages [2508.13118].

| Team structure | Base | RAG-Wiki / RAG-News |
|---|---:|---:|
| Homogeneous centralized | 20.0 | 50.0 / 60.0 |
| Heterogeneous centralized | 30.0 | 43.3 / 63.3 |
| Homogeneous decentralized | 33.3 | 40.0 / 43.3 |
| Heterogeneous decentralized | 26.7 | 50.0 / 50.0 |
| Homogeneous hierarchical | 23.3 | 40.0 / 43.3 |
| Heterogeneous hierarchical | 30.0 | 36.7 / 70.0 |
| Homogeneous argumentative | 23.3 | 43.3 / 46.7 |
| Heterogeneous argumentative | 30.0 | 46.7 / 53.3 |

Several empirical patterns are explicit. Retrieval consistently improves success, particularly for centralized and hierarchical teams. RAG-News often yields larger gains than RAG-Wiki, which the paper attributes likely to procedural or narrative fit with the game. Argumentative teams also benefit substantially, indicating synergy between critical review and external grounding [2508.13118].

Ablations were conducted on retrieval depth and chunk size using the homogeneous centralized team. For retrieval depth, RAG-Wiki yielded win rates of 46.7 at top-1, 50.0 at top-3, and 46.7 at top-5; RAG-News yielded 60.0 at top-1, 60.0 at top-3, and 63.3 at top-5. The paper interprets this as performance stability across $k$, with small $k$ often sufficient and larger $k$ potentially introducing noise. For chunk size, RAG-Wiki improved from 33.3 with 1k chunks to 50.0 with 5k chunks, whereas RAG-News was 63.3 with 1k chunks and 60.0 with 5k chunks. The reported interpretation is that larger chunks help preserve context for technical documentation, while narrative material remains coherent even with smaller chunks [2508.13118].

The qualitative error analysis identifies four recurrent issues: retrieval misses from vague queries, residual hallucinations when irrelevant passages are injected, groupthink in non-argumentative teams, and over-retrieval increasing cognitive load. The paper states that argumentative teams help reframe queries, retrieved context reduces hallucinations, and top-3 often strikes a balance between informativeness and overload.

## 6. Realistic breach reconstructions

Beyond synthetic tabletop play, AutoBnB-RAG is validated on public-breach-based reconstructions of three real-world incidents: **North Face credential stuffing**, **Cock.li Roundcube exploit**, and **Gluestack NPM supply-chain compromise**. The stated purpose is to test whether retrieval-backed multi-agent play can reconstruct complex multi-stage attacks under more realistic conditions [2508.13118].

The most detailed case is the **Gluestack NPM supply-chain attack** from June 2025, using a homogeneous centralized team and completing in six turns. The attack mapping is: **Supply Chain Attack** for Initial Compromise, **Weaponizing Active Directory** for Pivot, **Gmail/Tumblr/Salesforce/Twitter** as C2/Exfiltration, and **Malware Injection Into Client Software** for Persistence. Turn 1 uses SIEM Log Analysis with a roll of 9 and a +3 modifier to reveal Weaponizing Active Directory. Turn 2 fails on Endpoint Analysis and triggers retrieval. Turn 3 uses Endpoint Security Protection Analysis to reveal Malware Injection Into Client Software. Turn 4 uses Network Threat Hunting to reveal Supply Chain Attack. Turn 5 fails on Firewall Log Review and again triggers retrieval, this time returning intelligence on third-party service C2 patterns. Turn 6 uses Network Threat Hunting to reveal the multi-platform C2 stage. The reported outcome is full reconstruction in six turns, with retrieval described as strategically steering the team after failures [2508.13118].

Two additional reconstructions are summarized more compactly. In the **North Face credential stuffing** case, the team finishes in eight turns, with retrieval used in 4 of 8 turns; UEBA reveals internal password spray, SIEM failure prompts retrieval, Server Analysis confirms credential stuffing, and later turns expose HTTPS exfiltration and persistence via new user creation. In the **Cock.li Roundcube exploit** case, the team uses all 10 turns, with retrieval in 6 of 10 turns; Server Analysis reveals web server compromise, and later procedures uncover local privilege escalation, HTTP exfiltration, and registry-based persistence [2508.13118].

These reconstructions matter because they shift the framework from pure game optimization to evidentiary plausibility. The paper does not claim operational deployment, but it does show that retrieval-backed multi-agent dialogue can be mapped onto recognizable attack narratives rather than only abstract card states.

## 7. Limitations, deployment guidance, and relation to adjacent RAG research

The paper identifies several threats to validity. Results depend on GPT-4o, and generalization to other LLMs is not assessed. RAG-Wiki is curated but limited to 125 pages and may omit niche threats. RAG-News is synthetic and may bias reasoning toward narrative patterns reflected in its generation template. The embedding model and re-ranking stack are unspecified. No formal significance tests are reported. Robustness against noisy or poisoned retrieval corpora is unknown. Multi-agent GPT-4o simulations incur compute and cost overhead. The authors also caution that synthetic narratives should not be conflated with real incidents and that mixing public breach reports with generated content requires care to avoid misinterpretation [2508.13118].

The implementation and deployment guidance is correspondingly pragmatic. Retrieval should be triggered after investigative failures or uncertainty, with concise queries anchored to observed signals such as logs, endpoints, or traffic anomalies. For technical corpora, larger chunks around 5k characters preserved context and improved win rate in the reported setup; for narrative corpora, 1k to 5k both worked. The suggested starting point for retrieval depth is $k=3$. Centralized and hierarchical teams benefited most from retrieval, so the paper recommends a strong captain who can time retrieval and synthesize evidence. To mitigate hallucinations, defenders can be required to cite retrieved passages before procedure selection, actions not grounded in retrieved or in-game evidence can be penalized, and irrelevant retrieval should trigger query reformulation. Operationally, the paper recommends curated, versioned technical corpora aligned with local EDR, SIEM, and ATT&CK mappings, together with rotating breach reports and internal IR playbooks; monitoring should include retrieval usage per turn, success rate, time-to-containment, retrieval frequency, and post-retrieval reveal rate [2508.13118].

Within the broader RAG literature, AutoBnB-RAG is adjacent to, but distinct from, several lines of work. **AutoRAGTuner** automates construction, execution, evaluation, and optimization of RAG pipelines through a declarative JSON orchestration layer, a modular component registration mechanism, the Domain-Element Model, and adaptive Bayesian optimization [2605.02967]. **RAGRouter-Bench** frames adaptive RAG routing as context-dependent paradigm selection over query–corpus pairs and evaluates LLM-only, NaiveRAG, GraphRAG, HybridRAG, and IterativeRAG under unified quality and cost metrics [2602.00296]. **“Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup”** studies a boolean gate that decides whether retrieval is needed after a baseline answer, emphasizing cost-aware retrieval triggering rather than collaborative multi-agent incident simulation [2403.00820]. **MBA-RAG** instead treats retrieval strategies as arms in a contextual $\epsilon$-greedy multi-armed bandit, optimizing a reward that balances correctness and retrieval steps [2412.01572]. **RAGONITE** combines SQL-query results over an induced database with dense retrieval over verbalized RDF facts and supports iterative retrieval for conversational QA over knowledge graphs [2412.17690]. **Structured RAG for Answering Aggregative Questions** constructs a structured corpus representation at ingestion time and translates natural-language questions into formal queries to support counting, ranking, grouping, and related aggregative operations [2511.08505].

Taken together, these neighboring systems situate AutoBnB-RAG within a wider transition from monolithic retrieve-and-generate pipelines toward adaptive, structured, and workflow-sensitive retrieval. A plausible implication is that future incident response systems could combine AutoBnB-RAG’s collaborative evidence-grounded reasoning with declarative tuning, adaptive routing, or explicit retrieval gating, but the present paper evaluates only the retrieval-backed multi-agent framework described above.

Source: https://www.emergentmind.com/topics/autobnb-rag