Co-RedTeam: Collaborative AI Red Teaming

• Co-RedTeam is an integrated red-teaming framework uniting cybersecurity and AI methodologies to systematically evaluate adversarial vulnerabilities in AI-infused systems.
• It employs multi-agent orchestration and iterative feedback loops to uncover, validate, and mitigate risks across models, data pipelines, and socio-technical interfaces.
• Empirical studies demonstrate that its coordinated, game-theoretic approach can boost exploitation success rates by up to 65%, thus significantly enhancing system security.

Co-RedTeam refers to the orchestrated, collaborative, and often multi-agent red-teaming of AI-enabled systems, unifying methodologies and tooling from both cybersecurity and AI red-teaming traditions. It encompasses the design, execution, sharing, and continuous refinement of adversarial emulation workflows targeting both AI components and the broader socio-technical system. Co-RedTeam frameworks leverage formal threat models, multi-stage adversary simulation, rigorous lifecycle management, and diverse human and agent-based participation to systematically uncover, quantify, and address vulnerabilities and risks specific to AI-infused infrastructure (Sinha et al., 14 Sep 2025, &&&1&&&, He et al., 2 Feb 2026).

1. Foundational Concepts and Definitions

Co-RedTeam is defined as a domain-integrated red-team engagement paradigm that transcends the siloed treatment of “cyber red teaming” and “AI red teaming,” instead performing end-to-end adversary emulation against the complete attack surface—models, software, data, infra, and human interfaces. Key elements include (Sinha et al., 14 Sep 2025):

• Structured Lifecycle: Borrowed from cyber operations, Co-RedTeam enforces engagement rulesets, scoping, exploitation chains, comprehensive reporting, and post-engagement review.
• AI-Specific Threat Modeling: Targeting vulnerabilities unique to machine learning systems such as adversarial examples, model extraction, prompt injection, data poisoning, membership inference, and emergent behavior modes.
• Holistic Scope: Extends assessments beyond models to include APIs, data pipelines, deployment infra, interfaces, and socio-technical touchpoints, encompassing both technical and non-technical risks.
• Collaborative, Multi-Stakeholder Workflows: Incorporation of internal and external domain experts, ethical hackers, sector specialists, and, increasingly, multi-agent AI systems (Ahmad et al., 24 Jan 2025, He et al., 2 Feb 2026).
• Responsible Disclosure and Mitigation: Formal frameworks for coordinated vulnerability disclosure (CVD), particularly vital for unpatchable or systemic AI flaws.

Co-RedTeam is not a simple combination of cybersecurity pentesting and AI audit, but a singular discipline for the modern AI-enabled threat landscape (Sinha et al., 14 Sep 2025).

2. Methodological Frameworks and Architectures

Several concrete frameworks and systematizations exemplify Co-RedTeam, spanning collaborative human workflows, agent-based orchestration, and cross-organizational intelligence sharing.

a) Multi-Agent and Automated Orchestration

The “Co-RedTeam” framework for LLM-driven cybersecurity tasks structures vulnerability analysis as a two-stage process—static discovery and dynamic exploitation—using a manager (orchestrator) and a collection of specialized agents (He et al., 2 Feb 2026):

Stage	Agents	Functionality
Discovery (Stage I)	Analysis, Critique	Enumerate vulnerability hypotheses and subject them to risk review
Exploitation (Stage II)	Planner, Validation, Execution, Evaluation	Construct, refine, and test exploit chains via execution feedback
Memory System	Shared long-term memory	Retain and retrieve patterns, strategies, and low-level actions for transfer learning

Task flow is execution-grounded and iterative, with agents refining hypotheses and attack plans over repeated cycles informed by real execution traces and memory retrieval.

b) Cooperative External Red Teaming

In practice, Co-RedTeam incorporates human experts and external organizations, leveraging:

• Team Diversity: Recruiting domain-specific red teamers, security professionals, and non-technical experts to probe for a broad class of risks (e.g., legal, medical, cultural) (Ahmad et al., 24 Jan 2025).
• Guided and Open-Ended Interaction: Structured prompt templates and open exploration are both utilized, with outputs feeding directly into regression test suites and model evaluations.
• Feedback Loops: Iterative cycles of adversarial input discovery, triage, mitigation recommendation, and retesting.
• Operational Safeguards: Clear access control, documentation protocols, and adversity/mitigation logging ensure responsible handling of information hazards and zero-day risks.

c) Cross-Organizational Intelligence Generation and Sharing

Frameworks like CTI4AI implement full red-team-to-intelligence-sharing pipelines:

• Red Team Engine: Automated tools (e.g., ART toolkit) generate structured adversarial scenarios.
• Threat Intelligence Encoder (TIE): Converts findings into shareable intelligence artifacts (AITI objects: AI-specific extension of STIX).
• TAXII/MISP Integration: Secure APIs for multi-stakeholder push/pull and federated CTI (Cyber Threat Intelligence) dissemination (Nguyen et al., 2022).

3. Formal Problem Structures and Quantitative Metrics

Co-RedTeam leverages problem-structuring methods from both security analysis and computational game theory, supporting rigorous measurement.

a) Multi-Agent, Multi-Round Red-Teaming Game

The Dynamic Red Team Game (RTG) formalism structures LLM red-teaming as a team extensive-form adversarial game, spanning both token-level MDP for generation and sentence-level dialogue games. Objectives include maximizing exploitability or robustness in multi-round interaction (Ma et al., 2023):

$\mathcal{G} = (\mathcal{T}, \mathcal{D})$

with population-based Policy-Space Response Oracles (PSRO) converging to $\epsilon$-Nash equilibria.

b) Security Analysis with Feedback and Memory

Formally, for codebase $C$, execution environment $E$, candidate hypotheses $H$, plan $P$, and memory $M$:

• Discovery: $H^0 = D(C, M)$; $H^* = R(H^0, C, M)$
• Iterative Exploitation: Plan and refine actions $P^t$, update using validation, execution ($r^t$), and evaluation feedback ($f^t$)
• Learning: After a successful exploit, extract and aggregate into structured, reusable memory for transfer across environments (He et al., 2 Feb 2026).

c) Risk and Coverage Metrics

Co-RedTeam employs:

• Quantitative Risk Score: $$R(v) = \mathrm{Impact}(v) \times \mathrm{Likelihood}(v)$$
• Coverage: $$\alpha = |V_\text{found} \cap V_\text{model}| / |V_\text{model}|$$
• Test/Exposure Metrics: $$R = \frac{\text{successful attacks}}{\text{total tests}}$$
• Severity-Weighted Exposure: $S = \frac{\sum_i w_i s_i}{\sum_i w_i}$
• Vulnerability Reduction: $$\mathit{VR} = 1 - \frac{R_\text{post}}{R_\text{pre}}$$; higher values indicate mitigated risk
• Exploitability (RTG): $\mathrm{Expl}(\sigma)$ for iterative equilibrium assessment

These operationalize progress, resilience, and the effectiveness of both manual and automated adversarial testing (Sinha et al., 14 Sep 2025, Ahmad et al., 24 Jan 2025, He et al., 2 Feb 2026, Ma et al., 2023).

4. System Architectures, Tooling, and Knowledge Sharing

Advances in systemization align Co-RedTeam with principles of reproducibility, transparency, and broad applicability.

a) Modular, Multi-Agent Architectures

Agent-based orchestration, such as that implemented in Co-RedTeam (LLM agents), involves explicit roles—analysis, planning, validation, execution, evaluation—teamed under an orchestrator. Long-term memory stores patterns and trajectories, supporting continual learning (He et al., 2 Feb 2026).

Ablation studies demonstrate the necessity of:

• Execution feedback: Removes causes catastrophic decline in successful exploitation rates (−40–47%)
• Long-term memory: Dramatic drop in performance when omitted, confirming crucial role in generalization and cumulative learning
• Critique, code-browsing, validation: Each component delivers measurable improvement in detection/exploitation accuracy versus baselines

b) Threat Intelligence Standards and Data Models

CTI4AI illustrates a pipeline with:

Component	Role	Example Implementation
Red Team Engine	Vulnerability discovery via ART/fuzzing	DARPA GARD ART toolkit
Threat Intelligence Encoder (TIE)	Encodes into AITI objects (AI-aware STIX superset)	Maps attack/attack-pattern/user/paradigm
Sharing Platform	Secure CTI distribution	TAXII API, MISP RESTful collections

Standardization (STIX, TAXII) and digital signatures enable composability and trust across organizations and federated environments (Nguyen et al., 2022).

5. Practical Impacts, Case Studies, and Limitations

Empirical Studies:

• Co-RedTeam with LLM agent orchestration outperforms strong baselines (vanilla prompting, code agents) on CyBench, BountyBench, CyberGym (He et al., 2 Feb 2026). Performance metrics:
  • Exploitation success rates up to 65%
  • Detection precision/recall up an order of magnitude over prior art
  • Significant gains attributable to iteration, execution, critique agents, and memory
• Game-theoretic approaches (GRTS for RTG) yield scalable, diverse, multi-modal attack discovery, exposing mode collapse in static red-teaming and establishing robust Nash-equilibrium defense policies (Ma et al., 2023).
• Human-in-the-loop Co-RedTeam (OpenAI) has led to the discovery and mitigation of “voice mimic” and “visual synonym” attack vectors in flagship models (e.g., GPT-4o, DALL-E 3), integrating findings into regression test stimuli and evaluation sets (Ahmad et al., 24 Jan 2025).

Limitations and Future Directions:

• Temporal drift and resource intensity continue to hinder repeatable human-in-the-loop campaigns; integration with automated tooling is necessary for sustainability (Ahmad et al., 24 Jan 2025).
• The risk scoring and taxonomy frameworks, while formalized in some CTI4AI-style systems, remain a work in progress—especially for multi-stage attack graphs and fully automated risk quantification (Nguyen et al., 2022).
• Unpatchable AI vulnerabilities necessitate long-horizon, coordinated disclosure regimes and defense-in-depth approaches, including layering, input validation, ensemble mitigations, and socio-technical governance (Sinha et al., 14 Sep 2025).
• Current agent-based pipelines are limited by the capabilities of backbone LLMs; performance, convergence rates, and memory transferability scale nonlinearly with model capability (He et al., 2 Feb 2026).

6. Generalization: Diversity, Game Theory, and Scaling

Theoretical and empirical evidence supports the importance of population diversity, iteration, and game-theoretic framing in scalable Co-RedTeam.

• Population Diversity and Coverage: Dynamic RTG with GRTS avoids mode collapse, ensuring a wide spectrum of attack and defense strategies (multi-modal LLM red-teaming mirrors the heterogeneity of human red-teamer populations) (Ma et al., 2023).
• Convergence Guarantees: Exploitability under GRTS converges to $\epsilon$-Nash at $O(1/T)$; iterative multi-agent pipelines saturate performance after sufficient refinement loops.
• Hybrid Human-AI Integration: Human oversight (oracle policies, ranking, annotation) closes gaps left by fully automated systems, allowing grounded validation of emergent, previously unseen attacks (Ahmad et al., 24 Jan 2025, Ma et al., 2023).
• Cooperative Red-Team Extensions: Integration of automated red-team ensembles and cross-organization platforms (federated TAXII, feedback dashboards, sighting feeds) enables real-time, distributed, mutually reinforcing adversarial testing and threat sharing (Nguyen et al., 2022).

A plausible implication is that as AI models become more complex and take on greater decision-making responsibility, scalable Co-RedTeam practices blending automation, agent collaboration, human expertise, and standardized intelligence sharing will become central not only to system assurance, but to the overall scientific and operational governance of AI deployment.