Red-Teaming Competition in AI

Updated 30 July 2025

Red-teaming competition is a structured adversarial exercise that examines the security, robustness, and compliance of AI systems.
Competitions leverage both human expertise and automated techniques to simulate realistic deployment and identify systemic vulnerabilities.
Benchmarks like ART and metrics such as attack success rate provide actionable insights to guide improvements in model safety.

A red-teaming competition is a structured adversarial exercise that systematically probes the security, robustness, compliance, and behavioral boundaries of AI models or agentic systems, commonly employing both human expertise and automated methods. The central objective is to elicit harmful, unauthorized, or unexpected behaviors under controlled, yet realistic, deployment scenarios, reveal systemic vulnerabilities, and rigorously test or benchmark safety claims prior to real-world deployment. Recent competitions have expanded in scale, scope, and methodological sophistication, reflecting the emergent complexity and risks of LLM-powered agentic systems (Zou et al., 28 Jul 2025).

1. Foundational Definitions and Historical Context

Red teaming in AI and cybersecurity derives from military and intelligence practices, where designated adversaries were tasked with actively challenging operational readiness or system integrity (Majumdar et al., 7 Jul 2025). In the AI domain, early efforts focused on model-level “jailbreak” or adversarial testing—deliberately crafting queries to bypass safeguards and expose harmful behavior. However, there is a current trend toward a broader systems-focused view, examining sociotechnical contexts, agent architectures, and the full lifecycle from inception through deployment and retirement (Majumdar et al., 7 Jul 2025, Wang et al., 30 May 2025). Competitions catalyze the maturation of red teaming from local “model attack” events to scalable evaluations of agentic, multi-agent, and system-level deployments.

2. Competition Structure and Methodology

A prototypical red-teaming competition, as exemplified by the Gray Swan Arena event (Zou et al., 28 Jul 2025), includes:

Participants: Hundreds to thousands of red teamers, often organized into waves, leveraging both individual creativity and mass action.
Targets: Multiple “frontier” AI agents, typically LLM-driven, instrumented with simulated or real tools, memory, and web access, deployed in sandboxed environments. In (Zou et al., 28 Jul 2025), 22 agents were benchmarked.
Scenarios: Realistic deployment contexts encoding operational constraints, policy documents, data access controls, regulatory requirements, and tool integrations (such as API endpoints for code, finance, or communication).
Attack Surface: Adversarial strategies range from prompt-injection and context manipulation (both direct and via third-party data) to nuanced, multi-turn dialogue attacks and indirect system exploitations.
Data Capture: Automated recording of interactions, iterative feedback on attack success (e.g., policy violation flagging), and dynamic scoreboards monitoring violation rates and attack impact.

The competition usually proceeds in staged “waves,” each unveiling new agent capabilities or task domains to stress-test ongoing defenses (Zou et al., 28 Jul 2025).

3. Behavioral Taxonomies and Attack Modalities

Behavioral categories operationalized in competition scenarios often include:

Behavior Category	Example Violation Types	Measurement/Indicator
Confidentiality Breach	Leaking private/sensitive data	Attack Success Rate (ASR)
Conflicting Objectives	Overriding safety policies for adversarial goals	ASR, scenario-specific
Prohibited Content/Info	Generating illicit, toxic, or regulated content	LLM-based or rule-based
Prohibited Actions	Unsafe or unauthorized API/sim tool usage	Observed agent behavior

Participants execute prompt-injection at scale (1.8 million attacks in (Zou et al., 28 Jul 2025)), encompassing both direct manipulations (tag or system prompt overrides) and indirect vectors (planting adversarial payloads in third-party data, reflective attacks via tools or plugins).

Multi-turn, iterative probing is essential: agents are susceptible to attacks building across conversation states. The ART benchmark (Zou et al., 28 Jul 2025) curates successful cases for both direct and indirect attacks, serving as both an evaluative corpus and a practical attack generator.

4. Results, Metrics, and Benchmarking

Critical findings from large-scale red-teaming competitions include:

Policy Violation Prevalence: Nearly all agents, regardless of size, capability, or inference resources, exhibited high violation rates—often approaching 100% with sufficiently persistent and adaptive attack attempts, typically within 10–100 queries (Zou et al., 28 Jul 2025).
Transferability: Successful attack strategies procured high transferability: attacks devised for one model were often effective on others (including models from distinct organizations). High cosine similarity in adversarial embedding space (cos θ > 0.9) signaled universal weaknesses.
Lack of Robustness Correlation: There was limited or no correlation between agent robustness and model size, training data scale, or compute budget, undermining the assumption that larger or more powerful models are necessarily more robust.
Benchmarks Established: The ART benchmark (Zou et al., 28 Jul 2025) contains thousands of filtered adversarial samples, acting as a gold standard for evaluating new agentic defenses and promoting reproducible, evolving adversarial assessment.

Attack Success Rate (ASR):

Let $n_{succ}$ denote the number of policy-violating sessions, and $n_{total}$ the total number of sessions in a scenario, then:

$\text{ASR} = \frac{n_{succ}}{n_{total}}$

This is the primary evaluation metric reported in the literature and enables cross-model, cross-task comparisons.

5. Implications for Agent Robustness and Security Research

The persistence and universality of agent vulnerabilities revealed in competition settings (Zou et al., 28 Jul 2025) prompt several actionable insights:

Defenses Must Move Beyond Model Scaling: Model scaling alone does not yield meaningful advances in adversarial robustness.
Necessity of Layered and Adaptive Defenses: Defenses must include runtime anomaly detection, policy enforcement at multiple system boundaries, and dynamic response mechanisms capable of handling iterative, context-aware attacks.
Value of Curated Benchmarks: The ART benchmark provides the grounds for standardized, challenge-driven adversarial evaluation and iterative model improvement, addressing the “moving target” problem in LLM safety.

6. Systemic and Socio-Technical Considerations

Recent analyses (Majumdar et al., 7 Jul 2025, Gillespie et al., 12 Dec 2024) emphasize the importance of pairing model-level adversarial evaluation with examination of the deployment environment, human-in-the-loop factors, disclosure practices, and behavioral drift. Competitions demonstrate that emergent risks cannot be isolated to technical artifacts, but are constructed at the intersection of system integration, user behavior, policy specification, and adversarial creativity.

Bridging technical and organizational boundaries, and nurturing multi-functional (engineering, legal, ethical, sociological) red teams, is identified as essential for realistic, impactful competitions.

7. Recommendations and Future Directions

The primary recommendations for future red-teaming competitions are:

Design challenges around full-system deployments including interfaces, tool use, user interaction, and monitoring—replicating the breadth of real-world operational conditions (Wang et al., 30 May 2025, Majumdar et al., 7 Jul 2025).
Incorporate benchmarks like ART to ensure repeatable, transferable evaluation (Zou et al., 28 Jul 2025).
Establish scenario-specific safety requirements rather than relying solely on generic bias/harm definitions, aligning with practical deployment risk profiles (Wang et al., 30 May 2025).
Promote adaptive adversarial and defensive methodologies to avoid overfitting to known attack vectors and capture emergent, cross-model vulnerabilities.
Integrate coordinated disclosure, labor protections for red teamers, and interdisciplinary oversight to address both technical and social dimensions of AI safety (Gillespie et al., 12 Dec 2024, Majumdar et al., 7 Jul 2025).

The ongoing evolution of red-teaming competitions reflects the co-evolution of adversarial methods and defense strategies, and their design serves as a proxy for how the research community approaches the security, safety, and trustworthiness challenges of deploying complex AI agents at scale.