Red-Team MARL: Adversarial Multi-Agent Strategies

• Red-team multi-agent reinforcement learning is a domain where adversarial teams are trained to expose rare system failures and induce hazardous states in competitive or cooperative settings.
• Key methodologies include centralized training with decentralized execution, explicit opponent modeling, and coordinated deception to manage partial observability and non-stationarity.
• Empirical studies show that red-team formulations, such as in autonomous driving and cyber defense, dramatically increase failure detection rates and improve system robustness.

Searching arXiv for recent and directly relevant work on red-team multi-agent reinforcement learning and adjacent adversarial/team-MARL formulations. Red-team multi-agent reinforcement learning denotes a class of multi-agent reinforcement learning formulations in which one or more agents or teams are optimized to expose failure modes, induce hazardous states, or strategically oppose a target policy or opposing team under interactive dynamics. In the recent literature, this includes explicit adversarial background-vehicle teams for autonomous-driving corner-case generation, repeated red/blue language-model games, and dual-policy cyber-defense systems in which an internal Red-Team counterbalances a Blue-Team controller (Chen et al., 21 Jul 2025, Ma et al., 2023, Zhang et al., 6 Apr 2026). The area is not methodologically uniform: some work studies direct adversarial search, some studies team coordination under hostile environments, and some uses a “Red-Team” as an internal conservative counter-policy rather than as an external attacker.

1. Scope and taxonomic boundaries

At its narrowest, red-team MARL refers to settings where a learned adversarial team is trained to discover rare failures of a defended system. The clearest example in the supplied literature is emergency-braking validation, where background vehicles with interference capabilities are treated as red-team agents and optimized to uncover corner cases outside the data distribution (Chen et al., 21 Jul 2025). A broader usage includes dynamic red/blue dialogue games for LLM safety, where red-team LLMs maximize unsafe or toxic blue outputs across multiple rounds, and automated attacker frameworks in which a learned attacker interacts with fixed judges and targets; the latter is directly relevant but is not MARL in the strict sense (Ma et al., 2023, Beutel et al., 2024).

The boundary matters because the literature uses adversariality in several distinct senses. The general MARL background report centers on mixed cooperative and competitive environments, Dec-POMDP-style partial observability, and centralized training with decentralized execution (CTDE) (Kapoor, 2018). Other papers supply structurally relevant ingredients without presenting explicit red teams: room clearance learns a blue team against scripted hostile opposition; team sports studies competitive/cooperative structure, reward shaping, and opponent-specific exploitation without red-teaming; and team regret minimization focuses on within-team cooperation in mixed battle environments rather than explicit opponent modeling or minimax training (Charlesworth et al., 2021, Zhao et al., 2019, Yu et al., 2019).

2. Formal problem settings

Two formalisms dominate the literature. Cooperative teams with decentralized execution are often written as a Dec-POMDP,

$$\langle \mathcal{S},\mathcal{N},\mathcal{A},P,r,\mathcal{O},\gamma,b_1\rangle,$$

with local observations $o^i$, local actions $a^i$, joint action $\mathbf{a}$, and shared reward inside a team (Yu et al., 2019). Team-competitive settings are alternatively written as a Markov game

$$\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R} \rangle,$$

where rewards are per-agent and teams $\mathcal{N}(k)$ may be aligned internally while competing across teams (Koley et al., 2023).

Explicit adversarial planning introduces stronger game structure. Model-based two-agent adversarial control uses separate latent strategy variables and a max-min objective,

$$\max_{Z_x} \min_{Z_y} \mathbb{E}\left[ \sum_{t=1}^{H} r(x_{t},y_{t}) \right],$$

subject to jointly generated future trajectories (Krupnik et al., 2019). Safety-critical red-team driving is formulated as a Constraint Graph Representation Markov Decision Process,

$M_{CG}=(S,A,P,R,\gamma,\Omega,G),$

which augments an MDP with graph structure $G$ and a constraint set $\Omega$ so that red-team vehicles remain traffic-compliant while interfering with the AV (Chen et al., 21 Jul 2025). Red/blue LLM interaction is formalized hierarchically as

$o^i$0

combining a token-level Markov Decision Process for Token Generation with a sentence-level extensive-form team game in dialogue (Ma et al., 2023).

Not all red-team formulations are attacker-versus-defender games. In C-MADF, the Blue and Red policies

$o^i$1

operate inside the same constrained MDP, and the paper is explicit that this is not a classical zero-sum adversarial game: the Red-Team is a conservatively shaped counterfactual policy regularizer rather than a model of the external attacker (Zhang et al., 6 Apr 2026).

3. Methodological patterns

A recurrent design pattern is CTDE. In the general MARL report, CTDE is motivated by local execution under partial observability and centralized critics for coordination and credit assignment (Kapoor, 2018). Team regret minimization applies that pattern to battle-like domains by redefining the “player” as the whole cooperative team and introducing accumulated team regret,

$o^i$2

then recovering decentralized execution through additive or shaping-form decomposition of team regret into per-agent regret quantities. The BVRM and BVRM-shaping variants add differentiable particle-filter belief inference for partially observable team-vs-team battle environments (Yu et al., 2019).

A second pattern is explicit opponent awareness. RAC augments an actor-critic backbone with a self-role encoder $o^i$3, an opponent-role predictor $o^i$4, and role-conditioned policies $o^i$5. Its role objective combines a mutual-information-style identifiability term, a diversity loss within teams, and an opponent-role prediction loss,

$o^i$6

so that agents learn diverse, dynamic roles while conditioning execution on predicted opponent roles (Koley et al., 2023).

A third pattern is coordinated deception or hierarchical command. In multi-agent deception, graph-based MARL adds agent, environment, and opponent state encoders, attention-based same-team message passing, and a two-stage coverage-then-deception curriculum so that cooperative agents learn to mislead an adversary about a protected target (Ghiya et al., 2020). In room clearance, a feudal hierarchy separates a commander that issues room-level orders from goal-conditioned subordinates that learn only to execute those orders, with low-level reward shaping for correct order completion and high-level reward shaping for speed, persistence of clearance, or civilian protection (Charlesworth et al., 2021).

A fourth pattern is model-based adversary synthesis. Multi-step generative models learn a joint predictive world model over interacting agents while encouraging disentanglement between agent-specific latent variables. In the competitive setting, separate latent strategy codes make it possible to plan one agent against another in latent space rather than learn both policies directly, and practical optimization uses multiple restarts as a proxy for max-min search (Krupnik et al., 2019).

Population and diversity methods dominate LLM-oriented red teaming. GRTS uses a Double-Oracle/PSRO-style loop with PPO best responses, restricted-game equilibrium computation, exploitability, and semantic diversity to mitigate mode collapse in multi-round red/blue dialogue games (Ma et al., 2023). Automated single-attacker red teaming contributes a modular reward-engineering template rather than a full MARL system: goal generation, attacker policy, target model, and judges are separated, and the total reward is multiplicative,

$o^i$7

which is directly extensible to future multi-agent attacker populations (Beutel et al., 2024).

Safety-critical versions add explicit constraints. DC-GPPO combines hard action filtering, graph convolutional state encoding, and a dual-constrained advantage,

$o^i$8

to keep interfering vehicles realistic while maximizing threat (Chen et al., 21 Jul 2025). Real-world craft-robot competition adds PPO, CTDE, shared within-team policies, alternating frozen-opponent training to reduce non-stationarity, and Out of Distribution State Initialization (OODSI) to retrain from deployment states that were off-distribution relative to the training simulator (Zhao et al., 24 Feb 2026).

4. Representative systems and empirical evidence

In mixed cooperative-competitive battle environments, regret-based within-team coordination is already sufficient to change tactical behavior. In the Magent battle game, a symmetric two-army combat domain with 64-agent and 256-agent settings, the proposed regret-based methods outperform IQL, ARM, COMA, and VDN in win rate and average reward, and the reported ranking is

$o^i$9

with BVRM-shaping learning an offensive strategy and BVRM without shaping learning a more defensive one (Yu et al., 2019). In team-competitive Markov games, RAC outperforms both MAAC and a role-only Team baseline in Touch-Mark and Market, with qualitative behaviors such as blocker-and-scorer coordination and split resource acquisition (Koley et al., 2023).

Explicit red-team scenario generation produces much sharper failure induction. In emergency braking, the baseline collision rate without interference is 5%, whereas red-team interference raises collision rate to 75% under single-vehicle interference and 85% under multi-vehicle interference with DC-GPPO; travel time, average lateral acceleration, and average speed shift in ways consistent with earlier and more stressful evasive dynamics (Chen et al., 21 Jul 2025). In multi-round LLM games, GRTS reduces exploitability from about 6.2 to about 0.8 over 15 iterations and shows that later rounds reveal vulnerabilities not exposed by first-turn attacks (Ma et al., 2023). In automated single-attacker red teaming, the central empirical tradeoff is between effectiveness and diversity: for safety jailbreaks, vanilla RL reaches $a^i$0 success but near-zero diversity, while multi-step RL retains high success with substantially higher attack and style diversity (Beutel et al., 2024).

Cyber defense and robotics extend red-team MARL beyond simulation-only adversarial search. On CICIoT2023, full C-MADF achieves precision 0.997, recall 0.961, F1 0.979, and false-positive rate 1.8%; the specific ablation w/o Red Team yields precision 0.963, recall 0.963, F1 0.963, and false-positive rate 21.2%, which the paper interprets as evidence that structured hypothesis testing mitigates reasoning drift under ambiguous evidence (Zhang et al., 6 Apr 2026). In real-world craft-robot competition, OODSI improves Sim2Real performance by 20% in the abstract, and the combined PPO+DR+OODSI setting raises Gazebo success to $a^i$1 in the cooperative task and $a^i$2 in the competitive task; qualitative behaviors include blocking and stealing (Zhao et al., 24 Feb 2026).

Earlier adversarially structured environments already exposed key ingredients of red-team behavior even when the hostile side was not learned explicitly. In a deception game, increasing deception reward reduces the frequency with which a heuristic adversary selects the true target, with the most pronounced 2-agent effect being a drop from $a^i$3 to $a^i$4 target selects (Ghiya et al., 2020). In a two-robot competitive manipulation task, model-based adversarial planning with disentangled latent strategies outperforms MADDPG and can synthesize stronger adversaries even from cooperative-task data (Krupnik et al., 2019).

5. Limitations, misconceptions, and conceptual disputes

A recurring misconception is that any competitive MARL with two sides is red-team MARL. The sources here repeatedly resist that simplification. Team sports is explicitly not a red-teaming paper; room clearance learns only blue agents against scripted hostile opposition; and team regret minimization is a coordination method for cooperative agents operating in mixed battle environments, not explicit opponent modeling or minimax training (Zhao et al., 2019, Charlesworth et al., 2021, Yu et al., 2019).

A second misconception is that “red team” always means “external attacker.” In fact, the literature contains several different red-team roles. The deception paper reports only heuristic-adversary results; emergency-braking red teams attack a fixed AV policy rather than a co-adapting defender; RAC relies on centralized training-time access sufficient to compare predicted opponent roles with opponents’ self-role distributions; and C-MADF is explicit that its Red-Team does not model the external APT actor but instead regularizes the Blue-Team’s decisions conservatively (Ghiya et al., 2020, Chen et al., 21 Jul 2025, Koley et al., 2023, Zhang et al., 6 Apr 2026).

Scalability and evaluation remain open issues. Model-based adversary synthesis studies only two agents and relies on approximate max-min optimization in latent space (Krupnik et al., 2019). Automated LLM red teaming highlights reward hacking, grader dependence, and the tension between diversity and effectiveness (Beutel et al., 2024). Across the literature, many systems assume privileged centralized information during training, stylized environments, or fixed target policies, so “red-team MARL” currently denotes a family of partially overlapping approaches rather than a single settled paradigm.

6. Research directions

Several directions are explicit in the literature. The general MARL report emphasizes CTDE, stochastic policies, and policy ensembles or self-play as responses to non-stationarity and exploitability (Kapoor, 2018). Team regret minimization identifies opponent-behavior modeling as future work, implying that strong within-team coordination under TONE is not yet equivalent to adaptive adversary reasoning (Yu et al., 2019). LLM red-teaming work argues for diverse attacker populations as a proxy for heterogeneous human red-teamers, while modular automated red teaming separates goal generation, attack policy learning, and judging into components that could become distinct agents in a fuller MARL system (Ma et al., 2023, Beutel et al., 2024).

This suggests that the next phase of red-team MARL is likely to combine explicit attacker populations, constrained action spaces, and deployment-robust training loops. A plausible implication is an overview of constrained red-team scenario generation from autonomous driving, OOD-state retraining from real-world robot competition, and disagreement-aware internal critics from cyber defense: one set of agents would search for failures, another would regularize realism or caution, and both would be trained against adaptive blue policies rather than fixed targets (Chen et al., 21 Jul 2025, Zhao et al., 24 Feb 2026, Zhang et al., 6 Apr 2026). Under that interpretation, red-team MARL is less a single algorithm than an architecture family for adversarial search under coordination, partial observability, safety constraints, and strategic interaction.