Multi-Agent Reinforcement Learning Framework

• Multi-Agent Reinforcement Learning (MARL) is defined as a framework where multiple agents learn in complex environments using Markov games or POMDPs, addressing challenges like partial observability and non-stationarity.
• The framework emphasizes centralized training with decentralized execution, utilizing techniques like parameter sharing and actor-worker-learner pipelines to improve sample efficiency and system stability.
• Co-evolutionary safety mechanisms, such as Evo-MARL and AdvEvo-MARL, introduce adversarial robustness by jointly optimizing attacker and defender strategies, achieving significant reductions in attack success rates.

Multi-Agent Reinforcement Learning (MARL) frameworks provide the algorithmic and systems backbone for optimizing sequential decision-making in environments with multiple interacting learning agents. MARL frameworks address a spectrum of technical challenges—such as credit assignment, partial observability, non-stationarity, safety, and scalability—by integrating architectural, algorithmic, and engineering innovations. This article synthesizes central principles, recent co-evolutionary paradigms, representative distributed and safety-focused frameworks, and quantitative effects, as established in leading research (Pan et al., 5 Aug 2025, Pan et al., 2 Oct 2025, Qi et al., 2022).

1. Formal Problem Structure and Threat Models

A MARL framework formalizes learning in a Markov game or partially observable Markov decision process (POMDP) with $N$ agents. The environment is typically encoded as

$$\mathcal{E} = (\mathcal{S}, \{\mathcal{A}_i\}_{i=1}^N, \mathcal{T}, \{r_i\}_{i=1}^N, \gamma)$$

where $\mathcal{S}$ is the state space (potentially containing full history for language-agent systems), $\mathcal{A}_i$ the action space for agent $i$, $\mathcal{T}$ the transition kernel, $r_i$ the (possibly team-shared or individual) reward, and $\gamma$ the discount factor.

Contemporary frameworks such as Evo-MARL and AdvEvo-MARL generalize the paradigm to adversarial settings with co-evolving attackers and defenders (Pan et al., 5 Aug 2025, Pan et al., 2 Oct 2025). The attacker’s action space can consist of prompt-based adversarial inputs or policy-level perturbations; defenders produce output sequences and are jointly tasked with achieving correct, safe, and robust system outputs.

A canonical threat model posits attackers capable of arbitrary prompt-based manipulations, with adversarial content possibly propagating along chains or arbitrary MAS communication topologies.

2. Training Architectures: Parameter Sharing, Distributed Execution, and System Decomposition

A central architectural principle is the separation of concerns across training and execution:

• Centralized Training, Decentralized Execution (CTDE):

All agents may access (partial or full) global information during training (e.g., for a centralized critic) but act using only local histories or observations at deployment.

• Parameter Sharing:

A single policy network (e.g., an autoregressive transformer) with role embeddings is shared among all agent instances, enabling substantial parameter efficiency and transfer of safety-relevant representations (Pan et al., 5 Aug 2025).

• Actor–Worker–Learner Pipelines:

Sample efficiency and training throughput are greatly enhanced by deploying multiple asynchronous environment-interacting actors, co-located policy-inference workers, and a centralized learner that updates global network weights independently of data collection. This design enables 6–8$\times$ speedups over standard synchronous pipelines and is standard in distributed MARL frameworks such as DMCAF (Qi et al., 2022).

Component	Description	Key Benefit
Actors	Interact with environment to gather raw trajectories; run in parallel processes/threads	High sample throughput
Workers	Receive state from actors, compute actions under local (possibly stale) parameters	Inference decoupling
Learner	Samples joint transitions, performs SGD updates, syncs fresh policy parameters to workers	Efficient learning

Parameter-pull staleness is tolerated and, in value-based methods, can even stabilize learning.

3. Co-evolutionary and Safety-Internalized MARL

Emergent open-domain MAS (e.g., LLM-based tool-using agents) face adversarial risks (jailbreak, prompt-injection). Modern frameworks embody explicit co-evolution and safety internalization (Pan et al., 5 Aug 2025, Pan et al., 2 Oct 2025):

• Evo-MARL:

Simultaneously co-trains task agents (defenders) to perform both primary and adversarially robust functions without relying on centralized guard modules. A parameter-sharing policy optimizes a joint reward blending safety and utility. Attackers evolve via mutation and crossover within a prompt population, selected based on empirical attack success rates.

• AdvEvo-MARL:

Extends Evo-MARL with role-separated attacker and defender agents, each trained with distinct policy networks. A novel public group baseline is used for advantage estimation, lowering variance and aligning intra-group learning, and is vital for stable joint optimization.

Framework	Attacker Representation	Defender Representation	Core Safety Innovation
Evo-MARL	Evolutionary prompt pool	Shared-policy transformer + role emb.	Internalized safety via GRPO
AdvEvo-MARL	LLM-based, SFT-warmed + RL attacker	LLM defenders, fully independent	Public baseline for group RL

Quantitative findings include up to 22% absolute ASR reduction, with utility (accuracy) non-trivially improved, demonstrating the non-necessity of safety–utility trade-offs.

4. Optimization and Loss Formulations

The optimization of defender agents integrates policy gradients and evolutionary selection:

• Defender Objective:

Group Relative Policy Optimization (GRPO) generalizes PPO to $G$ concurrent rollouts per query, applying KL regularization and clipping, with per-sample advantage estimates.

$$J_{\mathrm{GRPO}}(\theta) = \mathbb{E}\left(\frac{1}{G}\sum_{g=1}^G \frac{1}{T} \sum_{t=1}^{T} \min(r_{g,t}(\theta)\hat{A}_{g,t}, \mathrm{clip}(r_{g,t}(\theta),1-\varepsilon,1+\varepsilon)\hat{A}_{g,t}) - \beta\,\text{KL}\right)$$

• Attacker Evolution:

Attack fitness is empirically estimated as fraction of successful attacks on the current defender. Top-$K$ candidates undergo reproduction via crossover and mutation per generation.

• AdvEvo-MARL Minimax:

Implements alternating updates optimizing defenders to maximize (defender reward $-$ attacker reward), while attackers simultaneously explore the reward landscape.

A salient implementation in AdvEvo-MARL is the public baseline: mean reward across group members is used as the advantage baseline, synchronizing gradients and mitigating excessive response length truncation and instability.

5. Empirical Results and Quantitative Metrics

Experiments on red-teaming and reasoning benchmarks consistently validate the efficacy of safety-internalized co-evolutionary MARL (Pan et al., 5 Aug 2025, Pan et al., 2 Oct 2025):

Model	JailBreakV ASR	HarmBench ASR	MATH Acc (%)	Creative Writing (%)
MAS-1.5B (orig)	22.6	69.0	43.0	8.2
MAS-1.5B + Evo-MARL	17.4 (↓5.2)	48.0 (↓21.0)	48.0 (+5.0)	8.6 (+0.4)
MAS-3B (orig)	51.7	76.0	57.0	13.8
MAS-3B + Evo-MARL	46.5 (↓5.2)	68.0 (↓8.0)	60.0 (+3.0)	15.2 (+1.4)

Defender–attacker co-evolution stabilizes with dynamic attacker populations. Public baseline ablations in AdvEvo-MARL show training can collapse (ASR rises, terse outputs) when group baselines are omitted. Experiments on different agent communication topologies (chain/tree/complete) confirm the generality of the safety internalization effect.

6. Current Limitations and Future Extensions

The principal computational bottleneck is multi-agent LLM training at scale; current studies run with $N=3$ due to hardware constraints. Evolutionary search over attacker prompts is sample-inefficient for high-dimensional attack spaces.

Possible extensions include:

• Incorporation of persistent memory to facilitate accumulation of attack/defense knowledge.
• Generalization beyond chain-structured interaction topologies to arbitrary MAS communication graphs.
• Risk-sensitive reward shaping, reward decomposition, and theoretical study of minimax convergence in non-stationary attacker–defender distributions (Pan et al., 2 Oct 2025).
• Scaling to larger populations and integration of multimodal and real-time agent architectures.

Co-evolutionary safety-internalized MARL frameworks (e.g., Evo-MARL and AdvEvo-MARL) provide evidence that robust, scalable, and utility-preserving safety mechanisms can be embedded directly into the task policy. These approaches eliminate external single-point-of-failure guard modules, ensure cross-agent safety coordination, and open further avenues for principled adversarial robustness research in multi-agent systems.