APyMARL: Adversarial c-MARL Toolkit

• Adversarial PyMARL is a modular toolkit that extends PyMARL to enable dual-team adversarial experiments in collaborative multi-agent reinforcement learning.
• It implements advanced attack methods, including Align and Hadamard, under black-box, observation-only constraints to assess system vulnerabilities.
• The platform supports simultaneous training and benchmarking of diverse MARL algorithms through reproducible experiments using a flexible YAML/JSON configurator.

Adversarial PyMARL (APyMARL) is a modular research toolkit and attack methodology developed for the study of adversarial vulnerability and algorithm-vs-algorithm evaluation in collaborative multi-agent reinforcement learning (c-MARL) settings. It extends the PyMARL codebase both by enabling dual-team adversarial experiments and introducing advanced black-box adversarial attacks against deployed MARL systems, supporting both observation-based attacks under real-world constraints and comprehensive adversarial benchmarking (Andam et al., 12 Aug 2025, Li et al., 18 Dec 2025).

1. Design Objectives and Core Extensions

APyMARL was designed to overcome three major limitations of prior MARL research ecosystems: (1) single-team-only training versus static built-in AI adversaries, (2) lack of built-in support for algorithm-vs-algorithm benchmarking, and (3) the absence of realistic, constrained adversarial attack modules for vulnerability assessment.

The principal objectives are as follows:

• Disentangle environment hosting from learning algorithms via a Gym-style multi-agent wrapper with full dual-team support.
• Enable simultaneous training, evaluation, and benchmarking of any two opposing MARL algorithms—policy-based or value-based—within the same micro-battle scenario.
• Permit comprehensive automation and reproducible experiments via a flexible YAML/JSON configurator, covering scenario setup, algorithm selection, adversarial modes, and all hyperparameters.
• Retain PyMARL’s modularity and PyTorch-based extensibility for research workflows (Li et al., 18 Dec 2025).

On the attack side, APyMARL directly addresses the shortcomings of previous adversarial evaluation, which typically assumed either full white-box access or unrealistic surrogate training budgets. Its novel attacks operate under black-box, observation-only settings, matching practical deployment scenarios (Andam et al., 12 Aug 2025).

2. System Architecture

APyMARL comprises three main architectural components:

• Multi-Agent Interaction Wrapper: Offers a gym-like API for parallel episode rollout, returning per-team observation lists, shaped rewards, termination flags, and auxiliary info. Internally, it maps discrete team or agent actions into StarCraft II’s PySC2/s2client-proto protocol, supporting fine-grained battlefield control.
• Dual-Team Adversarial Controller: Orchestrates two parameter-independent policy/victim learners. At every timestep, it collects observations from the environment for both teams, computes their respective joint actions, applies these via the API, and handles independent trajectory buffers and update logic for each team.
• Parameter Configurator: Centralizes experiment declaration (scenario, attack type, algorithm pairing, learning rates, replay buffer sizes, and evaluation frequency) in a machine-readable configuration file. The “runner” reads the config, sets up the above modules, and launches the training or testing loop (Li et al., 18 Dec 2025).

The attack pipeline integrates seamlessly as a PyMARL “AttackModule,” callable after observation collection and before agent policy evaluation, supporting both Align and Hadamard attack flavors as described below (Andam et al., 12 Aug 2025).

3. APyMARL Attack Methodologies

3.1 Align Attack

The APyMARL-Align attack is constructed under the following core constraints:

• Black-box setting: No access to the victim policies’ weights, gradients, actions, or rewards during test-time.
• Observation-only access: Collects a buffer ($T^c=1{,}000$–$5{,}000$) of multi-agent observations by passive interception. All surrogate training is done on this buffer only.
• Small perturbations: Perturbation vectors $\delta_i$ per agent are constrained in $\ell_\infty$ to maximum amplitude $\epsilon\in\{0.03,\ldots,0.2\}$ for SMAC/RWARE, with wider intervals up to $0.75$ for LBF.

Algorithmically, the attack trains a reconstruction network $f_\theta$ with the objective $f_\theta(o_{-i})\rightarrow o_i$ such that, if the agent observations are “aligned” (i.e., not manipulated), $f_\theta$ can accurately reconstruct individual agent views from the rest. The adversary then seeks adversarial perturbations $\delta$ that maximize the mean squared error of $5{,}000$0, solved as

$5{,}000$1

This maximization is executed using a projected gradient descent (PGD)-style loop for $5{,}000$2 steps. For $5{,}000$3 (i.e., FGSM), the method achieves >95% of maximal misalignment in a single gradient evaluation (Andam et al., 12 Aug 2025).

3.2 Hadamard Attack

If no observation buffer can be collected (the strictest black-box constraint), the attack falls back to the Hadamard method: generating orthogonal perturbations for each agent using rows from the Sylvester Hadamard matrix, scaled by $5{,}000$4, which guarantees maximal angular separation and decorrelation of the noise injected into each agent’s observation (Andam et al., 12 Aug 2025).

Variants include targeted attacks (“lighter Align”) where only a subset of agents, selected by their reconstruction error contributions, are perturbed.

4. Supported Adversarial Experimentation Modes

APyMARL provides two primary algorithmic paradigms for adversarial training and evaluation:

4.1 Dual-Algorithm Paired Adversary

In this mode, two algorithms (possibly of different families, e.g., QMIX vs VDN) are initialized either randomly or from checkpoints and co-evolve concurrently in the environment. Both update policies online, with performance measured by rolling episodic win-rate and other metrics. There is no privileged “ground truth” bot—evolution is fully adversarial and dynamic (Li et al., 18 Dec 2025).

4.2 Multi-Algorithm Mixed Adversary

Here, one learning algorithm (the “red” team) is trained against a static pool of $5{,}000$5 frozen opponents, which are uniformly sampled each episode. The red learner must adapt to behavioral diversity presented by this pool, improving generalization across different adversary strategies (Li et al., 18 Dec 2025).

Both paradigms are configured through the same API, with “mode” and “mixed_pool” settings in the experiment config.

5. Algorithmic Formulations and Training Protocols

APyMARL uses standard MARL loss formulations for all benchmarked algorithms:

• Value-based losses (QMIX, VDN, QPLEX, etc.): optimize empirical squared TD-errors on joint Q-functions, with team rewards averaged across agents at each timestep:

$5{,}000$6

• Policy-gradient methods (COMA, FOP, DOP): apply centralized critics, actor updates, and regularizers as per each algorithm’s objective;
• Reward shaping and penalties: for each agent $5{,}000$7 on team $5{,}000$8:

$5{,}000$9

with cumulative team reward averaged over agents and penalties for non-terminal timeouts.

The entire training and evaluation cycle can be script-driven (see Section 5 Usage Examples in (Li et al., 18 Dec 2025)) using the APyMARL config and runner interfaces.

6. Empirical Evaluation and Observed Performance

Empirical evaluation of APyMARL adversaries and adversarial training protocols spans 22 distinct scenarios (LBF, RWARE, SMACv2), with eight canonical MARL algorithms assessed in paired and mixed adversary settings (Andam et al., 12 Aug 2025, Li et al., 18 Dec 2025).

Attack Effectiveness:

• Align and Hadamard attacks substantially outperform random noise baselines, halving episodic return at $\delta_i$0–$\delta_i$1 and matching white-box PGD baseline in many regimes.
• Zero-knowledge Hadamard matches Align in highly cooperative or low-observability tasks.
• Targeted attacks (lighter Align + Hadamard) deliver an additional 6–12% return drop.
• Length of episodes on LBF benchmarks increases by 200–300% under the Align method.

Training and Generalization:

• In dual-algorithm paired mode, all algorithms show high win-rate fluctuation due to joint adaptation dynamics; value-based methods tend to improve steadily, while policy gradient methods like DOP surpass others on challenging maps.
• In mixed adversary mode, training against a diverse fixed pool yields smoother win-rate curves and improved generalization to built-in AI bots.
• PCA analysis evidences a broader tactical repertoire under adversarial (paired or mixed) training than in fixed-AI regimes.

Sample Efficiency:

• The Align attack (with reconstruction network) achieves >95% of maximal attack strength with $\delta_i$2 observation samples versus millions for prior policy-simulation-based black-box attacks (Andam et al., 12 Aug 2025).

A summary of supported benchmark environments, algorithms, and attack comparison is shown below:

Task Type	Envs/Scenarios	Victim Algorithms	Attack Methods
LBF	10 (full/partial obs)	QMIX, MAPPO	Align, Hadamard
RWARE	6 (varied obs/range)	QMIX, MAPPO	Random, Whitebox
SMAC/SMACv2	6 + 10	QMIX, VDN, FOP, DOP, etc.	Align, Hadamard

7. Implementation, Configuration, and Limitations

Network and Attack Hyperparameters:

• Reconstruction networks: MLP, RNN (GRU), or Transformer architectures (varying hidden sizes and epochs by task) for $\delta_i$3;
• PGD parameters: $\delta_i$4–$\delta_i$5, $\delta_i$6 (default) or $\delta_i$7 (FGSM, for speed), step size $\delta_i$8;
• Environment-specific $\delta_i$9 values.

Integration with PyMARL:

• Plugin attack modules levered after observation collection, using runner hooks and configurable via new YAML fields.
• Supports observation perturbation and reinjection using the same API as action-masking.

Limitations:

• Paired adversarial training leads to volatile performance profiles and complicates convergence analysis.
• Existing algorithms are fragile under scenario asymmetry; even a 10–20% force imbalance triggers catastrophic failure.
• In mixed adversary mode, the static nature of the opponent pool precludes joint dynamic co-evolution.
• Large-scale or heterogeneous team scenarios remain challenging and expose credit assignment issues.

Future Directions:

• Dynamic evolution of opponent pools during mixed adversary training.
• More asymmetric and non-stationary micro-battle scenarios.
• Improved metrics and reward shaping to mitigate collapse under force imbalance (Li et al., 18 Dec 2025).

APyMARL consolidates both state-of-the-art adversarial attack methods for c-MARL and a robust adversarial experimental platform, driving advances in the field’s understanding of robustness, generalization, and competitive dynamics. The integration of sample-efficient, practically constrained attacks and scalable adversarial benchmarks distinguish APyMARL as a foundational toolkit for the next generation of MARL research (Andam et al., 12 Aug 2025, Li et al., 18 Dec 2025).