MAGIC-MASK: Explainability in Multi-Agent RL

• MAGIC-MASK is a perturbation-based explainability framework for multi-agent reinforcement learning that leverages mask networks to identify critical states.
• The framework integrates adaptive exploration and decentralized inter-agent communication to efficiently uncover salient state representations amid partial observability and dynamic coordination.
• Empirical evaluations show MAGIC-MASK outperforms baselines by improving reward stability, inter-agent fidelity, and precise critical state detection for robust policy optimization.

MAGIC-MASK is a mathematically grounded framework for perturbation-based explainability in Multi-Agent Reinforcement Learning (MARL). It unifies mask-based saliency, adaptive exploration, policy regularization, and decentralized inter-agent collaboration to generate localized, interpretable explanations for agent decisions under partial observability and dynamic coordination requirements. By extending state-masking techniques from single-agent to multi-agent domains, MAGIC-MASK enables robust identification and sharing of critical states among agents, accelerating both explanation discovery and policy optimization (Maliha et al., 30 Sep 2025).

1. Limitations of Single-Agent Explainability and Motivation

Traditional post-hoc explainability paradigms in reinforcement learning, such as perturbation-based StateMask, are limited when applied to multi-agent scenarios:

• Explanations derived by masking actions and observing reward impact consider only single-agent trajectories, neglecting inter-agent dependencies that can amplify or mask effects of perturbations.
• Computational cost increases quadratically with team size if applied naively per agent.
• Absence of mechanisms for sharing discovered saliency masks causes redundant exploration and delayed identification of jointly critical states.
• These approaches generally presume full observability, excluding the practical realities of partial observability or mixed cooperative-competitive dynamics in multi-agent Markov decision processes (MDPs) and partially observable MDPs (POMDPs).

MAGIC-MASK is designed to address these constraints by formulating collaborative multi-agent saliency discovery, undergirded by black-box access to states, actions, and rewards (Maliha et al., 30 Sep 2025).

2. Mathematical Formalism: Multi-Agent Masking and Reward Fidelity

MAGIC-MASK models the environment as an N-agent system $$(\mathcal S,\{\mathcal A^i\}_{i=1}^N,P,\{r^i\}_{i=1}^{N},\gamma)$$ where each agent $i$ follows a policy $\pi_{\theta}^i$ over local states $s_t^i$. Each agent maintains a parametric mask network $M_{\phi}^i$ generating a soft saliency score $m_t^i$ for state $s_t^i$.

Action selection at time $t$ is governed by the mask: $$a_t^i = \begin{cases} a_t^i \sim \pi_{\theta}^i(\cdot|s_t^i), & m_t^i > \tau,\ a_t^i \sim \mathrm{Uniform}(\mathcal A^i), & m_t^i \le \tau \end{cases}$$ Perturbed trajectories are analyzed via the masked return $R_i^{\pi,M}=\sum_{t=0}^T \gamma^t r_t^i$. The expected fidelity gap: $$\Delta_i(\phi) = \mathbb{E}_{\tau} [|R_i^{\pi} - R_i^{\pi,M}|]$$ quantifies reward sensitivity to perturbation at critical states. Mask network optimization employs the surrogate loss: $$\mathcal{L}_{\textrm{mask}}(\phi^i) = \mathrm{MSE}(\mathbb{E}_t[m_t^i],\tau)$$ which regulates the proportion of actions subject to randomization, balancing exploration versus fidelity.

Proximal Policy Optimization (PPO) is stabilized under mask-induced perturbations by augmenting the surrogate objective with a KL-divergence regularizer: $$\mathcal{L}_{\textrm{PPO+KL}}(\theta^i) = \mathcal{L}_{\textrm{PPO}}(\theta^i) - \beta\,\mathbb{E}_t[D_{KL}(\pi_{\theta_{\textrm{old}}}^i(\cdot|s_t^i)\|\pi_{\theta}^i(\cdot|s_t^i))]$$ $\beta$ modulates the policy shift tolerance due to masking.

3. Algorithmic Structure: Saliency Discovery, Collaboration, and Policy Update

MAGIC-MASK agents jointly execute the following algorithmic steps per episode:

• Adaptive Exploration: Each agent $i$ deploys $\epsilon$-greedy exploration with an exponentially decaying $\epsilon_t^i = \epsilon_0 \exp(-\lambda t)$ to ensure broad state coverage complementary to randomization induced by mask networks.
• Mask-Based Saliency: Saliency mask scores $m_t^i = M_{\phi}^i(s_t^i)$ are central in designating critical states. States yielding low $m_t^i$ produce maximal reward deviation under action perturbation—these regions are deemed explanations for policy behavior.
• Inter-Agent Communication: Each agent maintains a communication buffer $\mathrm{Comm}_t^i = \{s_t^i | m_t^i \le \tau\}$, broadcasting only compact indices or scores. The global saliency set $$\mathrm{Comm}_t = \bigcup_{i=1}^N \mathrm{Comm}_t^i$$ guides decentralized and asynchronous prioritization in exploration and mask network tuning, preventing redundant probes.
• Policy and Mask Update: Policy $\pi_{\theta}^i$ is updated via PPO+KL. Mask network $M_{\phi}^i$ is trained with $\mathcal L_{\mathrm{mask}}(\phi^i)$ and a reward-preservation term scaled by $\lambda_f$—with $\Delta_i(\phi)$ approximated by the KL divergence over returns.

Ablation confirms the irreplaceability of communication: suppressing $\mathrm{Comm}_t$ drops reward by $\approx$20–25% and fidelity by $\approx$10–12 points.

4. Evaluation: Benchmarks, Metrics, and Quantitative Performance

MAGIC-MASK is validated on diverse environments, encompassing turn-based (Connect4), card games (Doudizhu), classical Atari (Pong), continuous control (Multi-Agent Highway), and cooperative sports (Google Research Football). Benchmarked metrics include:

• Final Average Reward (episodic return, higher is better)
• KL Divergence (between original and masked policy, lower is more stable)
• Inter-Agent Fidelity (correlation of saliency masks across agents, higher denotes agreement)
• Reward Drop After Perturbation (percentage decline when randomizing actions at critical states; higher indicates precise critical state identification)

Metric / Env	MAGIC-MASK	StateMask	LazyMDP	EDGE	ValueMax
Final Reward (Connect4)	40.2	37.6	36.2	24.8	35.8
Final Reward (Pong)	22.5	19.8	18.0	19.0	21.0
KL Divergence (Pong)	0.08	0.12	0.10	0.13	0.11
Fidelity (Pong)	0.92	0.78	0.85	0.80	0.87
Reward Drop (Pong) (%)	15.1	12.4	11.2	13.1	10.5

A plausible implication is that coordinated masking and peer exchange improve both the coverage and fidelity of critical state identification.

5. Explainability Outputs and Transferability

MAGIC-MASK produces localized saliency overlays, heatmaps, and temporal maps that explicate agent decision-making. In continuous control domains (e.g., highway driving), colored overlays denote regions where own or peer perturbations produced significant reward loss, guiding proactive lane selection and safety-aware behavior. Football domain heatmaps delineate possession-critical states influencing coordinated pass/dribble decisions.

Practitioners are provided with sparse, time- and space-localized explanations, communication logs, and transferability evidence: saliency masks learned in high-density traffic transfer partially to less familiar densities, expediting adaptation.

6. Future Extensions and Directions

Open extensions include:

• Scaling MAGIC-MASK to larger teams employing sparse, hierarchical, or bandwidth-constrained communication topologies.
• Extending saliency protocols to heterogeneous or mixed cooperative-competitive scenarios such as StarCraft2 or multi-agent traffic control.
• Augmenting mask-based explanations with rationales human operators can interpret (e.g., "chose braking because pedestrian region flagged critical by peer").

MAGIC-MASK leverages a unified mathematical framework to advance explainability, sample efficiency, stability, and robustness in MARL. These properties are evidenced by superior results in reward, fidelity, and inter-agent agreement over state-of-the-art baselines (Maliha et al., 30 Sep 2025).