Coach-Assisted MARL Framework

• Multi-agent reinforcement learning frameworks are systems where multiple agents use reinforcement learning to interact, coordinate, and adapt, even under failure conditions.
• The coach-assisted framework dynamically adjusts agent crash rates based on performance metrics, enhancing policy robustness in both grid-world and StarCraft II scenarios.
• A resampling rule limits the number of crashed agents per episode, ensuring training stability and sustained performance during high-failure conditions.

A multi-agent reinforcement learning (MARL) framework describes the architectural, mathematical, and algorithmic design for enabling a group of agents to interact with an environment—and each other—using reinforcement learning principles. Such frameworks are critical for scalable artificial intelligence in decentralized, cooperative, or competitive settings where agents may fail, must share information, or need to robustly coordinate under dynamic or adversarial conditions. This article explicates the technical innovations and implementation details of the coach-assisted MARL framework for handling unexpected crashed agents, as formalized in (Zhao et al., 2022). The focus is on robustness to agent failures during cooperative tasks, adaptive training, and empirical results in both grid-world and StarCraft II micromanagement scenarios.

1. Formalization of the Crashed Dec-POMDP

The coach-assisted MARL framework extends the standard decentralized partially observable Markov decision process (Dec-POMDP) to include agent failure ("crash") events. The augmented model is defined by

$$M = \langle N, S, A, \Omega, P, O, R, \gamma, \alpha \rangle$$

where:

• $N = \{g_1, ..., g_n\}$: set of agents.
• $S$: global state space.
• $A$: individual agent action space, with joint action $\vec{a} = (a_1, ..., a_n) \in A^n$.
• $\Omega$: observation space, $o_i \sim O(\cdot | s)$ per agent.
• $P(s' | s, \vec{a})$: transition kernel.
• $R: S \times A^n \to \mathbb{R}$: shared team reward.
• $\gamma \in [0, 1)$: discount factor.
• $\alpha \in [0, 1]$: per-agent crash probability.

At the start of each episode $t$, a coach agent selects a crash rate $\alpha_t$. Each agent $i$ is assigned a crash flag $c_i \sim \mathrm{Bernoulli}(\alpha_t)$, and the vector $\vec{c}$ is fixed for the episode. A crashed agent (i.e., $c_i = 1$) selects only random or no-op actions; uncrashed agents follow learned policies $\pi_i(a_i | \tau_i)$ with action-observation histories $\tau_i = (o_i^0, a_i^0, o_i^1, a_i^1, ..., o_i^T)$. The training objective is

$$J(\pi) = \mathbb{E}\left[\sum_{t=0}^\infty \gamma^t R(s^t, \vec{a}^t)\right]$$

where the expectation is over the crash-augmented environment dynamics.

2. Coach-Assisted Framework Architecture

The architecture comprises two main components:

• Learning Agents: Each agent independently learns its policy via a central-training, decentralized-execution (CTDE) algorithm (e.g. QMIX), conditional on not being crashed in the episode.
• Virtual Coach Agent: At the start of each episode, selects the crash rate $\alpha_t$ and samples the crash vector $\vec{c}$. After the episode, observes a performance metric $e_t$ (such as average win rate or episode return) and updates $\alpha_{t+1}$ according to a selected coaching strategy.

Episode Iteration:

1. Coach sets $\alpha_t$ and samples $\vec{c}$ under a crash constraint.
2. Agents execute the episode rollout using $\vec{c}$.
3. System records performance metric $e_t$.
4. Coach updates $\alpha_{t+1}$ via function $F(\alpha_t, e_t, \beta)$.
5. Agents' value/policy networks are updated based on collected transitions.

3. Coaching Strategies and Resampling Rule

Three coaching strategies are formalized:

• Fixed Crash Rate: $F(\alpha_t, e_t, \beta) = \alpha_t$ (constant $\alpha$).
• Curriculum Learning: $\alpha_1 = 0$, then $F(\alpha_t, e_t, \beta) = \alpha_t + \Delta\alpha$ until a specified maximum.
• Adaptive Crash Rate:

$$F(\alpha_t, e_t, \beta) = \alpha_t + \rho \cdot (\mathbf{1}_{e_t \geq \beta} - \alpha_t)$$

where $\beta$ is a performance threshold and $\rho$ controls adaptation speed. When $e_t \geq \beta$, the crash rate is increased (greater difficulty); otherwise, it is decreased.

Resampling Rule: Standard Bernoulli draws of $c_i$ can yield episodic crash counts much above or below $n\alpha_t$. To avoid training instability, resampling enforces at most $K=\lceil n\alpha_t\rceil$ crashed agents per episode: $\sum_{i=1}^n c_i \leq K$ Episodes violating this bound are discarded, and $c$ is resampled, stabilizing episode difficulty.

4. Training Algorithm and Implementation Details

The training loop is formally presented:

\begin{algorithmic}[1] \Require Crash update %%%%39%%%%, threshold %%%%40%%%%, rate %%%%41%%%%, max episodes %%%%42%%%%, steps per episode %%%%43%%%% \State Initialize agent parameters %%%%44%%%%; set %%%%45%%%% \For{episode %%%%46%%%% to %%%%47%%%%} \State %%%%48%%%% \State %%%%49%%%% \Repeat \State Sample %%%%50%%%% for %%%%51%%%% \Until{%%%%52%%%%} \State Run episode with crash vector %%%%53%%%% \State Compute %%%%54%%%% (win rate / return) \State Update agent networks via CTDE (e.g., QMIX loss) \State Coach observes %%%%55%%%% for next %%%%56%%%% \EndFor \end{algorithmic}

• Base Learner: QMIX with central training and individual agent polices for decentralized execution.
• Performance Metric: Adaptable (win rate, episode reward), chosen by the implementer but must be consistent for the adaptive update.
• Resource Requirements: Standard for QMIX-based CTDE; the coach module introduces negligible overhead.
• Scaling Considerations: The resampling rule ensures stability for large $n$; coach overhead grows with episode frequency, not agent count.

5. Empirical Evaluation: Grid-World and StarCraft II

Grid-World Toy Example:

• 10×10 grid; 2 agents (success if both buttons pressed in ≤20 steps).
• QMIX baseline fails under single agent crash: $\sim 0\%$ success.
• Adaptive coaching (with resampling): $>90\%$ success in crashed scenarios.

StarCraft II Micromanagement Tasks (SMAC):

• Scenarios: 3s_vs_5z, 8s_vs_3s5z, 3s5z_vs_3s5z, 8m_vs_5z.
• Crash model: crashed agents take uniform random valid actions.
• Strategies: fixed $\alpha \in \{0.01, 0.05, 0.10\}$, curriculum ($\Delta\alpha$), adaptive (best: $\beta=0.65, \rho=0.003$).
• Training: 2M steps, 5 seeds, 128 test runs/model.
• Metric: win rate at test crash rates.

Scenario	Test $\alpha$	Baseline QMIX	Fixed–0.05	Curriculum	Adaptive$^-$	Adaptive+Resample
3s_vs_5z	0.10	57 ± 15	73 ± 10	75 ± 6	72 ± 10	79 ± 7
8s_vs_3s5z	0.10	69 ± 5	67 ± 6	67 ± 6	69 ± 6	71 ± 12

Adaptive strategy with resampling delivers the highest robustness to crashes, outperforming both fixed and curriculum strategies by 3–5% in absolute win rate under severe crash scenarios. In all evaluated maps and crash rates, adaptive coaching is superior regarding team performance retention in the presence of agent failures.

6. Implementation Trade-Offs and Deployment Considerations

• Compared to naive fixed-rate training, the adaptive coach instructs agents to dynamically adjust to failure rates, yielding policies robust even to catastrophic teammate loss.
• Resampling Bounds prevent over-iteration on pathological high-crash episodes that are rarely encountered in deployment, thus focusing sample complexity on feasible scenarios.
• Hyperparameter Selection: The adaptation rate ($\rho$) and performance threshold ($\beta$) must be tuned for the domain; empirically, moderate $\rho$ avoids sudden jumps in difficulty.
• Limitation: The present framework only models episode-initial crashes (no mid-episode failure/recovery), and crashed agents follow a simplified random/no-op policy. Real-world failure modes may require richer modeling.

7. Strengths, Limitations, and Future Directions

Strengths:

• Provides a mathematically grounded protocol for training crash-tolerant multi-agent policies via a coach module that adjusts environmental stochasticity.
• Empirically validated on both synthetic and complex multi-agent control benchmarks, demonstrating improvements in robustness and success rates.
• The adaptive coaching protocol is directly implementable using contemporary deep MARL libraries, with only minor modifications to the CTDE loop.

Limitations:

• Assumes agent crashes occur only at the episode start, with no in-episode recovery.
• Simplifies crash dynamics to random/no-op actions; does not cover partial degradations (e.g., sensor faults).
• Adaptive update function is scalar and uses a fixed performance threshold; does not exploit multi-objective or risk-sensitive criteria.

Potential Extensions:

• In-episode fault occurrence and recovery modeling.
• Enriching the crash model to include a spectrum of failures (partial, stochastic, or delayed recovery).
• Meta-learning or reinforcement-learned coaching strategies, enabling end-to-end adaptation of the crash curriculum.
• Application to mixed cooperative–competitive or larger heterogeneous agent teams.

The coach-assisted MARL framework addresses a significant realism gap in multi-agent learning by systematically exposing agents to probabilistic team-member failures during training and equipping the team with crash-robust policies validated through rigorous benchmarking.