---
title: 'MAC-PO: Multi-Agent Regret-weighted Replay'
url: https://www.emergentmind.com/topics/multi-agent-regret-weighted-replay-mac-po
type: topic
---

# MAC-PO: Multi-Agent Regret-weighted Replay

Multi-Agent Regret-weighted Replay via Collective Priority Optimization (MAC-PO) is a prioritized experience replay framework specifically designed for cooperative multi-agent reinforcement learning (MARL) under the centralized training and decentralized execution (CTDE) paradigm. MAC-PO rigorously formulates the replay prioritization problem as a collective regret minimization over sampling weights, yielding a closed-form expression for sample priorities that integrates temporal-difference (TD) error, optimality gap, on-policy likelihood, and novel joint action factors unique to multi-agent settings. Empirical evaluation demonstrates MAC-PO's efficacy on benchmark environments, where it consistently surpasses established alternatives in sample efficiency and final performance [2302.10418].

## 1. Multi-Agent Markov Game Setting and Experience Replay

The MARL context considered is a decentralized partially observable Markov decision process (Dec-POMDP) operating under CTDE:

- **Agents and Actions**: The agent set is $A = \{1,\ldots, n\}$, with each agent $a$ selecting $u_a \in U^a$. The joint action is $u = (u_1, \ldots, u_n) \in U^1 \times \cdots \times U^n$.
- **State and Transitions**: Global state $s\in S$ evolves via $P(s'|s, u)$, with all agents sharing reward $r(s, u)$ and a common discount factor $\gamma \in [0, 1)$.
- **Observations and Policies**: Each agent gets local observations $z \sim O(s, a)$ and maintains private histories $\tau_a \in (Z \times U^a)^*$. Decentralized policies $\pi^a(u_a|\tau_a)$ define the joint policy $\pi(u|\tau) = \prod_a \pi^a(u_a|\tau_a)$.
- **Experience Replay**: A finite buffer $\mathcal{D}$ stores transitions $(s, u, r, s', ...)$, generated off-policy. Standard approaches sample transitions uniformly or with fixed priorities. In MARL, uniform experience replay is sub-optimal, as it ignores sample importance and inter-agent policy dependencies.

## 2. Regret Minimization and the Weighted Bellman-Error Objective

MAC-PO's central objective is to minimize policy regret, narrowing the gap between the expected return of the current joint policy and a nominal optimal joint policy.

- **Policy Regret**: For joint policy $\pi$, expected (discounted) return is $\eta(\pi) = \mathbb{E}_{\tau_0 \sim d_0, s_0 \sim \cdots} \left[\sum_{t=0}^\infty \gamma^t r_t\right]$, with regret defined as
  $$
  \mathop{\mathrm{regret}}(\pi) = \eta(\pi^*) - \eta(\pi),
  $$
  where $\pi^*$ is optimal.
- **Weighted Bellman-Error Minimization**: At iteration $k$, with current estimate $Q_{k-1}$, MAC-PO fits $Q_k$ by minimizing a prioritized, weighted Bellman error:
  $$
  Q_k = \arg\min_{Q \in \mathcal{Q}} \mathbb{E}_{(s, u) \sim \mu}\Big[w_k(s, u) \cdot (Q(s, u) - \mathcal{B}^* Q_{k-1}(s, u))^2\Big],
  $$
  s.t. $w_k(s, u) \ge 0,\, \sum_{s, u} \mu(s, u) w_k(s, u) = 1$, where $\mathcal{B}^* Q_{k-1}(s, u) = r(s, u) + \gamma \mathbb{E}_{s'}[\max_{u'} Q_{k-1}(s', u')]$ is the Bellman target and $w_k$ are sampling-priority weights to be optimized.

- **Meta-optimization**: The core problem is to optimize $w_k$:
  $$
  \min_{w_k \ge 0,\, \mathbb{E}_\mu[w_k] = 1} \eta(\pi^*) - \eta(\pi_k)
  $$
  with $\pi_k$ the Boltzmann policy induced by $Q_k$.

## 3. Regret Relaxation, Lagrangian Duality, and Optimal Priorities

MAC-PO formulates a tight surrogate upper bound for policy regret, based on discounted Q-value errors:

- **Surrogate Loss**: By Kakade's lemma and Jensen's inequality, regret is bounded as
  $$
  \eta(\pi^*) - \eta(\pi_k) \le \mathbb{E}_{(s, u) \sim d^{\pi_k}}\left[|Q_k(s, u) - Q^*(s, u)|\right],
  $$
  replaced by $\mathcal{L}_{\mathrm{obj}}(w) = -\log \mathbb{E}_{d^{\pi_k}}\Big[\exp(-|Q_k(s, u) - Q^*(s, u)|)\Big]$.

- **Lagrangian Formulation**: With $p_k(s, u) = \mu(s, u) w_k(s, u)$ and duals $\lambda, \psi(s, u)$:
  $$
  \mathcal{L}(p, \lambda, \psi) = -\log \mathbb{E}_{d^{\pi_k}}\left[\exp(-|Q_k - Q^*|)\right]
  + \lambda \Big(\sum_{s, u} p_k(s, u) - 1\Big)
  - \sum_{s, u} \psi(s, u) p_k(s, u)
  $$

- **Implicit Differentiation**: The Q-iteration solution is analytically differentiated w.r.t. $p_k$ via the implicit function theorem; $\frac{\partial Q_k}{\partial p_k}$ involves the Bellman differences.

- **Closed-form Priority**: Karush-Kuhn-Tucker (KKT) conditions yield the optimal sampling priority (Theorem 1):
  $$
  w_k(s, u) = \frac{p_k(s, u)}{\mu(s, u)} = \frac{1}{Z^*}[E_k(s, u) + \epsilon_k(s, u)]
  $$
  where
  $$
  E_k(s, u) =
    \left(\frac{d^{\pi_k}(s, u)}{\mu(s, u)}\right)
    |Q_k(s, u) - \mathcal{B}^* Q_{k-1}(s, u)|
    \exp(-|Q_k(s, u) - Q^*(s, u)|)
    \left[1 + \sum_{i=1}^n \prod_{j \ne i} \pi^j_k(u_j|s) - n \prod_{i=1}^n \pi^i_k(u_i|s)\right]
  $$
  $Z^*$ is a normalization factor; $\epsilon_k(\cdot)$ is negligible for low return-to-state probability.

## 4. Practical Prioritization: Exact and Approximate Methods

The computation of optimal priorities in the multi-agent setting involves non-trivial joint-policy dependencies:

- **Exact MAC-PO**: For each transition $(s, u)$:
  $$
  \text{priority} \propto |\delta_k(s, u)| \exp(-|Q_k(s, u) - Q^*(s, u)|)
  \left[1 + \sum_{i=1}^n \prod_{j \ne i} \pi^j_k(u_j | s) - n \prod_{i=1}^n \pi^i_k(u_i | s)\right]
  $$
  with $\delta_k(s, u) = Q_k(s, u) - \mathcal{B}^* Q_{k-1}(s, u)$.

- **Approximation Scheme**: To avoid the $\mathcal{O}(|U|^n)$ cost of computing full joint probabilities, MAC-PO introduces a three-level partition based on Theorem 2:
  - **High**: Exactly one agent’s local policy has $\pi^a_k(u_a|s) \approx 0$ and all others $\approx 1$;
  - **Low**: $\prod_i \pi^i_k \approx 0$ or $\approx 1$;
  - **Medium**: Otherwise.
  
  Assigning weights $\{\alpha_l, \alpha_m, \alpha_h\}$ to these levels, final transition priority is computed by multiplying Bellman error, value enhancement, and level weight.

## 5. Implementation and Optimization Procedure

The MAC-PO algorithm proceeds as follows:

1. **Initialization**: Set parameter vectors $\theta$, target $\theta^-$, and empty replay buffer $\mathcal{D}$.
2. **Sampling**: Collect trajectories $(s_k, u_k, r_k, s_{k+1})$ under $\epsilon$-greedy policy $\pi_k$; store in $\mathcal{D}$.
3. **Minibatch Update**:
   - Sample $b$ transitions uniformly.
   - For each $(s, u)$ pair:
     - Compute $Q_k(s, u;\theta)$, Bellman target $y = r + \gamma Q_{k-1}(s', u'^*; \theta^-)$.
     - Compute per-agent Boltzmann policies $\pi^a_k(u_a|s)$.
     - Estimate $Q^*(s, u)$ (unrestricted mixing or greedy maximizer).
     - Compute $w_k(s, u)$ via closed-form or approximated priority.
   - Apply gradient step on weighted loss: $\theta \leftarrow \theta - \alpha \nabla_\theta \left(\frac{1}{b} \sum_i w_k(s_i, u_i) (Q_k(s_i, u_i) - y_i)^2\right)$.
   - Periodically update $\theta^- \leftarrow \theta$.

Core implementation details: networks employ Adam optimizer, target updates every 200 episodes, replay size $10^4$, minibatch $128$, learning rate $10^{-3}$, TD-$\lambda$ of $0.6$, with code built atop the PyMARL2 QMIX/WQMIX framework using NVIDIA 2080Ti GPUs.

## 6. Theoretical Analysis and Ingredients of Optimal Priority

Four key ingredients characterize optimal MAC-PO priorities:

1. **Bellman-error ($|\delta|$)**: Emphasizes transitions where the Q-estimate is inconsistent with the Bellman target.
2. **Value-enhancement ($\exp(-|Q - Q^*|)$)**: Downweights transitions where $Q_k$ is distant from optimal; highlights areas near the global optimum.
3. **On-policiness ($d^{\pi_k}(s, u) / \mu(s, u)$)**: Favors transitions prevalent under the current policy, modulating off-policy bias.
4. **Joint-action Probabilities**: The term $1 + \sum_{i=1}^n \prod_{j \ne i} \pi^j_k(u_j|s) - n\prod_{i=1}^n\pi^i_k(u_i|s)$ biases toward transitions where one agent’s decision is the unlikely bottleneck, a property unique to MARL.

This combination provably reduces a regret surrogate upper bound, systematically biasing policy improvement toward minimizers of the true regret functional.

## 7. Empirical Validation and Results

MAC-PO was evaluated on two principal benchmarks:

- **Predator-Prey**: A grid-world with 8 predators and 8 prey; performance is measured as average episodic return.
- **SMAC**: StarCraft Multi-Agent Challenge maps, including 3s_vs_5z (standard), 5m_vs_6m (hard), and MMM2 (super-hard); performance is the win rate over 32 evaluation episodes.

### Baselines Compared

| Category                                             | Algorithms              |
|------------------------------------------------------|-------------------------|
| Single-agent replay adapted to MARL                  | PER, PSER, DisCor, ReMERN |
| Value-decomposition MARL                             | QMIX, WQMIX, QPLEX      |
| Actor-critic MARL                                   | VDAC, FOP, DOP          |

### Key Empirical Findings

- **Win rate improvement**: On SMAC, MAC-PO improves final win rates by 4–16% relative to the next-best baseline and shows reduced variance.
- **Sample efficiency**: On Predator-Prey, MAC-PO achieves faster convergence and higher asymptotic returns, particularly under harsh conditions ($p = -1.5$).
- **Approximation efficiency**: The approximate priority scheme performs within a few percent of optimal while dramatically reducing computational cost.
- **Ablation study**: Excluding the joint-probability term on the hardest SMAC map drops win rate by ~18%, affirming its necessity in MARL prioritization.

## 8. Additional Formalism and Implementation Notes

Key computational primitives include:

- **Bellman operator**: $\mathcal{B}^* Q(s, u) = r(s, u) + \gamma \mathbb{E}_{s'}[\max_{u'} Q(s', u')]$.
- **Boltzmann policy**: $\pi^a_k(u_a|s) = \exp(Q^a_k(s, u_a)/\tau) / \sum_{u'} \exp(Q^a_k(s, u')/\tau)$, with temperature $\tau \approx 1$.
- **Weighted loss–priority equivalence**: Non-uniform sampling with weight $w_i$ is equivalent to uniform sampling with per-sample loss scaled by $1/w_i$ [Fujimoto et al. 2020, as cited in 2302.10418].
- **Replay and training details**: Batch size 128, buffer 10,000, learning rate $10^{-3}$, Adam optimizer, target update every 200 episodes, TD-$\lambda = 0.6$.

MAC-PO thus establishes a rigorous, regret-minimizing framework for prioritized multi-agent replay, integrating centralized regret criteria with agent-wise decentralized policies. The combination of classical temporal-difference measures and new multi-agent joint-action structures allows MAC-PO to attain consistent performance advantages over baseline methods across challenging collaborative domains [2302.10418].

Source: https://www.emergentmind.com/topics/multi-agent-regret-weighted-replay-mac-po