Quality-Aware Exploration Budget Allocation for Cooperative Multi-Agent Reinforcement Learning

Abstract: Cooperative multi-agent reinforcement learning (MARL) requires agents to discover joint strategies in a combinatorially large state-action space, yet effective coordination configurations are exceedingly rare. Intrinsic motivation, which augments task rewards with novelty bonuses, is a popular approach for driving exploration, but its effectiveness hinges on the exploration intensity $β$, where too large a value overwhelms the task signal and causes coordination collapse, while too small a value prevents discovery of rare strategies. We address two complementary challenges: adapting $β$ globally over training, and allocating the exploration budget across agents whose intrinsic reward signals vary in reliability. Our framework combines a return-conditioned sigmoid schedule (RCB) for global intensity control with a per-agent Reward Signal Quality (RSQ) metric that concentrates the exploration budget on agents with reliable signals. The core insight is that agents receiving noisy intrinsic rewards should explore less aggressively, and this allocation can be determined automatically from signal-to-noise statistics. Successor Distance (SD), a quasimetric intrinsic reward, naturally produces distinguishable per-agent signal quality, completing the framework with convergence and ordering preservation guarantees. On seven cooperative benchmarks (MPE, SMAX, MABrax), our method achieves top-tier returns across all environments.

• The paper introduces a framework combining Return-Conditioned Beta scheduling with Reward Signal Quality allocation to adapt global exploration and prioritize agents whose intrinsic rewards have higher signal-to-noise ratios.
• The method uses Successor Distance rewards and clipped affine modulation, achieving positive corridor returns (+190 ± 224), leading results on tag (+4930 ± 404), and strong performance across seven cooperative benchmarks.
• The results show that quality-aware allocation prevents noisy agents from destabilizing coordination, while limitations include dependence on Successor Distance, shared team rewards, and approximate policy equilibration assumptions.

Motivation and problem statement

Intrinsic motivation is the dominant paradigm for driving exploration in cooperative multi-agent reinforcement learning (MARL): each agent's reward is augmented as $r_i = r^{ext} + \beta \cdot r_i^{int}$, where the exploration intensity $\beta$ governs how strongly novelty bonuses influence learning. The authors of this paper observe that nearly all additive intrinsic-motivation methods apply a fixed $\beta$ to every agent throughout training, leaving two unresolved questions: how $\beta$ should adapt over the course of learning, and how the exploration budget should be distributed across agents whose intrinsic rewards differ in reliability. The paper argues that these are not secondary tuning concerns but first-order determinants of success — an agent whose novelty signal is dominated by noise can destabilize team coordination regardless of how much exploration budget it receives, while a globally mis-scaled $\beta$ either overwhelms the task signal or fails to discover rare coordinated configurations.

The proposed framework addresses both questions with three components. A Return-Conditioned Beta (RCB) schedule adapts global intensity via a sigmoid of team return, with provable convergence. A Reward Signal Quality (RSQ) metric allocates per-agent budget based on the signal-to-noise ratio (SNR) of each agent's intrinsic reward stream, preserving the ordering of the information-theoretically optimal allocation. Successor Distance (SD) (2605.01865), a quasimetric intrinsic reward measuring temporal reachability between states, supplies the per-agent signal quality variation that RSQ requires.

Return-Conditioned Beta schedule

RCB tracks team return through an exponential moving average $R_{\mathrm{ema}}$ and sets

$$\beta^{(k)} = \beta_{\min} + (\beta_{\max} - \beta_{\min}) \cdot \sigma\!\left(\kappa(R_{\mathrm{target}} - R_{\mathrm{ema}}^{(k)})\right),$$

so that $\beta$ saturates near $\beta_{\max}$ early in training and decays toward $\beta_{\min}$ as returns improve. Under an assumption that steady-state return $\beta$0 is continuous and bounded, and that observed returns are noisy around it, the RCB feedback loop ($\beta$1 return $\beta$2) forms a contraction whenever

$\beta$3

where the factor $\beta$4 comes from the maximum slope of the logistic sigmoid. Banach's fixed-point theorem then yields a unique equilibrium, geometric contraction of tracking error at rate $\beta$5, and an explicit stochastic noise floor of order $\beta$6. The closed-form Lipschitz constant gives practitioners a verifiable hyperparameter inequality before training; empirically, with $\beta$7, the product stays near $\beta$8 on SMAX, far below unity.

A notable robustness claim is that $\beta$9 and $\beta$0 are not precision-sensitive: because the sigmoid transition spans roughly $\beta$1 return units, shifting $\beta$2 by hundreds of units barely changes the $\beta$3 trajectory. Sensitivity sweeps on corridor confirm that all values of $\beta$4 in $\beta$5 yield positive mean return, with only extreme values ($\beta$6) destabilizing training.

Reward Signal Quality and water-filling allocation

For per-agent allocation, the paper models each agent's intrinsic reward as a noisy channel carrying signal power $\beta$7 against noise variance $\beta$8. Under a Gaussian channel model — justified as conservative, since the Gaussian minimizes mutual information among distributions with given moments — maximizing total information gain subject to a squared-intensity budget recovers classical water-filling:

$\beta$9

Rather than implementing exact water-filling, which assigns zero budget to agents whose estimated SNR falls below the water level, the authors use a clipped affine modulation:

$\beta$0

with defaults $\beta$1 (corresponding to SNR = 1), $\beta$2, and $\beta$3. A proposition establishes that this affine scheme preserves the SNR-ordering of the optimal allocation, with strict ordering whenever both weights are unsaturated. The empirical case for avoiding exact water-filling is strong: on corridor, exact water-filling yields a mean return of $\beta$4 versus $\beta$5 for the affine modulation, because noisy EMA-based SNR estimates trigger irreversible suppression of agents whose exploration is actually needed for coordination.

RSQ is invariant to positive rescaling of intrinsic rewards, eliminating per-task calibration, and six of ten framework hyperparameters are shared across all seven environments.

Successor Distance as the enabling intrinsic reward

RSQ can differentiate agents only if the intrinsic reward itself produces distinguishable per-agent signal quality. SD satisfies this condition naturally: an agent exploring open space receives consistent, large temporal-distance values (high $\beta$6, low $\beta$7, high RSQ), while an agent whose experience depends on teammates' behavior — e.g., one stuck at a bottleneck — receives small, erratic values (low RSQ). Each agent maintains its own encoder trained with symmetric InfoNCE on local state features, using the intrinsic reward $\beta$8.

This dependence is also the framework's principal structural constraint. Replacing SD with policy entropy, RND, or count-based rewards produces near-uniform RSQ across agents, collapsing all modulation weights to $\beta$9 and yielding strongly negative returns on corridor ($\beta$0, $\beta$1, and $\beta$2 respectively, versus $\beta$3 for SD). The formal requirement is stated as Assumption 4 (a minimum pairwise RSQ gap $\beta$4), and the paper concedes plainly that only SD has been shown to satisfy it.

Experimental results

Evaluation covers seven cooperative benchmarks implemented in JaxMARL: MPE corridor (8 agents through a narrow bottleneck) and tag (6 predators vs. 2 scripted prey); SMAX combat tasks 3s5z (8 vs. 9 agents) and 27m (27 vs. 30); and three continuous MABrax locomotion tasks (ant_4x2, ant_ball, halfcheetah_6x1). All methods share domain-standard backbones (MAPPO for MPE/SMAX, IPPO for MABrax), use 10 seeds per configuration, and are compared against eight baselines including COIN, MAVEN, a Lagrangian advantage-balancing method, tuned fixed-$\beta$5 Linear, and ablated RCB-only and RSQ-only variants. Significance is assessed with Welch's $\beta$6-test at $\beta$7.

Method	Corridor	Tag	3s5z	27m	ant_4x2	ant_ball	hc_6x1
MAPPO/IPPO	−1477 ± 288	+3622 ± 565	0.542 ± 0.009	0.430 ± 0.015	1659 ± 41	830 ± 158	2720 ± 114
COIN	−2729 ± 388	+216 ± 663	0.193 ± 0.017	0.120 ± 0.034	2724 ± 427†	1702 ± 173†	1678 ± 476
MAVEN	−1584 ± 432	+2329 ± 374	0.450 ± 0.094	0.238 ± 0.039	2282 ± 270	1498 ± 128	2398 ± 103
Lagrangian	−1238 ± 1019	+2400 ± 180	0.200 ± 0.056	0.258 ± 0.065	2623 ± 427†	1697 ± 292†	2874 ± 141
Linear	−259 ± 708†	+3542 ± 410	0.585 ± 0.013†	0.449 ± 0.003	2509 ± 341	1540 ± 182	2746 ± 114
Ours	+190 ± 224	+4930 ± 404	0.591 ± 0.014	0.447 ± 0.005†	2940 ± 312	1846 ± 261	2870 ± 113†

(† denotes statistical tie with the best; IPPO row omits dashes for MPE/SMAX where it is not applicable.)

Several results stand out. On corridor, the proposed method is the only tested method achieving positive final returns, with $\beta$8 lower standard deviation than the second-best Linear baseline — the reliability property the authors attribute to the interaction of per-agent attenuation and global scheduling. On tag, it leads by a wide margin (+4930 versus +3823 for RSQ-only). On SMAX-27m, it is statistically indistinguishable from Linear at convergence, but the decisive contrast is exploration safety: without RSQ, raising fixed $\beta$9 beyond 0.15 drives mean return below 0.2 at $R_{\mathrm{ema}}$0 and near zero at 0.5, whereas the full framework operates safely at $R_{\mathrm{ema}}$1 by selectively attenuating noisy agents. This demonstrates that RSQ enables exploration intensities that would otherwise cause training collapse — the paper's central practical claim.

Ablations clarify component roles. Neither RCB-only nor RSQ-only matches the combination on corridor. RSQ-only is the dominant component on SMAX-27m ($R_{\mathrm{ema}}$2, statistically tied with the full method), where the narrow return range keeps $R_{\mathrm{ema}}$3 nearly constant and limits RCB's contribution; on wide-return-range tasks both components contribute actively. Any nonzero modulation strength $R_{\mathrm{ema}}$4 improves corridor performance over $R_{\mathrm{ema}}$5, with $R_{\mathrm{ema}}$6 best and $R_{\mathrm{ema}}$7 lower standard deviation. RSQ dynamics analysis shows meaningful differentiation on corridor (inter-agent RSQ gaps exceeding 0.3 during active exploration) and symmetric roles on tag and halfcheetah producing near-uniform RSQ — consistent with halfcheetah being the one environment where the method merely ties Lagrangian rather than leading.

Limitations and open questions

The paper is candid about three constraints. First, RSQ requires an intrinsic reward that produces per-agent signal quality variation; among the sources tested, only SD does so, and extending RSQ to a broader family of intrinsic rewards remains open. Second, the framework assumes cooperative settings with a shared team reward, excluding competitive and mixed-motive scenarios. Third, the convergence guarantee rests on Assumption 1 — that the policy approximately equilibrates at each $R_{\mathrm{ema}}$8 before the schedule updates — a standard but nontrivial timescale-separation condition whose violation under fast policy updates is not analyzed. Dynamic team compositions and topology-aware intrinsic designs that exploit communication-graph structure are identified as specific untested extensions.

Conclusion

This paper reframes exploration management in cooperative MARL as a budget-allocation problem over noisy channels, combining a convergent return-conditioned global schedule with an SNR-based per-agent modulation that provably preserves the ordering of the information-optimal allocation. The empirical record — top-tier or tied returns on all seven benchmarks, uniquely positive returns on corridor, and safe operation at intensities that collapse unmodulated baselines — supports the claim that signal quality, not just signal design, is a controlling variable in multi-agent exploration. The framework's dependence on SD's quasimetric structure for quality differentiation is the main boundary condition on its generality.