CERMIC: Calibrated Curiosity for MARL

• CERMIC is a plug-and-play exploration module that calibrates intrinsic curiosity using inferred multi-agent context to enhance exploration in decentralized, partially observable reinforcement learning.
• It employs an information bottleneck formulation and robust optimization to distinguish socially meaningful transitions from noisy surprise, addressing challenges like the Noisy-TV problem.
• Empirical evaluations on VMAS, Melting Pot, and SMACv2 show that integrating CERMIC with methods like MAPPO and QMIX significantly improves performance in sparse-reward settings.

CERMIC, expanded as Curiosity Enhancement via Robust Multi-Agent Intention Calibration, is an exploration module for decentralized, communication-free, partially observable multi-agent reinforcement learning (MARL) with sparse rewards. The paper title uses the stylized form “Cermic,” while the method section and algorithm label use “CERMIC/Cermic,” all referring to the same approach. Its central claim is that intrinsic motivation in MARL should not treat all novelty uniformly: curiosity should be calibrated by inferred multi-agent context so that agents suppress noisy surprise and preferentially explore socially meaningful, information-rich transitions. The module is presented as a lightweight plug-and-play component for base learners such as MAPPO and QMIX, and is evaluated on VMAS, Melting Pot, and SMACv2 (Pan et al., 25 Sep 2025).

1. Problem setting and conceptual motivation

CERMIC addresses a specific failure mode of curiosity-driven exploration in MARL. Standard intrinsic-motivation schemes often rely on prediction error, uncertainty, or surprise, but in sparse-reward settings these signals are vulnerable to environmental stochasticity and the classic Noisy-TV problem. The paper also argues that existing curiosity methods exhibit a uniform novelty bias: they reward unexpected observations without distinguishing between irrelevant randomness and novelty induced by other agents’ behavior. In decentralized MARL, this distinction is consequential because peer behavior novelty can encode latent task structure, coordination opportunities, or opponent intentions (Pan et al., 25 Sep 2025).

The target setting is a communication-less partially observable MARL problem modeled as a POMDP

$$(\mathcal{O}, \mathcal{A}, \mathbb{P}, \mathcal{R}, N, \gamma).$$

Each agent receives a local observation $o_t^i$, takes an action $a_t^i$, and all agents receive a shared extrinsic reward $r_t^e$. The method is described from the perspective of one agent, with the agent index omitted for brevity. Within that setting, CERMIC asks not only whether a transition is surprising, but whether the surprise remains meaningful after calibration by inferred social context.

The motivating analogy is developmental rather than mechanistic: the method is inspired by the idea that human children adapt exploratory behavior by observing peers. In algorithmic form, this becomes contextual calibration. Raw curiosity is filtered and rescaled using a latent contextual feature $f_n^o$ inferred from other agents, so that surprise aligned with informative peer behavior is amplified while noise-like surprise is attenuated.

2. Information-bottleneck formulation and calibrated surprise

CERMIC is built on the Information Bottleneck (IB) principle. In general form, IB seeks a compressed representation $Z$ of input $X$ that preserves information about target $Y$:

$\max I(Z;Y)-\alpha I(X;Z).$

For exploration, the paper defines a latent variable $X_t$ extracted from the current state-action pair $o_t^i$0 and optimizes

$o_t^i$1

The first term rewards predictive information about the next state and therefore targets informative novelty rather than arbitrary novelty. The second term penalizes excessive dependence on the current state-action pair, enforcing compression and filtering (Pan et al., 25 Sep 2025).

The exploration term $o_t^i$2 is lower-bounded variationally. With a predictor $o_t^i$3, the resulting training objective is

$o_t^i$4

This makes exploration equivalent to learning a latent state that preserves predictive information about future transitions.

The compression term is handled through a Gaussian approximation $o_t^i$5, yielding

$o_t^i$6

with

$o_t^i$7

Here $o_t^i$8 functions as a raw surprise or compression-cost variable. Rather than minimizing it directly, the paper imposes a distributionally robust chance constraint requiring $o_t^i$9 to remain within a calibrated confidence region with high probability.

The distinguishing step is to make the ambiguity set context dependent. Instead of using a static global threshold, CERMIC centers the uncertainty set around a context-conditioned mean $a_t^i$0. The robust optimization is then reduced, via Cantelli’s inequality, to two tractable penalties:

$a_t^i$1

and

$a_t^i$2

The upper term suppresses excessively large surprise; the lower term prevents collapse to a trivial low-surprise representation. The final loss is described as minimizing the two calibrated exploitation penalties while maximizing the exploration objective, weighted by $a_t^i$3.

3. Intention inference and multi-agent contextual calibration

CERMIC’s calibration signal is derived from an inferred multi-agent context. The module contains observation and state encoders, a latent exploratory model $a_t^i$4, a variational predictor $a_t^i$5, an intention/context inference module, a calibration function, and an intrinsic reward module. The next-observation encoder uses momentum-updated parameters $a_t^i$6 for stability (Pan et al., 25 Sep 2025).

The default context-inference implementation uses a dynamic graph $a_t^i$7. Nodes encode predictive state representations of agents; edges encode relative spatial relationships; a memory queue stores intention graphs over time; an MLP-based detector determines whether peers are observed and whether graph memory should be updated; and a GNN processes the resulting graph into the contextual feature $a_t^i$8. The paper also studies GRU-based memory, but graph memory performs best in ablations.

Because contextual inference can be unreliable early in training or under partial observability, CERMIC introduces a task-adaptive reliability scalar

$a_t^i$9

and defines the calibrated mean through

$r_t^e$0

The interpretation given in the paper is explicit: if inferred intentions are inaccurate, or if they do not align with reward-relevant task dynamics, then $r_t^e$1 is small and context has little influence; if peer behavior and task rewards are consistent, then $r_t^e$2 increases and context meaningfully calibrates curiosity.

Because the mutual information term is intractable, the paper estimates it with an InfoNCE-style bound:

$r_t^e$3

The score function is implemented as bilinear plus softmax. Positive samples come from the memory module, while negative samples are generated by adding noise to $r_t^e$4. This construction makes the calibration factor depend on whether the inferred social context is predictive of current task-relevant dynamics rather than merely present.

The context module itself is pretrained with auxiliary objectives. The detector uses binary cross-entropy, the node encoder predicts another agent’s latent state, and the edge encoder predicts relative positions. These auxiliary tasks make “intention calibration” operational rather than metaphorical: CERMIC does not observe communication channels or global state, but it does attempt to infer a structured belief over other agents from local observations alone.

4. Intrinsic reward and theoretical grounding

CERMIC’s intrinsic reward is based on Bayesian surprise over the module’s own parameters $r_t^e$5:

$r_t^e$6

Here $r_t^e$7 contains transitions from the past $r_t^e$8 episodes. A transition is intrinsically rewarding when it would substantially update the agent’s belief about the CERMIC model parameters. The total reward is then

$r_t^e$9

This gives the intrinsic bonus an information-gain interpretation rather than a pure prediction-error interpretation (Pan et al., 25 Sep 2025).

Because the actual module is implemented with non-Bayesian neural networks, the appendix derives a practical approximation using a learned representation $f_n^o$0 and the data-processing inequality. The intractable marginal over $f_n^o$1 is then approximated by a standard Gaussian,

$f_n^o$2

This approximation aligns the intrinsic-reward computation with the earlier Gaussian variational treatment used in the IB compression term.

The paper also gives a more formal theoretical justification in linear MDPs. Under a linear dynamics model $f_n^o$3, with Gaussian priors on parameters and Gaussian transition noise, the intrinsic reward is shown to be sandwiched by a UCB-style exploration bonus:

$f_n^o$4

The corresponding KL information gain reduces to

$f_n^o$5

Using

$f_n^o$6

the paper connects $f_n^o$7 to the standard uncertainty bonus form $f_n^o$8. The theorem is explicitly limited to the stated assumptions—linear MDPs and Gaussian models—but it supplies a rationale for why CERMIC’s intrinsic reward should decay with uncertainty rather than remain permanently destabilizing.

5. Training pipeline, empirical evaluation, and observed behavior

Operationally, CERMIC follows a fixed per-episode routine. Observations $f_n^o$9 are encoded into $Z$0; a latent exploratory state is sampled as $Z$1; the next-state embedding is predicted to compute $Z$2; the raw surprise term $Z$3 is evaluated; social context $Z$4 is inferred from graph memory over detected peers; the reliability factor $Z$5 is estimated with InfoNCE; robust exploitation losses are computed; the combined CERMIC loss is optimized; the intrinsic reward is added to extrinsic reward; and the actor-critic is updated from collected experiences. CERMIC is integrated into MAPPO and QMIX. Reported implementation details include Adam with learning rate $Z$6, Adam epsilon $Z$7, discount factor $Z$8, soft target update $Z$9, maximum frames $X$0, on-policy collection with 36,000 frames per batch, 60 envs per worker, minibatch size 2400, 30 minibatch iterations, off-policy train batch size 1024, replay buffer 1,500,000, and training via BenchMARL on NVIDIA Quadro RTX 8000 GPUs (Pan et al., 25 Sep 2025).

The benchmark suite comprises 9 VMAS tasks, 4 Melting Pot tasks, and 2 SMACv2 tasks. VMAS uses mean episodic reward, Melting Pot uses mean episodic return, and SMACv2 uses mean test win rate. Several environments are converted to explicitly sparse-reward settings by removing dense shaping rewards or intermediate combat rewards. The baseline set includes MAPPO, QMIX, MAPPO-$X$1-greedy, CPM, QMIX-SPECTra, MAPPO-DB, MACE, and QPLEX-ICES.

The main empirical claim is that CERMIC-augmented methods achieve state-of-the-art performance on 12 of 16 scenarios. MAPPO-Cermic is usually the strongest overall, while QMIX-Cermic also improves substantially over plain QMIX. The strongest gains occur on Melting Pot, where rewards depend on emergent, dynamic inter-agent interactions rather than fixed rules. Concrete examples reported in the table include Balance, where MAPPO-Cermic reaches **67.3* versus MAPPO 60.3, MAPPO-DB 55.2, MACE 60.0, and ICES 63.2; Wheel, where MAPPO-Cermic reaches *$X$2** versus MAPPO $X$3 and MAPPO-DB $X$4; and SMACv2 results of 0.70 on protoss5v5* and 0.48 on zerg5v5* for MAPPO-Cermic.

The qualitative analysis aligns with those aggregate results. State embeddings and latent states indicate that CERMIC agents make more close approaches and intersections, suggesting active motion toward observing peers rather than blind imitation. Latent states $X$5 are reported as more dispersed around current state embeddings and concentrated near trajectory endpoints, indicating heightened curiosity before behavioral shifts. Inter-agent encounters often trigger intrinsic reward spikes. The paper’s interpretation is that CERMIC becomes curious about both environmental novelty and social novelty, but calibrates the latter rather than treating every peer-induced perturbation as mere noise.

6. Ablations, limitations, and position within MARL exploration research

The ablation studies clarify which parts of the method are load-bearing. Removing $X$6 significantly hurts adaptation to sparse rewards and slows learning. Removing the exploitation losses $X$7 makes the system unstable, reduces noise filtering, and produces higher variance and fluctuations. Smaller $X$8 values enforce stricter chance constraints, yielding more conservative exploration, lower variance across seeds, and lower mean performance. Larger $X$9 increases the emphasis on the exploration-calibration tradeoff, tends to reduce immediate task return, and improves stability while lowering variance. Graph-based memory outperforms GRU-based memory, and pretrained detection/intention modules significantly accelerate convergence (Pan et al., 25 Sep 2025).

The robustness experiments introduce “random box” environmental noise. Under that perturbation, CERMIC maintains more stable curves, narrower error bands, and a stronger relative advantage than in noise-free settings. In transfer-style evaluation, agents trained under one reward density and evaluated under sparse rewards degrade less severely than competing methods. The paper interprets this as evidence that CERMIC helps agents infer task structure from peer behavior rather than depending only on dense reward shaping.

The limitations are stated directly. Intention modeling remains hard in communication-free, partially observable settings; early learning can be slower because intention models must be learned from scratch; scalability concerns arise from graph construction and the one-step latent transition model in more complex or multimodal tasks; and performance still depends strongly on the quality of contextual inference, even though $Y$0 mitigates unreliable context. Future extensions proposed in the paper include better pretrained intention modules and even LLM-like components for bootstrapping inference.

Within the broader MARL literature, CERMIC is positioned against three families of alternatives. Relative to standard curiosity methods, it rejects uniform novelty bonuses in favor of socially contextualized calibration. Relative to centralized or communication-based exploration schemes, it is designed for decentralized execution without privileged global state or explicit communication. Relative to uncertainty-based MARL bonuses, it attempts to correct miscalibration under partial observability by conditioning surprise on inferred intentions and task-aligned context. A plausible implication is that CERMIC is most consequential in environments where other agents’ behavior is itself a primary carrier of task information; the reported Melting Pot results are consistent with that interpretation.