CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning

Abstract: Offline multi-agent reinforcement learning (MARL) enables policy learning from fixed datasets, but is prone to coordination failure: agents trained on static, off-policy data converge to suboptimal joint behaviours because they cannot co-adapt as their policies change. We introduce CODA (Coordination via On-Policy Diffusion for Multi-Agent Reinforcement Learning), a diffusion-based multi-agent trajectory generator for data augmentation that samples conditioned on the current joint policy, producing synthetic experience which reflects the evolving behaviours of the agents, thereby providing a mechanism for co-adaptation. We find that previous diffusion-based augmentation approaches are insufficient for fostering multi-agent coordination because they produce static augmented datasets that do not evolve as the current joint policy changes during training; CODA resolves this by more closely simulating on-policy learning and is a meaningful step toward coordinated behaviours in the offline setting. CODA is algorithm-agnostic and can be layered onto both model-free and model-based offline reinforcement learning pipelines as an augmentation module. Empirically, CODA not only resolves canonical coordination pathologies in continuous polynomial games but also delivers strong results on the more complex MaMuJoCo continuous-control benchmarks.

• The paper introduces a diffusion-based trajectory generator that mimics joint policy evolution in offline multi-agent setups.
• It leverages both classifier and classifier-free guidance to steer synthetic trajectories, ensuring they remain support-constrained to empirical data.
• Empirical results on polynomial games and MaMuJoCo tasks demonstrate enhanced coordination and performance compared to standard offline MARL methods.

Coordination via On-Policy Diffusion in Multi-Agent Offline Reinforcement Learning: An Expert Analysis of CODA

Motivation and Problem Setting

The study "CODA: Coordination via On-Policy Diffusion for Multi-Agent Offline Reinforcement Learning" (2604.23308) addresses a critical limitation in cooperative multi-agent offline RL: coordination failure arising from the lack of joint policy evolution in fixed-data regimes. Unlike online MARL, where agents interactively update policies and co-adapt, offline methods are constrained to a static dataset sampled from an unknown behavior policy $\boldsymbol{\pi}_{\mathrm{off}}$. This breaks the adaptation feedback loop fundamental to multi-agent coordination and leads to miscalibrated, incompatible agent behaviors even under the Centralized Training with Decentralized Execution (CTDE) paradigm.

The standard offline approach employs Best Response Under Data (BRUD) updates, in which each agent optimizes against the dataset's static teammate actions. This approach can drive convergence to suboptimal joint equilibria, a phenomenon the authors analytically and empirically dissect. The work targets this non-adaptivity by proposing a mechanism to approximate joint policy evolution strictly offline, via trajectory-level generative augmentation.

The CODA Algorithm: Conceptual Overview

CODA introduces a diffusion-based trajectory generator for policy-dependent synthetic data augmentation. CODA's key insight is to conditionally sample joint trajectories from a generative model—trained on the offline dataset—such that the sampled trajectories are biased towards regions of the trajectory space with higher likelihood under the current joint policy $\boldsymbol{\pi}_{\text{curr}}$. This creates a policy-dependent pseudo-distribution that mimics the feedback-driven joint policy evolution characteristic of online MARL, thereby restoring co-adaptation signals.

Unlike prior diffusion-based MARL augmentations, which produce static proxy datasets, CODA employs either:

• Classifier Guidance: When the policy is high-dimensional, classifier-based gradients of the joint log-policy (optionally regularized by a scale $\lambda$) are used to steer trajectory sampling, leveraging techniques from Policy-Guided Diffusion (PGD).
• Classifier-Free Guidance (CFG): When a compact policy descriptor is available, trajectories are conditioned directly on this representation during both training and generation, interpolating conditional/unconditional diffusion scores.

Both approaches modify the sampling process so that the synthetic trajectories remain support-matched to the empirical dataset but are increasingly likely under the evolving joint policy. This process is strictly algorithm-agnostic and can be layered over standard offline model-free or model-based RL methods.

Mechanistic Details and Theoretical Justification

CODA models the full-horizon joint trajectory using a centralized 1D temporal U-Net backbone, diffusing $(s_0,\mathbf{o}_0,\mathbf{a}_0,r_0, ..., s_H)$ as a single object. Policy-conditioned generation is realized with a practical guided score:

$$\nabla_{\hat{\tau}} \log p(\hat{\tau}; \sigma) + \lambda \sum_{t=0}^{H-1} \nabla_{\hat{\mathbf{a}}_t} \log \boldsymbol{\pi}_{\text{curr}}(\hat{\mathbf{a}}_t | \hat{\mathbf{o}}_t)$$

This targets the surrogate distribution:

$$\tilde{p}_{\boldsymbol{\pi}_{\text{curr}}}(\tau) \propto p_{\boldsymbol{\pi}_{\text{off}}}(\tau) \exp \left( \lambda \sum_{t=0}^{H-1} \log \boldsymbol{\pi}_{\text{curr}}(\mathbf{a}_t|\mathbf{o}_t) \right)$$

where $p_{\boldsymbol{\pi}_{\text{off}}}$ is the offline behavioral prior. This form ensures all generated experience is anchored to empirical support—if the current policy drifts out-of-distribution, the guidance saturates but remains support-constrained, mitigating extrapolation error.

Crucially, the gradient term is joint: all agents are simultaneously updated according to the gradient of the combined policy log-likelihood, formally capturing cross-agent coordination.

Empirical Results: Polynomial Games and MaMuJoCo

CODA is evaluated on both low-dimensional polynomial games (where coordination and miscoordination can be diagnosed directly), and on high-dimensional continuous locomotion tasks (MaMuJoCo).

Polynomial Games

The multiplication game and the Twin Peaks game—parametric, continuous-action analogues of classic coordination matrix games—exhibit intrinsic multi-agent optima with well-characterized failure modes for BRUD-style algorithms. CODA, through on-policy diffusion, enables convergence to the globally optimal joint strategy, while all baselines (including Q-guided diffusion) collapse to suboptimal solutions dictated by dataset statistics or action boundary effects.

Figure 1: Comparing policy evolution for the multiplication game—CODA with on-policy conditioning achieves optimum returns, unlike MADDPG and prior diffusion augmentations.

Figure 2: In Twin Peaks, CODA uniquely mitigates miscoordination by aligning agents' updates with the evolving joint policy; baselines collapse to centrally located, non-coordinated outcomes.

MaMuJoCo

In the higher-dimensional 2HalfCheetah and 4Ant settings, CODA's guidance scale $\lambda$ robustly modulates the on-policy likelihood of synthetic trajectories. Empirically, increasing guidance (up to a tuned maximum) increases the probability that sampled rollouts are compatible with the current policies, also reflected in improved downstream performance when used for policy learning.

Figure 3: On-policy log-likelihood of CODA-generated trajectories increases with guidance, confirming effective policy steering.

CODA delivers pronounced gains in dataset regimes with moderate-to-high expert trajectory support (Replay, Good datasets), achieving the highest mean normalized test returns among all methods. Performance gains diminish in low-support datasets (Poor, Medium), where support limitations preclude effective guidance.

Figure 4: CODA outperforms unconditional diffusion and baseline augmentation on MaMuJoCo tasks when dataset support suffices; standard errors shown across seeds.

Performance breakdown by dataset quality further highlights that CODA's benefit is maximized when policy guidance can meaningfully bias sampling toward regions of high on-policy return while remaining within the empirical support.

Figure 5: Return dynamics for 2HalfCheetah with Poor dataset—CODA's augmentation mitigates miscoordination but is constrained by support.

Figure 6: Replay dataset—sustained gains from CODA-guided rollouts in multi-agent offline RL.

Figure 7: Medium dataset—guidance benefit intermediate, reflecting support limitations.

Figure 8: Good dataset—CODA yields the highest normalized return, closely matching ground-truth expert boundary.

Implications and Prospects

CODA formalizes a practical, scalable mechanism for approximating joint policy evolution in strictly offline multi-agent RL, directly addressing the central challenge of restoring adaptation and coordination without environment interaction. The algorithm demonstrates strong, numerically robust improvements over existing methods in both controlled theoretical and complex empirical regimes.

Key implications include:

• Theoretical: CODA's strategy provides a generic bridge between generative modeling and policy optimization in multi-agent settings. Conditioning generative models on agent policies offers a flexible path for simulating non-stationary adaptation even under severe data constraints.
• Practical: Algorithm-agnostic augmentation via CODA can be seamlessly integrated with both model-free and model-based RL pipelines, extending the practical usability of offline MARL in domains where interaction is costly, risky, or infeasible.
• Limitations: Benefits plateau or degrade if dataset support is insufficient to align the synthesized distribution with the evolving policy, suggesting the need for adaptive or hybrid approaches, potentially mixing model-based rollout with diffusion conditioning.
• Future Directions: The authors suggest adaptive guidance scheduling, mixing with model-based imagination, and exploring connections to trust-region and safety-constrained learning under severe OOD risks.

Conclusion

CODA marks a substantial advance in the domain of multi-agent offline RL by operationalizing on-policy co-adaptation via joint trajectory diffusion. Its design rigorously addresses the previously-unstudied distributional mismatch intrinsic to standard offline MARL learning algorithms, producing strong empirical and theoretical evidence for the necessity of policy-dependent augmentation. Extending this paradigm opens new directions for robust, support-aware agent coordination in non-interactive learning settings.