---
title: 'CoDiMAD: Diffusion Distillation for Multi-Robot Coordination'
url: https://www.emergentmind.com/topics/codimad
type: topic
---

# CoDiMAD: Diffusion Distillation for Multi-Robot Coordination

Searching arXiv for the specified paper and closely related works mentioned in the provided data.
CoDiMAD, short for Cooperative Diffusion-based Multi-Agent Distillation, is a three-stage privileged distillation framework for communication-free multi-robot coordination under partial observability. It addresses a Dec-POMDP setting in which \(N\) robots act in a shared environment, receive only local sensor observations at execution time, have no inter-agent communication, and must maximize a shared team reward. The framework trains a privileged oracle with MAPPO, constructs an offline dataset of local-observation–oracle-action pairs, and distills the oracle into decentralized students parameterized as conditional denoising diffusion probabilistic models. Its central claim is that, in multi-agent systems with information asymmetry, the conditional oracle-action distribution induced by a fixed local observation is inherently multi-modal, so deterministic distillation collapses valid coordination modes to an invalid average, whereas diffusion-based distillation can model and sample coherent modes [2607.09587].

## 1. Problem formulation and information asymmetry

CoDiMAD is formulated for decentralized multi-robot coordination in a Dec-POMDP
\[
\langle N, \mathcal{S}, \{\mathcal{A}_i\}_{i=1}^N, \{\mathcal{O}_i\}_{i=1}^N, \mathcal{P}, \mathcal{R}, \gamma \rangle.
\]
The global state is \(s_t \in \mathcal{S}\), each agent \(i\) outputs a continuous 2D velocity command \(\mathbf{a}_i^t \in \mathcal{A}_i \subset \mathbb{R}^2\) with components \([v_x, v_y]\), the joint action is \(\mathbf{a}_t \in \mathcal{A}\), the transition kernel is \(\mathcal{P}: \mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})\), and the reward is shared through \(\mathcal{R}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\). Each agent receives only a local observation \(\mathbf{o}_i^t \in \mathcal{O}_i\), and the goal is to learn decentralized policies
\[
\pi_i(\mathbf{a}_i^t \mid \mathbf{o}_i^{1:t})
\]
that maximize
\[
J(\pi) = \mathbb{E}\Big[\sum_{t=0}^{H} \gamma^t \mathcal{R}(s_t,\mathbf{a}_t)\Big].
\]

The information structure is asymmetric between teacher and student. All observations are encoded as \(32\times 32\) bird’s-eye-view grids. The oracle policy receives privileged global information through a dual-stream input. Its local stream \(\mathbf{o}_i^{\mathrm{loc}}\in\mathbb{R}^{2\times32\times32}\) contains a LiDAR occupancy grid and an ego-velocity field. Its global stream \(\mathbf{o}_i^{\mathrm{glo}}\in\mathbb{R}^{3\times32\times32}\) contains an ego-position heatmap over the full arena, a global teammate position map, and a task-state map such as global targets, box position, or evader position. By contrast, the deployable student sees only
\[
\mathbf{o}_i \in \mathbb{R}^{4\times32\times32},
\]
consisting of the same egocentric LiDAR and velocity maps plus cropped local-only maps of teammates and task state within a limited sensing radius [2607.09587].

This asymmetry is the basis for privileged policy distillation. The oracle is a shared deterministic policy trained under centralized training and decentralized execution, with access to global maps; the student is decentralized, sensor-limited, and deployable. The paper states that direct local MARL from partial observations is difficult because exploration is hard in sparse or delayed reward problems, many distinct global states can induce the same local observation, and agents without communication cannot resolve that ambiguity during execution. This suggests that CoDiMAD treats the local student’s learning problem not as direct policy optimization from scratch, but as knowledge transfer from a globally informed expert.

## 2. Multi-modality and deterministic mode averaging

The paper’s key formal observation is that even when the oracle is deterministic at the state level, conditioning on local observation induces a multi-modal action distribution. Let \(\mu_{\pi^O}(s)\) denote the deterministic oracle action at state \(s\), and let the student observe only \(\mathbf{o}_i = h(s)\). If the posterior over global states given a local observation decomposes into \(C\ge 2\) configuration classes \(\mathcal{S}_j\) with weights \(w_j>0\) and \(\sum_j w_j=1\), and if the oracle action within each class is locally Gaussian with mean \(\boldsymbol{\mu}_j\) and covariance \(\sigma_j^2\mathbf{I}\), with separation
\[
\Delta \triangleq \min_{j\neq l}\|\boldsymbol{\mu}_j-\boldsymbol{\mu}_l\| \gg \sigma_{\max},
\]
then the induced conditional action distribution is approximately
\[
p_{\mathcal{D}}(\mathbf{a}\mid \mathbf{o}_i)\approx \sum_{j=1}^{C} w_j\, \mathcal{N}(\mathbf{a};\boldsymbol{\mu}_j,\sigma_j^2 \mathbf{I}),
\]
that is, a well-separated Gaussian mixture [2607.09587].

The examples given are structurally multi-agent. In coverage, a robot with the same local map may need to go left or right depending on teammates’ positions outside its sensor range. In pursuit, a robot may need to rotate clockwise or counterclockwise to encircle the evader depending on the global formation. The local observation therefore aliases multiple global coordination modes.

This leads directly to the failure mode of deterministic distillation. Standard behavior cloning with a deterministic student \(f_\phi(\mathbf{o}_i)\) and an MSE objective minimizes
\[
\mathbb{E}_{(\mathbf{o}_i,\mathbf{a})\sim\mathcal{D}}\|\mathbf{a}-f_\phi(\mathbf{o}_i)\|^2.
\]
For a fixed \(\mathbf{o}_i\), the optimal prediction is the conditional mean. In the two-mode case,
\[
p_{\mathcal{D}}(\mathbf{a}\mid\mathbf{o}_i)=w\,\mathcal{N}(\mathbf{a};\boldsymbol{\mu}_1,\sigma^2\mathbf{I}) +(1-w)\,\mathcal{N}(\mathbf{a};\boldsymbol{\mu}_2,\sigma^2\mathbf{I}),
\]
the MSE-optimal deterministic student is
\[
f^*(\mathbf{o}_i) = w\boldsymbol{\mu}_1+(1-w)\boldsymbol{\mu}_2.
\]
This point lies between the valid modes. Its distance to the nearest mode is
\[
\min_{j\in\{1,2\}} \| f^*(\mathbf{o}_i) - \boldsymbol{\mu}_j \| = m\Delta,\quad m=\min(w,1-w),
\]
and the density at that prediction satisfies
\[
p_{\mathcal{D}}\big(f^*(\mathbf{o}_i)\mid\mathbf{o}_i\big) \le (2\pi\sigma^2)^{-d_a/2}\exp\!\Big(-\frac{m^2\Delta^2}{2\sigma^2}\Big).
\]
Hence, in the well-separated regime \(m\Delta \gg \sigma\), the deterministic prediction has exponentially small density under the oracle-induced conditional distribution [2607.09587].

The practical interpretation given in the paper is that averaging “go left” and “go right” may produce “go straight into obstacle,” and averaging incompatible pushing strategies may produce a motion with incorrect angle or timing. The resulting behavior is described as invalid or hesitant rather than committed to a coherent coordination mode. CoDiMAD is designed specifically to avoid this mode-averaging collapse.

## 3. Three-stage framework

CoDiMAD consists of three stages: oracle training, offline dataset construction, and diffusion-based student distillation [2607.09587].

In Stage 1, a privileged oracle \(\pi^O\) is trained with MAPPO under CTDE. The oracle actor takes the dual-stream observation \((\mathbf{o}_i^{\mathrm{loc}},\mathbf{o}_i^{\mathrm{glo}})\) and outputs a 2D velocity through a diagonal Gaussian policy parameterized by mean and log-standard deviation. The critic is centralized and receives global information. Optimization follows standard MAPPO policy gradient and value loss, with the objective
\[
J(\pi^O) = \mathbb{E}_\tau\left[\sum_t \gamma^t \mathcal{R}(s_t,\mathbf{a}_t)\right],
\]
using PPO-style clipped objectives as in Yu et al., 2022.

In Stage 2, the oracle actor is frozen and executed deterministically by taking the mean of its Gaussian output. For each time step and each agent \(i\), the framework records
\[
(\mathbf{o}_i^t,\mathbf{a}_i^t,d_i^t),
\]
where \(\mathbf{o}_i^t\) is the student’s local observation, \(\mathbf{a}_i^t\) is the oracle action, and \(d_i^t\) indicates episode termination for resetting recurrent state. Only successful, collision-free episodes are retained. Actions are then z-score normalized as
\[
\mathbf{a} \leftarrow (\mathbf{a} - \boldsymbol{\mu}_a)/\boldsymbol{\sigma}_a.
\]

In Stage 3, a shared student policy \(\pi_\theta\) is trained via conditional denoising diffusion over actions. The student conditions on local observation history through a CNN+GRU encoder and models the oracle actions as samples from a conditional distribution \(p_{\mathcal{D}}(\mathbf{a}\mid \mathbf{o})\) using a DDPM. At deployment, each agent runs the same \(\pi_\theta\) independently, uses no communication, and samples actions with fast DDIM inference using 20 denoising steps. The privileged information is thus used only during training, while execution is fully decentralized and communication-free [2607.09587].

A plausible implication is that the framework separates the difficult global coordination learning problem from the deployment constraint. The oracle solves the coordination problem with privileged inputs, and the student learns a local conditional distribution over oracle actions rather than a single regressor.

## 4. Diffusion parameterization, conditioning, and sampling

CoDiMAD applies diffusion directly in continuous action space \(\mathbb{R}^2\). For each normalized oracle action sample \(\mathbf{a}^{(0)}\in\mathbb{R}^2\), the forward diffusion process at step \(k\in\{1,\dots,T\}\), with \(T=200\), is
\[
\mathbf{a}^{(k)} = \sqrt{\bar{\alpha}_k}\,\mathbf{a}^{(0)} + \sqrt{1-\bar{\alpha}_k}\,\boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}),
\]
where \(\bar{\alpha}_k=\prod_{j=1}^k \alpha_j\) is defined by a cosine noise schedule with \(\beta \in [10^{-4}, 0.02]\) and \(s=0.008\). The reverse process is parameterized by a neural network \(\hat{\boldsymbol{\epsilon}}_\theta\) that predicts the added noise from the noisy action and a conditioning context \(\mathbf{c}\) encoding local observation history [2607.09587].

The student architecture has two components. The observation encoder \(f_\theta\) receives \(\mathbf{o}_i^t\in\mathbb{R}^{4\times32\times32}\), uses a CNN to extract spatial features, uses a GRU to maintain temporal hidden state \(\mathbf{h}_i^t\), and outputs a conditioning embedding \(\mathbf{c}_i^t\). The noise-prediction network \(\hat{\boldsymbol{\epsilon}}_\theta\) takes as input the noisy action \(\mathbf{a}^{(k)}\in\mathbb{R}^2\), a sinusoidal embedding of timestep \(k\), and the conditioning vector \(\mathbf{c}_i^t\). It is implemented as a cascade of residual MLP blocks, and the combined conditioning signal \(\mathbf{c}_i^t + \mathbf{t}_k\) is injected into each block in a FiLM-style manner. The output is a 2D estimate of the injected noise.

Training uses the simplified noise-prediction DDPM loss. A batch \((\mathbf{o},\mathbf{a}^{(0)},d)\) is sampled from the offline dataset, observation history is encoded as \(\mathbf{c}=f_\theta(\mathbf{o};\mathbf{h})\), the GRU hidden state is reset when \(d=1\), a diffusion step \(k\) and Gaussian noise \(\boldsymbol{\epsilon}\) are sampled, and the loss is
\[
\mathcal{L}=
\mathbb{E}_{k,\,\boldsymbol{\epsilon},\,(\mathbf{a}^{(0)},\mathbf{o})\sim\mathcal{D}}
\!\left[
(1-\bar{\alpha}_k)
\left\|
\boldsymbol{\epsilon}
-
\hat{\boldsymbol{\epsilon}}_\theta\big(\mathbf{a}^{(k)}, k, \mathbf{c}\big)
\right\|^2
\right].
\]
The factor \((1-\bar{\alpha}_k)\) downweights nearly clean samples and emphasizes high-noise states, where denoising depends more strongly on the observation-conditioned prior [2607.09587].

For execution, full 200-step DDPM reverse sampling is replaced by DDIM with \(K=20\) sub-sampled steps. Time indices \(0=\tau_0 < \tau_1 < \cdots < \tau_K=T\) are predefined, the process is initialized from \(\mathbf{a}^{(\tau_K)}\sim\mathcal{N}(0,I)\), and the DDIM recursion produces \(\mathbf{a}^{(0)}\), which is then denormalized using \((\boldsymbol{\mu}_a,\boldsymbol{\sigma}_a)\) to obtain the final 2D velocity command. The policy is therefore stochastic:
\[
\pi_\theta(\mathbf{a}_i^t \mid \mathbf{o}_i^{1:t}) = \text{DDIM-Sample}\big(\hat{\boldsymbol{\epsilon}}_\theta, f_\theta(\mathbf{o}_i^{1:t})\big).
\]
The paper notes that DDIM sampling is deterministic conditional on the initial noise; different initial noise seeds generate different valid actions or modes, while no communication is required because each agent samples independently from its own local history [2607.09587].

Architecturally, the oracle actor uses a dual-stream CNN encoder, a GRU, and an MLP head outputting mean and log-standard deviation for a diagonal Gaussian over \([v_x,v_y]\). The student uses a single-stream CNN over the local 4-channel BEV, a GRU with hidden size 128 shared across agents, and a diffusion head with 3 residual MLP blocks of hidden dimension 128. Because all environments use continuous 2D actions, no discrete-action adaptation is needed.

## 5. Theoretical characterization of diffusion recovery

The theoretical section contrasts deterministic collapse with diffusion recovery. Proposition 1 formalizes that, under partial observability, the deterministic oracle policy induces a mixture distribution over actions once conditioned only on a local observation. Each mixture component corresponds to a distinct coordination mode. Corollary 1 then shows that deterministic MSE regression returns the conditional mean, which lies between modes and can fall into a region of exponentially small density [2607.09587].

The corresponding positive result is stated in terms of score-based generative modeling. For diffusion step \(k\), let \(q_k(\mathbf{a}^{(k)}\mid\mathbf{c})\) denote the marginal distribution of noisy actions conditioned on the observation embedding \(\mathbf{c}\). The optimal noise-prediction network satisfies
\[
\hat{\boldsymbol{\epsilon}}_\theta^*(\mathbf{a}^{(k)},k,\mathbf{c})
=
-\sqrt{1-\bar{\alpha}_k}\,\nabla_{\mathbf{a}^{(k)}} \log q_k(\mathbf{a}^{(k)}\mid\mathbf{c}).
\]
Given infinite data and model capacity, the reverse diffusion process with exact scores samples from the true conditional distribution \(p_{\mathcal{D}}(\mathbf{a}\mid \mathbf{o}_i)\), following the score-based generative modeling perspective associated with Song et al., 2021. Under the assumptions of Proposition 1, the diffusion-based student can therefore reconstruct the multi-modal conditional action distribution rather than only its mean [2607.09587].

The paper is explicit that practical models have finite capacity and are trained with finite data and finite diffusion steps. Nevertheless, the qualitative advantage remains: diffusion models estimate a conditional distribution, whereas deterministic behavioral cloning estimates only a point prediction. This suggests that CoDiMAD’s performance gains are tied not merely to stochasticity, but to structured sampling from coherent coordination modes.

## 6. Experimental results and qualitative behavior

Experiments are conducted in three cooperative tasks in a \(200\times 200\) continuous arena with \(N=3\) agents and horizon 200: Cooperative Coverage, Pursuit–Evasion, and Box Pushing [2607.09587]. Coverage rewards first-time cell visits, penalizes collisions, and includes a completion bonus. Pursuit–Evasion uses 3 pursuers against 1 APF-controlled evader whose maximum speed is \(2\times\) that of the pursuers; rewards include a capture bonus within radius \(r_e\), distance shaping, and collision penalties. Box Pushing requires 3 robots to push a heavy box to a goal zone, with rewards for moving the box toward the goal, bonuses for simultaneous multi-agent contact, and collision penalties.

The baselines are MAPPO-Oracle as a privileged, non-deployable upper bound; MAPPO-Local as direct local MARL; BC-RNN as deterministic behavior cloning with the same CNN+GRU encoder and an MLP regression head; and CoDiMAD w/o RNN, which uses the same diffusion head without a GRU.

The main quantitative results, reported as mean \(\pm\) standard deviation over 3 seeds, are summarized below.

| Task | Method | Result |
|---|---|---|
| Coverage | Oracle | 97.6% coverage, 0.42 collisions |
| Coverage | MAPPO-Local | 83.7%, 0.71 |
| Coverage | BC-RNN | 80.9%, 10.37 collisions |
| Coverage | CoDiMAD w/o RNN | 94.8%, 0.81 |
| Coverage | CoDiMAD | **95.7%**, 0.55 |
| Pursuit–Evasion | Oracle | 99.1% capture, 0.83 collisions |
| Pursuit–Evasion | MAPPO-Local | 14.2%, 1.89 |
| Pursuit–Evasion | BC-RNN | 90.3%, 3.41 |
| Pursuit–Evasion | CoDiMAD w/o RNN | 84.0%, 1.41 |
| Pursuit–Evasion | CoDiMAD | **90.6%**, **0.80** collisions |
| Box Pushing | Oracle | 98.2% success, 1.18 collisions |
| Box Pushing | MAPPO-Local | 18.0%, 1.17 |
| Box Pushing | BC-RNN | 6.5%, 10.32 |
| Box Pushing | CoDiMAD w/o RNN | 68.9%, 2.39 |
| Box Pushing | CoDiMAD | **72.2%**, 2.37 |

In Coverage, CoDiMAD attains 95.7% coverage with 0.55 collisions, compared with 80.9% and 10.37 collisions for BC-RNN. The paper states that this corresponds to +14.8 percentage points in coverage and a collision reduction from 10.37 to 0.55, approximately \(19\times\), consistent with the mode-averaging explanation. In Pursuit–Evasion, both CoDiMAD and BC-RNN achieve about 90% capture, but CoDiMAD reduces collisions from 3.41 to 0.80, approximately \(4.3\times\) fewer. In Box Pushing, CoDiMAD substantially outperforms both MAPPO-Local and BC-RNN, which the paper attributes to the high sensitivity of the task to precise multi-agent coordination [2607.09587].

The ablations indicate that temporal history matters. CoDiMAD w/o RNN underperforms the full model in all three tasks, and the paper highlights the effect particularly in Pursuit–Evasion, where CoDiMAD w/o RNN is about 6.6 points worse and more collision-prone. Training uses \(T=200\) diffusion steps, while inference uses DDIM with \(K=20\), a 10× reduction that the paper reports as maintaining performance while making runtime feasible.

The qualitative analysis focuses on the learned action distribution. For a fixed local observation in Pursuit–Evasion, the paper selects its 200 nearest neighbors in observation space and plots their oracle actions. The resulting plot shows two well-separated clusters corresponding to two coordination modes. When 200 CoDiMAD samples are drawn from the same conditioning embedding using different initial noise seeds, the samples cover both modes closely. By contrast, BC-RNN produces a single cross located between the clusters, matching the theoretical prediction of deterministic mode averaging. In rollout visualizations from identical initial conditions, five CoDiMAD runs with different diffusion noise produce diverse but coherent trajectories, whereas five BC-RNN runs are identical. The paper interprets this as evidence that CoDiMAD samples meaningful policy modes rather than injecting arbitrary noise [2607.09587].

## 7. Positioning, limitations, and prospective extensions

CoDiMAD is positioned within CTDE and privileged policy distillation. The paper states that privileged distillation has been used in single-robot settings such as vision-based driving and legged locomotion, and contrasts its approach with multi-agent distillation methods such as CTDS and KD-MARL, which mostly rely on deterministic regression. Its stated distinction is twofold: it explicitly analyzes mode-averaging failure under multi-modality and replaces deterministic regression with stochastic generative modeling of actions [2607.09587].

It is also situated relative to diffusion-based decision-making. The paper cites Diffuser, Diffusion Policy, and later work as demonstrating diffusion’s ability to model multi-modal trajectories or policies in single-agent settings, and mentions MIMIC-D as a multi-agent decentralized diffusion-policy method trained from human demonstrations. CoDiMAD’s stated novelty is to use diffusion not merely as a policy class but as a mechanism for privileged multi-agent distillation under partial observability, with supervision coming from a reinforcement-learned privileged oracle rather than human experts. This suggests a conceptual reframing: in communication-free multi-agent distillation, the student’s target may be better understood as conditional density estimation over oracle actions than as deterministic regression.

The paper identifies several limitations. Student performance is bounded by oracle quality, and offline distillation may fail on states outside the oracle’s state-action distribution. Filtering to successful, collision-free trajectories improves supervision quality but may reduce data coverage. CoDiMAD does not include online fine-tuning, so robustness under significant distribution shift is not guaranteed. Diffusion sampling with DDIM at 20 steps remains more expensive than a single forward pass through a deterministic MLP policy, and the paper notes use of an RTX 4090 while suggesting that resource-constrained deployment may require smaller networks or faster generative alternatives. Experiments are limited to \(N=3\) agents with 2D actions, and larger teams or higher-dimensional action spaces may require more expressive denoisers such as attention-based or graph-based architectures. The setup also assumes stationary dynamics, availability of full global state during oracle training, and simulation-only evaluation without physical deployment [2607.09587].

The future directions named in the paper include online fine-tuning of diffusion policies via diffusion-compatible RL algorithms such as IDQL, scaling to larger teams with attention-based or graph-based denoisers, real-world deployment in communication-constrained domains such as marine robotics, and exploration of faster generative backbones such as consistency models [2607.09587]. Within that trajectory, CoDiMAD’s main significance lies in formalizing and operationalizing a specific failure mode of deterministic privileged distillation in multi-agent partial observability: when local observations alias multiple globally distinct coordination contexts, a valid student policy may need to represent and sample from multiple cooperative modes rather than compress them into a single mean action.

Source: https://www.emergentmind.com/topics/codimad