Decentralized Diffusion Models (DDM)

Updated 30 May 2026

DDM is a framework where independent experts, each trained on a unique data partition using flow-matching or denoising objectives, are dynamically combined via learned routing.
It employs domain-specific transformer architectures and sparse Top-m routing to enhance perceptual quality and computational efficiency in applications like image, video, and multi-agent systems.
The system guarantees recovery of global diffusion behavior while providing resilience to hardware failures and scalability across distributed compute resources.

A Decentralized Diffusion Model (DDM) refers to a family of generative, learning, or decision systems in which constituent models—termed "experts" or "agents"—are trained or operate independently on non-overlapping subsets of data or in separate environments, without centralized synchronization or parameter sharing. At inference or deployment, these independent experts are dynamically combined using a learned routing mechanism or aggregation principle that allows the overall system to match or exceed the performance of monolithic, centrally trained diffusion models. DDM methods have been developed for high-dimensional data synthesis (images, video), nonparametric statistical learning over networks, multi-agent reinforcement learning, and biological collective decision-making.

1. Formal Definition and Canonical Training Setup

The canonical DDM is defined by the partitioning of a dataset $\mathcal{D}$ into $K$ disjoint clusters $\mathcal{C}_1, \dots, \mathcal{C}_K$ , each serving as the domain for an expert diffusion model $v^{(k)}_\theta$ . Expert $k$ is trained only on data from cluster $\mathcal{C}_k$ by minimizing a flow-matching or denoising objective, typically independently and asynchronously from the other experts.

Let $x_0$ denote clean input data, and $x_t = \alpha_t x_0 + \sigma_t \epsilon$ be the noised state at timestep $t$ (with schedule $\alpha_t, \sigma_t$ ). Each expert minimizes:

$K$ 0

where $K$ 1 is the target conditional flow or velocity field. No gradients, parameters, or activations are exchanged between experts during training. Each expert can run on separate compute hardware ("compute islands" (McAllister et al., 9 Jan 2025)).

A lightweight router model $K$ 2 is trained—often post hoc—to assign weights $K$ 3 over the experts for any given $K$ 4, either by classification or Bayesian unsupervised techniques.

At inference, the overall denoising or velocity field is a convex aggregation:

$K$ 5

from which noise-to-data trajectories are integrated via high-order ODE or SDE solvers.

2. Model Architectures and Routing Mechanisms

Paris 1.0 and 2.0 exemplify the expert-plus-router DDM architecture for image and video generation, respectively (Jiang et al., 3 Oct 2025, Rouzbayani et al., 25 May 2026). Each expert is a large diffusion backbone (DiT-XL/2, MM-DiT, etc.), typically a transformer with domain-specific extensions (e.g., spatiotemporal attention for video). Training is done on latent-space representations, e.g., HunyuanVAE for video, to enforce temporal or spatial coherency and efficiency.

The router is a parameter-efficient transformer or small CNN that receives the current noised latent and timestep (optionally with prompt embeddings), and outputs a softmax over the $K$ 6 experts. Inference employs Top- $K$ 7 routing (sparse, e.g., Top-1 or Top-2), full ensemble (all experts), or other weighted combinations. Sparse routing is critical for both efficiency and generation quality.

In policy learning and networked estimation, architectures may be U-Nets with cross-agent attention (MADiff (Zhu et al., 2023)), or local nonparametric regressors exchanging confidence-bounded summaries (decentralized nonparametric DDM (Wachel et al., 2023)).

3. Theoretical Guarantees and Emergent Properties

When each expert's loss is globally optimal for its data partition, and the router's weights recover the true posterior assignment, DDMs can be shown to provably recover the global marginal flow of a centralized diffusion model, thereby matching the asymptotic sample distribution (McAllister et al., 9 Jan 2025, Jiang et al., 3 Oct 2025). This holds for both continuous flow-matching and discretized score-matching settings:

$K$ 8

where $K$ 9 is the marginal flow under expert $\mathcal{C}_1, \dots, \mathcal{C}_K$ 0 and $\mathcal{C}_1, \dots, \mathcal{C}_K$ 1 is the router/posterior probability.

Crucially, the DDM framework provides inherent infrastructure resilience: experts can be trained and run asynchronously, tolerate hardware and network failures, and are compatible with heterogeneous compute resources (McAllister et al., 9 Jan 2025, Jiang et al., 3 Oct 2025).

4. Expert–Data Alignment Principle and Routing Effects

Recent systematic analyses demonstrate that DDM generation quality fundamentally depends on "expert–data alignment"—the alignment between the denoising state $\mathcal{C}_1, \dots, \mathcal{C}_K$ 2 and the cluster/domain on which an expert was trained (Villagra et al., 2 Feb 2026). Optimal routing selects experts whose training data centroid in a suitable embedding space (e.g., DINOv2 features) is closest to $\mathcal{C}_1, \dots, \mathcal{C}_K$ 3.

Empirically, sparse routing (Top-1 or Top-2) that prioritizes expert–data alignment outperforms full ensemble routing, which increases numerical stability but mixes conflicting expert predictions, dramatically degrading perceptual quality (e.g., FID 47.89 for full ensemble, versus 22.6 for Top-2 on Paris) (Villagra et al., 2 Feb 2026). Velocity alignment metrics and cluster–distance analyses confirm that quality gains are realized when routed experts cover the denoising state distribution.

Table: Routing Strategy Effects (from (Villagra et al., 2 Feb 2026))

Routing	FID (↓)	Mean Cluster Rank (↓)	Velocity Angle (°) (↓)
Top-1	30.60	1.54	3.6
Top-2	22.60	1.96	3.6
Full Ensemble	47.89	4.50	5.1

5. Applications: Image, Video, Multi-Agent, and Network Learning

Image and Video Generation: Paris 1.0 and 2.0 demonstrate that text-to-image and text-to-video diffusion models can be trained fully decentralized, matching or exceeding monolithic baselines on FID, Frechet Video Distance (FVD), CLIP-text similarity, and sample aesthetic scores at fixed compute (Jiang et al., 3 Oct 2025, Rouzbayani et al., 25 May 2026). Paris 2.0 achieved a reduction in FVD from 561.04 (monolith) to 279.01 (DDM), with improved CLIP alignment and aesthetic metrics, enabled by a causal video VAE and spatiotemporal transformer experts (Rouzbayani et al., 25 May 2026).

Policy and Coordination: MADiff adapts attention-based diffusion models for decentralized multi-agent reinforcement learning (offline RL). Each agent's diffusion policy conditions on local history, with cross-agent attention facilitating coordination. Parameter-sharing ablations, per-agent architecture variants, and history conditioning highlight the importance of shared representations and teammate modeling for decentralized planning without inter-agent communication (Zhu et al., 2023).

Nonparametric Network Learning: In decentralized function estimation, each agent computes Nadaraya–Watson estimates with explicit confidence, diffuses tuple summaries to immediate neighbors, and provably contracts error bounds over the network. This paradigm achieves global consistency and privacy-preserving network learning under minimal prior assumptions (Wachel et al., 2023).

Biological Decision Modeling: Drift–diffusion models of ant colony nest selection exemplify decentralized evidence accumulation and scaled decision accuracy, achieving exponential error reduction and $\mathcal{C}_1, \dots, \mathcal{C}_K$ 4 collective speedup versus individual decision times (Pradhan et al., 2021).

6. System Design, Infrastructure, and Limitations

DDMs eliminate the need for high-bandwidth, tightly coupled GPU clusters by enabling entirely asynchronous, topology-agnostic training: experts can be trained on geographically distributed, on-demand, or preemptible hardware (McAllister et al., 9 Jan 2025, Jiang et al., 3 Oct 2025, Rouzbayani et al., 25 May 2026). Stragglers and node failures affect only the corresponding expert, not the global system state.

Memory and inference costs scale with the number of experts ( $\mathcal{C}_1, \dots, \mathcal{C}_K$ 5 if all experts active per step), but sparse routing reduces per-inference compute to $\mathcal{C}_1, \dots, \mathcal{C}_K$ 6 experts per forward pass. Current limitations include the need for high-quality data partitioning—poor clustering impairs coverage and sample quality (McAllister et al., 9 Jan 2025)—and memory scaling for simultaneous expert loading. Mitigations include inference-efficient routing and post hoc distillation.

7. Quantitative Benchmarks and Empirical Trends

Comprehensive experiments across datasets (LAION-Aesthetics, ImageNet, MNIST, held-out video sets) show DDMs outperform monolithic diffusion models under iso-compute conditions. On LAION-Aesthetics, an 8-expert DDM achieved FID 6.08 (Top-1), outperforming a monolith (FID 8.49), and scaling experiments demonstrated feasible 24B-parameter models trained using 8×16-GPU nodes (McAllister et al., 9 Jan 2025).

For video generation, Paris 2.0 improved FVD from 561.04 (monolithic) to 279.01 (DDM), with CLIP similarity rising from 0.2032 to 0.2178, and aesthetic scores from 3.795 to 3.904 under matched compute. Sparse switching schedules, causal VAEs, and spatiotemporal transformers underpin temporal consistency and higher motion realism (Rouzbayani et al., 25 May 2026).

Empirically, optimal expert number is task- and data-dependent (e.g., $\mathcal{C}_1, \dots, \mathcal{C}_K$ 7 for LAION), and ablations confirm the necessity of feature-based clustering and Top- $\mathcal{C}_1, \dots, \mathcal{C}_K$ 8 routing for both efficiency and generative fidelity (Villagra et al., 2 Feb 2026, Jiang et al., 3 Oct 2025).

For further technical detail and full model recipes, see "Paris 2.0: A Decentralized Diffusion Model for Video Generation" (Rouzbayani et al., 25 May 2026), "Decentralized Diffusion Models" (McAllister et al., 9 Jan 2025), "Expert-Data Alignment Governs Generation Quality in Decentralized Diffusion Models" (Villagra et al., 2 Feb 2026), and "Paris: A Decentralized Trained Open-Weight Diffusion Model" (Jiang et al., 3 Oct 2025).