---
title: Composable Diffusion (CoDi) Framework
url: https://www.emergentmind.com/topics/composable-diffusion-codi
type: topic
---

# Composable Diffusion (CoDi) Framework

Composable Diffusion (CoDi) is a methodological framework for constructing generative models whose outputs are governed by the composition of independently trained diffusion sub-models. This paradigm enables principled, zero-shot combination of concepts, modalities, data sources, or policies at inference, leveraging the linearity of the score function in diffusion models and the product-of-experts formalism of energy-based models. CoDi and its variants address the exponential combinatorics of compositional generative tasks—such as assembling complex scene specifications, synthesizing multimodal content, or enforcing logical constraints—while enabling modular training, continual learning, selective forgetting, and training data compartmentalization.

## 1. Mathematical Foundation of Composable Diffusion

At the core of CoDi is the observation that denoising diffusion models, trained to represent conditional distributions $p_i(x)$, can be interpreted as implicit energy-based models (EBMs) with energy $E_i(x) \approx -\log p_i(x)$ [2206.01714]. Product-of-experts composition then yields a joint distribution:
\[
p_{\text{joint}}(x) \propto \prod_{i=1}^k p_i(x)
\]
which, in energy terms, is $E_{\text{joint}}(x) = \sum_i E_i(x)$. The corresponding score (the gradient of the log-density) is additive:
\[
\nabla_x \log p_{\text{joint}}(x) = \sum_{i=1}^k \nabla_x \log p_i(x)
\]
In conditional contexts (e.g., classifier-free guidance or attribute composition), this can generalize to arbitrary logic (AND, NOT) at the cost of independence assumptions:
\[
\nabla_x\log p(x \mid C_1=c_1 \land C_2=c_2) \approx \nabla_x\log p(x \mid C_1) + \nabla_x\log p(x \mid C_2) - \nabla_x\log p(x)
\]
[2503.01145, 2206.01714].

This foundation underpins a variety of practical architectures and composition schemes, including layer-wise compositional synthesis, multimodal joint generation, and score fusion across data partitions or modalities.

## 2. Theoretical Guarantees and Failure Modes

Projective composition is the formal guarantee sought in compositional diffusion: for projections $\Pi_i: \mathbb{R}^n \to \mathbb{R}^{d_i}$, the composed model $\hat{p}$ is a projective composition if $\Pi_i \sharp \hat{p} = \Pi_i \sharp p_i$ for all $i$—i.e., the composed distribution matches the marginals of each constituent in the relevant feature space [2502.04549].

Linear score composition achieves projective composition when certain conditional independence (factorized-conditional) or orthogonality assumptions hold in pixel or transformed feature spaces. Specifically, if the variable partitioning or an invertible feature map $\mathcal{A}$ induces mutual independence in the relevant coordinates, then reverse-diffusion with the composed score recovers the target distributive product.

Failing these assumptions results in pathological outcomes:
- “Bayes composition” with unconditional background $p_u$ generally fails to yield projective compositions and can exhibit fractional or ambiguous outputs.
- Non-orthogonal feature transforms invalidate the commutation between noising and reparameterization, leading to non-Lipschitz evolutions in Wasserstein space and generative instability.

The orthogonality heuristic—empirically validated using CLIP embeddings—offers a necessary, easily computable condition: mean difference vectors between constituent models and the background should be nearly orthogonal to predict successful composition in CoDi [2502.04549].

## 3. Organizational and Algorithmic Instantiations

### Modular Training and Inference-time Composition

Composable Diffusion is operationalized in various ways, all leveraging inference-time composition:
- **Component-wise concept models**: Each concept (attribute, object, relation) is modeled by a conditional denoising network; compositions are performed at sampling with no additional fine-tuning [2206.01714].
- **Sharded data compartmentalization**: Sub-models are trained on disjoint data or attribute subsets; score fusion at inference reconstructs or approximates the joint generative process, enabling selective forgetting, continual learning, and strict access control [2308.01937].
- **Cross-modality policies**: Policies are independently trained per observation modality (e.g., RGB, point cloud), then fused by weighted score addition to construct a multimodal controller without retraining [2503.12466].
- **Layered and spatial composition**: For tasks such as multi-layered image synthesis, per-layer denoisers with inter- and intra-layer attention modules are composed at each timestep [2403.11929].

Algorithmically, the generic composition scheme at each denoising step is:
\[
\epsilon_{\text{comp}} = \sum_i w_i \epsilon_i(x_t, t)
\]
where $\epsilon_i$ is the predicted noise for expert $i$ and $w_i$ are scalar or pixel-wise weights (e.g., softmax masks in view fusion [2402.02906]). Variants include explicit background score subtraction, classifier-free guidance, and logical (AND/NOT) operations.

### Training Objectives and Conditional Independence

Standard conditional diffusion models often fail to enforce the independence structure required by compositional inference, especially in non-uniform or partial training regimes [2503.01145]. This leads to demonstrable failures in generating correct logical attribute compositions (e.g., AND/NOT), as measured by Conformity Score and Jensen-Shannon divergence.

The CoInD method augments the score-matching objective with a Fisher divergence penalty:
\[
L_{\rm CI} = \mathbb E_{X,C}\left\|\nabla_X\log p_\theta(X|C) - \sum_i\nabla_X\log p_\theta(X|C_i) + (n-1)\,\nabla_X\log p_\theta(X)\right\|_2^2
\]
which is minimized alongside the denoising loss, ensuring the joint model factorizes as in the causal hypothesis. Practically, this corrects “composition by score addition” even under severe training support sparsity [2503.01145].

## 4. Application Domains and Empirical Behavior

### Structured Visual and Multimodal Generation

CoDi variants robustly generalize to combinatorial compositions far outside the training support:
- **Compositional visual generation**: For scenes and faces with multi-attribute or relational queries, CoDi outperforms unfactored diffusion and GAN-based methods in both adherence and FID, achieving correct binding and relational structure even for zero-shot attribute combinations [2206.01714].
- **Multi-layered synthesis**: LayerDiff introduces inter- and intra-layer attention together with self-mask guidance, attaining competitive FID and CLIP-Score to single-image baselines while enabling fine-grained layer control (inpainting, style transfer) [2403.11929].
- **Novel view synthesis**: ViewFusion leverages composable denoising across multiple pose-free views with per-pixel softmax score fusion, outperforming NeRF-based and transformer baselines on small-scale datasets [2402.02906].
- **Arbitrary input-output mapping**: The CoDi framework for any-to-any modality generation introduces bridging and latent alignments, supporting synchronized generation across text, image, audio, and video with competitive or state-of-the-art class fidelity (COCO-FID, MSR-VTT R@1, AudioCaps FAD) and semantic consistency metrics [2305.11846].

### Policy and Task Composition

- **Modality Composable Policies**: Modality-Composable Diffusion Policy (MCDP) fuses independently trained single-modality policies, yielding consistently higher task success rates than any constituent on the RoboTwin benchmark, especially when both unimodal experts are competent [2503.12466].
- **Instruction-following navigation**: ComposableNav decomposes natural language instructions into atomic motion primitives, each implemented as an independent diffusion model fine-tuned by RL. This compositional approach achieves superior generalization to unseen instruction conjunctions in both simulation and hardware (SR≈76% for $k=2$ instructions vs <40% for baselines) [2509.17941].

### Privacy, Forgetting, and Data Attribution

Compartmentalized Diffusion Models (CDMs) facilitate perfect selective forgetting, attribution, and a-la-carte access control: removing or updating a sub-model (trained on a data shard $D_i$) requires no retraining of others, and class-conditional FID increases by at most 10% relative to a monolithic model. Data attribution per sample is quantifiable via the proportion of diffusion path log-likelihood sourced from each shard [2308.01937].

## 5. Limitations, Practical Heuristics, and Extensions

Limitations of current CoDi methodology include:
- **Independence assumptions**: Empirical and theoretical results show that compositional score addition is only valid under strict conditional independence or sufficient orthogonality between component domains/features [2502.04549, 2503.01145].
- **Scalability**: As the number of composed primitives increases, generation quality can degrade due to accumulated conflicts in the score fields [2509.17941]. Approaches such as energy-based sampling, MCMC, or HMC may improve robustness at large $k$.
- **Attribute supervision**: Some approaches (e.g., CoInD) require labeled discrete attributes or reward heuristics, and do not yet generalize to unsupervised or open-set composition [2503.01145].
- **Heuristic validation**: The orthogonality-of-means heuristic provides a necessary (not sufficient) criterion for composition, evaluated via cosine similarity in CLIP or feature space [2502.04549].

Future directions include learned or meta-optimized weight selection for expert fusion, fully end-to-end parsing of compositional instructions, unsupervised factorization of component domains, and energy-based or sampling-improved joint generation methods [2509.17941, 2502.04549].

## 6. Representative Experimental Results

Empirical findings across domains consistently report that CoDi models, when composed under the outlined assumptions, achieve:
- Structured generalization to unseen combination queries: FFHQ attribute adherence $\sim$69% vs GAN baselines; multi-object CLEVR placement 31.4% accuracy vs 7.3% EBM baselines; compositional navigation SR 76% for $k=2$ vs <40% for prior methods [2206.01714, 2509.17941].
- Data compartmentalization with minimal generative penalty: 8-split FID 6.54 ($+10.3\%$ vs paragon FID 5.93), TIFA text-image alignment improved by 14.33% over monolithic models [2308.01937].
- Cross-modality or layer fusion with competitive or improved metrics: ViewFusion LPIPS 0.033, LayerDiff (+SMG) FID 21.3 outperforming vanilla SD, CoDi any-to-any generation matching or surpassing unimodal baselines [2402.02906, 2403.11929, 2305.11846].
- Enhanced logical composition and controlled sample generation: CoInD delivers $\sim$7x gains in Conformity Score and halves JSD compared to naively composed diffusion models under partial training support [2503.01145].

## 7. Summary Table: Core Methodological Elements

| Approach/Domain     | Training Structure     | Inference-time Composition | Notable Capabilities                    |
|---------------------|-----------------------|---------------------------|-----------------------------------------|
| Compositional Vision [2206.01714]      | Per-attribute model      | Score/energy summation         | Relational/logic comp, binding          |
| Any-to-any Modality [2305.11846]        | Unimodal LDMs + paired cross-attention | Latent cross-attention +  score sum | Synchronized, arbitrary joint outputs   |
| Compartmentalization [2308.01937]       | Data-sharded models      | Weighted score mixture         | Continual learning, selective forgetting|
| Multimodal Policy [2503.12466]          | Per-modality DPs         | Weighted score fusion          | Cross-modality, no retraining           |
| Multi-layered Composition [2403.11929]  | Multi-layer, prompt-conditioned | Inter-/intra-layer attention, mask guidance | Object-wise editing and control         |
| ComposableNav [2509.17941]              | RL-finetuned motion primitives | Parallel noise addition      | Instruction composition in navigation   |
| Logical Composition [2503.01145]        | CI-penalized joint model | Score composition (AND/NOT)   | Arbitrary Boolean attribute composition |

CoDi provides a theoretically grounded, empirically validated, and methodologically modular framework for generative model composition across vision, audio, policy, and control, establishing a toolkit for structured, scalable, and privacy-respecting generative inference.

Source: https://www.emergentmind.com/topics/composable-diffusion-codi