Papers
Topics
Authors
Recent
Search
2000 character limit reached

Energy-based Compositional Diffuser (ECD)

Updated 5 July 2026
  • Energy-based Compositional Diffuser (ECD) is a generative modeling approach that combines additive energy formulations with diffusion denoising to enforce the joint compatibility of constraints.
  • It implements mechanisms such as additive score composition, MCMC-style corrections, and energy-based cross-attention to fuse multiple factor experts without training separate models for each combination of conditions.
  • ECD has demonstrated enhanced sample quality and controllability across various domains, including image synthesis, text-to-image diffusion, and motion planning.

to=arxiv_search.search 天天中彩票足球 json {"5query5 Compositional Diffusion Planning5\5 OR 5\5 Image Decomposition with Diffusion Models5\5 OR 5\5 of Score-Based Diffusion Models for Model Composition5\5 OR 5\5 Energy-Based Models by Cooperative Diffusion Recovery Likelihood5\5 to=arxiv_search.search 神彩争霸 json {"5query5 compositional diffusion planning", "max_results": 5, "sort_by": "relevance"} to=arxiv_search.search 天天送彩票 json {"5query5 Image Decomposition with Diffusion Models", "max_results": 5, "sort_by": "relevance"} Energy-based Compositional Diffuser (ECD) denotes a line of diffusion-based generative modeling in which composition is specified through energies, energy gradients, or score-like denoising fields, and implemented inside an iterative reverse process rather than by training a separate monolithic conditional model for every combination of constraints. The term appears explicitly in “Energy-based Compositional Diffusion Planning” (&&&5query5&&&), but the broader formulation is distributed across closely related work on additive score composition, conditional reverse-step EBMs, energy-based cross-attention, latent-space compositional EBMs, and joint EBM–diffusion training (&&&5\5&&&). Across these variants, the recurring claim is that multiple factors, concepts, or local constraints can be represented as additive experts, while denoising, Langevin refinement, ODE flow, or Metropolis-corrected sampling enforces their joint compatibility.

5\5. Emergence of the ECD paradigm

ECD did not enter the literature as a single unified framework. Rather, several research threads converged on the same structural idea: composition is easiest to state in energy space, but diffusion models provide a practical denoising or score interface through which those energies can be deployed.

A concise genealogy is visible in a small set of representative papers.

Work arXiv id ECD-relevant contribution
“Controllable and Compositional Generation with Latent-Space Energy-Based Models” (&&&5 OR \5&&&) Additive latent energies and logical AND/OR/NOT
“Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models” (&&&5 OR \5&&&) Cross-attention as energy-based inference and compositional conditioning
“MCMC-Correction of Score-Based Diffusion Models for Model Composition” (&&&5 OR \5&&&) MH-like correction from score line integrals
“Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood” (Zhu et al., 2023) Multi-noise EBMs with additive conditional energies
“Generalized Contrastive Divergence” (Yoon et al., 2023) Joint EBM–diffusion training with diffusion as trainable sampler
“Compositional Image Decomposition with Diffusion Models” (&&&5\5&&&) Additive denoising experts inferred from a single image
“EnergyMoGen” (Zhang et al., 2024) Explicit latent-space ECD for compositional motion generation
“Energy-based Compositional Diffusion Planning” (&&&5query5&&&) Global trajectory energy from summed local bridge potentials

The earliest of these works used latent EBMs over a frozen generator. LACE defines a joint model

PRESERVED_PLACEHOLDER_5query5^

then samples in latent space from

PRESERVED_PLACEHOLDER_5\5^

so that multiple attributes are composed simply by summing energies (&&&5 OR \5&&&). This formulation already contains the core ECD motif: a base generator is preserved, while control is imposed through additive conditional energies.

Later work moved the same idea into diffusion models themselves. Decomp Diffusion argues that a denoiser can be interpreted as an energy gradient and composes factor experts by summing denoising outputs (&&&5\5&&&). Energy-based cross-attention treats latent-text interaction as an energy minimization problem and composes prompts by a linear combination of cross-attention outputs (&&&5 OR \5&&&). CDRL learns EBMs directly over diffusion noise levels and composes conditional energies in a product-of-experts form (Zhu et al., 2023). EnergyMoGen explicitly describes latent motion diffusion as an energy-based compositional diffuser and fuses latent-space and cross-attention energies (Zhang et al., 2024). ECD planning then turns this perspective into a formal statement: heuristic score stitching is generally non-conservative, whereas a sum of local bridge potentials defines a valid global energy and a conservative correction field (&&&5query5&&&).

5 OR \5. Mathematical basis: additive energies, additive scores, and their limits

The mathematical attraction of ECD lies in the simplicity of composition at the level of energies. If a target is defined by a product of experts,

PRESERVED_PLACEHOLDER_5 OR \5^

then the corresponding clean-data energy is additive,

PRESERVED_PLACEHOLDER_5 OR \5^

This identity underlies latent-space logical composition in LACE, conditional composition in CDRL, and local bridge composition in ECD planning (&&&5 OR \5&&&, Zhu et al., 2023, &&&5query5&&&).

Diffusion models enter because denoisers and scores can often be interpreted as gradients of energies. Decomp Diffusion states this explicitly: PRESERVED_PLACEHOLDER_5 OR \5^ If each factor zkz_k induces a conditional denoiser ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k), then the composed denoiser is

ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),

which corresponds, under the paper’s interpretation, to

xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).

This is the clearest additive-score formulation of ECD: factors do not need to share a latent bottleneck, only a common denoising or score space (&&&5\5&&&).

A parallel formulation appears in CDRL, where each reverse step is itself an EBM: pθ(xtxt+1)=1Z~θ,t(xt+1)exp(fθ(xt;t)12σt+12xtxt+12).p_\theta(x_t \mid x_{t+1}) = \frac{1}{\tilde Z_{\theta,t}(x_{t+1})} \exp\left( f_\theta(x_t; t) - \frac{1}{2\sigma_{t+1}^2}\|x_t - x_{t+1}\|^2 \right). For PRESERVED_PLACEHOLDER_5\5query5^ conditional experts, the paper gives

PRESERVED_PLACEHOLDER_5\5\5^

and the corresponding guided gradient

PRESERVED_PLACEHOLDER_5\5 OR \5^

Here ECD is neither latent arithmetic nor prompt concatenation; it is additive conditional energy at every reverse step (Zhu et al., 2023).

The planning formulation makes the same point in a more formal way. ECD defines a global trajectory energy

PRESERVED_PLACEHOLDER_5\5 OR \5^

with local bridge potential

PRESERVED_PLACEHOLDER_5\5 OR \5^

The score is then

PRESERVED_PLACEHOLDER_5\55^

so conservativity is guaranteed by construction (&&&5query5&&&).

A central caution in the literature is that this energy algebra does not automatically commute with diffusion time. The theoretical note associated with (&&&5 OR \5 OR \5&&&) states that, in general,

PRESERVED_PLACEHOLDER_5\56

and likewise

PRESERVED_PLACEHOLDER_5\57

for tempering. This means that a clean-data product-of-experts, a tempered density, or a guided posterior does not generally translate into the same algebra on noised marginals. The practical implication is that naive score arithmetic is often heuristic rather than exact (&&&5 OR \5 OR \5&&&).

5 OR \5. Core mechanisms of composition

The ECD literature contains several distinct mechanisms for implementing energy-based composition. They differ in where the energy is defined and how the reverse process is corrected, but they share the same additive logic.

The first mechanism is additive denoising or score composition. Decomp Diffusion infers PRESERVED_PLACEHOLDER_5\58 low-dimensional latent codes from a single image,

PRESERVED_PLACEHOLDER_5\59

and reconstructs or recombines an image by summing latent-conditioned denoisers: PRESERVED_PLACEHOLDER_5 OR \5query5^ Training minimizes

PRESERVED_PLACEHOLDER_5 OR \5\5^

The model thereby learns factor experts that can be recombined within a model and even across models, because the interface is the denoising vector in image space at timestep PRESERVED_PLACEHOLDER_5 OR \5 OR \5^ (&&&5\5&&&).

The second mechanism is conditional reverse-step EBMs with short-run refinement. CDRL learns a time-conditioned energy PRESERVED_PLACEHOLDER_5 OR \5 OR \5^ and an initializer

PRESERVED_PLACEHOLDER_5 OR \5 OR \5^

At each step, the initializer proposes PRESERVED_PLACEHOLDER_5 OR \55, then Langevin refinement targets the reverse conditional EBM: PRESERVED_PLACEHOLDER_5 OR \56 Composition is performed directly in the energy field, not as an auxiliary classifier (Zhu et al., 2023).

The third mechanism is Metropolis-style correction for score-based composition. Score-only diffusion models generally do not define a globally consistent scalar energy, so exact MH correction is unavailable. “MCMC-Correction of Score-Based Diffusion Models for Model Composition” replaces the endpoint energy difference by a line integral of the score field: PRESERVED_PLACEHOLDER_5 OR \57 leading to the MH-like rule

PRESERVED_PLACEHOLDER_5 OR \58

The paper is explicit that this matches exact MH only if the score is conservative; otherwise the line integral depends on the path and the method is only MH-like (&&&5 OR \5&&&).

The fourth mechanism is energy-based conditioning inside cross-attention. Energy-Based Cross Attention defines a conditional energy

PRESERVED_PLACEHOLDER_5 OR \59

a prior energy

PRESERVED_PLACEHOLDER_5 OR \5query5^

and a context update

PRESERVED_PLACEHOLDER_5 OR \5\5^

For multiple prompts PRESERVED_PLACEHOLDER_5 OR \5 OR \5, composition is realized as

PRESERVED_PLACEHOLDER_5 OR \5 OR \5^

with PRESERVED_PLACEHOLDER_5 OR \5 OR \5^ implementing negation (&&&5 OR \5&&&).

The fifth mechanism is joint energy–sampler training. Generalized Contrastive Divergence writes

PRESERVED_PLACEHOLDER_5 OR \55^

so the diffusion model acts as a trainable sampler or policy, while the energy acts as a reward or critic. This mechanism does not itself provide modular composition, but it supplies a training template for any ECD in which a diffusion model must internalize an energy-defined target rather than merely receive guidance at inference (Yoon et al., 2023).

5 OR \5. Canonical instantiations across domains

In image decomposition, Decomp Diffusion demonstrates that ECD-style additive denoising is not restricted to object slots. Its inferred factors include local objects in CLEVR and Tetris, but also global properties such as lighting, shadows, foreground/background, camera position, lighting position, facial features, hair shape, hair color, skin tone, facial expression, and color tone or sharpness in art (&&&5\5&&&). The paper’s notion of composition is therefore broader than object-centric segmentation: factors may be spatially localized or globally acting.

In text-to-image diffusion, Energy-Based Cross Attention localizes ECD inside the denoiser. Rather than composing global image-space scores, it treats cross-attention as an energy-based inference step and updates context vectors layer by layer within each denoising step. The method is explicitly training-free and hooks into every cross-attention block of Stable Diffusion; it was evaluated on multi-concept generation, text-guided inpainting, and synthetic and real image editing (&&&5 OR \5&&&).

In latent controllable generation, LACE provides an energy-based formulation on top of a frozen StyleGAN-like generator. It defines exact logical operators

PRESERVED_PLACEHOLDER_5 OR \56

PRESERVED_PLACEHOLDER_5 OR \57

PRESERVED_PLACEHOLDER_5 OR \58

Sampling is performed by a latent ODE

PRESERVED_PLACEHOLDER_5 OR \59

which yields a deterministic energy-gradient flow from Gaussian initialization (&&&5 OR \5&&&).

In human motion generation, EnergyMoGen explicitly frames latent motion diffusion as an ECD. Its latent-aware conjunction uses

PRESERVED_PLACEHOLDER_5 OR \5query5^

while negation uses

PRESERVED_PLACEHOLDER_5 OR \5\5^

A second semantic-aware energy is defined in cross-attention, with adaptive text update

PRESERVED_PLACEHOLDER_5 OR \5 OR \5^

The final denoiser is obtained by Synergistic Energy Fusion,

PRESERVED_PLACEHOLDER_5 OR \5 OR \5^

with PRESERVED_PLACEHOLDER_5 OR \5 OR \5^ (Zhang et al., 2024).

In planning, ECD becomes fully explicit. The long-horizon trajectory is not produced by averaging local chunk predictions, but by minimizing the sum of local bridge potentials. The resulting chunk score

PRESERVED_PLACEHOLDER_5 OR \55^

contains both an interior update and a boundary reaction term. The second term is the principal difference from heuristic stitching, and it is the reason the field is conservative (&&&5query5&&&).

5. Empirical profile

The empirical record of ECD-style methods is heterogeneous because the tasks differ, but several patterns recur: additive composition can preserve sample quality, it often improves controllability, and exact or approximate energy correction matters most in multimodal settings.

Decomp Diffusion reports reconstruction metrics on PRESERVED_PLACEHOLDER_5 OR \56 images of CelebA-HQ: FID PRESERVED_PLACEHOLDER_5 OR \57, KID PRESERVED_PLACEHOLDER_5 OR \58, LPIPS PRESERVED_PLACEHOLDER_5 OR \59; Falcor5 OR \5D: FID zkz_k5query5, KID zkz_k5\5, LPIPS zkz_k5 OR \5^; Virtual KITTI 5 OR \5: FID zkz_k5 OR \5, KID zkz_k5 OR \5, LPIPS zkz_k5; and CLEVR: FID zkz_k6, KID zkz_k7, LPIPS zkz_k8, outperforming COMET and object-centric baselines in the paper’s table (&&&5\5&&&). On Falcor5 OR \5D, the best Decomp Diffusion model with latent dimension zkz_k9 obtains MIG ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)5query5^ and MCC ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)5\5^, compared with COMET’s MIG ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)5 OR \5^ and MCC ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)5 OR \5^. It also generates CLEVR scenes with 8 objects even though training scenes contain only 5 OR \5^ objects, which the paper treats as evidence for compositional extrapolation.

LACE shows that additive energy logic can scale to high-resolution photorealistic synthesis. On CIFAR-5\5query5^ conditional generation, LACE-ODE achieves ACC 5query5.975 OR \5, FID 6.65 OR \5^, while LACE-LD gives ACC 5query5.95 OR \59, FID 5 OR \5.5 OR \5query5^; on FFHQ ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)5 OR \5, for the single-attribute case “glasses,” LACE-ODE obtains FID 5 OR \5query5.95 OR \5, ACC 5query5.998, versus StyleFlow FID 5 OR \5 OR \5.5query58, ACC 5query5.899 (&&&5 OR \5&&&). The same paper reports zero-shot logical composition at ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)5, with unseen combinations such as ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)6 substantially outperforming StyleFlow on the controlled attributes.

For score-corrected composition, the 5 OR \5D product-composition experiment in (&&&5 OR \5&&&) is the relevant controlled test. Reverse diffusion alone gives LL ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)7, ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)8, GMM ϵθ(xt,t,zk)\epsilon_\theta(x^t,t,z_k)9 for the score model, whereas HMC-5-line improves to LL ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),5query5, ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),5\5, GMM ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),5 OR \5^. The comparable explicit energy model with HMC yields LL ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),5 OR \5, ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),5 OR \5, GMM ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),5, so the score-line-integral correction is empirically close to energy-based correction in that setting.

CDRL demonstrates that explicit EBMs over diffusion marginals need not be prohibitively slow. On CIFAR-5\5query5, it reports FID 5 OR \5.5 OR \5\5^ for CDRL and FID 5 OR \5.68 for CDRL-large, compared with FID 9.58 for DRL; on ImageNet ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),6, CDRL reaches FID 9.5 OR \55^ (Zhu et al., 2023). The sampling-cost ablation shows that the cooperative initializer allows a reduction from ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),7 MCMC steps in DRL to ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),8 in CDRL, with better FID.

EnergyMoGen provides the clearest benchmark for an explicitly named ECD in latent diffusion. On HumanML5 OR \5D it reports Top-5\5^ R-Precision 5query5.55 OR \5 OR \5^, Top-5 OR \5^ 5query5.75\55, Top-5 OR \5^ 5query5.85\55, FID 5query5.5\5, MM-Dist 5 OR \5.95\55^, and Diversity 9.5 OR \588; on compositional generation from multiple texts on MTT, the latent-only model gives R@5\5^ 9.7, FID 5query5.95\5, Transition distance 5\5.6, the semantic-only model gives R@5\5^ 5\55.5\5, FID 5query5.569, Transition distance 5 OR \5.5 OR \5^, and the full Ours + SEF gives R@5\5^ 5\55.9, FID 5query5.65query5 OR \5^, Transition distance 5\5.6 (Zhang et al., 2024). The stated interpretation is direct: latent-aware energy improves motion smoothness, semantic-aware energy improves compositional correctness, and SEF balances the two.

The planning paper reports the most explicit task-level ECD gains. On OGBench stitching tasks, PointMaze Giant improves from CD ϵpred=kϵθ(xt,t,zk),\epsilon_{\text{pred}} = \sum_k \epsilon_\theta(x^t,t,z_k),9 to ECD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).5query5^; AntMaze Giant improves from CD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).5\5^ to ECD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).5 OR \5^; HumanoidMaze Giant improves from CD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).5 OR \5^ to ECD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).5 OR \5^ (&&&5query5&&&). Runtime on PointMaze Giant is essentially unchanged, CD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).5 s versus ECD xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).6 s, which supports the paper’s claim that the Markov-based reaction approximation preserves near-stitching speed while improving consistency.

6. Limitations, misconceptions, and open problems

A persistent misconception is that ECD is simply “adding scores.” The literature is more restrictive. Clean-data energy composition is algebraically simple, but the time-xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).7 score of a composed target is generally not the same algebraic combination of the component time-xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).8 scores (&&&5 OR \5 OR \5&&&). Decomp Diffusion’s additive rule works because the model is trained with that additive denoising law, not because score addition is universally exact (&&&5\5&&&).

A second misconception is that score-based correction and energy-based correction are equivalent. They are not. The MH-like rule based on line integration of the score agrees with exact MH only when the score field is conservative; otherwise the line integral depends on the path, and the resulting kernel has no exact invariance guarantee (&&&5 OR \5&&&). This distinction matters whenever precise target densities or calibrated accept–reject corrections are required.

Many ECD variants inherit structural limitations from their factorization. Decomp Diffusion fixes the number of components xkEk(xt,t;zk).\nabla_x \sum_k E_k(x^t,t;z_k).9, is not guaranteed to produce distinct factors, and can collapse if latent capacity is too large; it also works best on datasets with recurring structure and alignment, and computational cost scales roughly with pθ(xtxt+1)=1Z~θ,t(xt+1)exp(fθ(xt;t)12σt+12xtxt+12).p_\theta(x_t \mid x_{t+1}) = \frac{1}{\tilde Z_{\theta,t}(x_{t+1})} \exp\left( f_\theta(x_t; t) - \frac{1}{2\sigma_{t+1}^2}\|x_t - x_{t+1}\|^2 \right).5query5^ because the network is called once per component (&&&5\5&&&). CDRL’s additive composition assumes conditional independence of concepts given pθ(xtxt+1)=1Z~θ,t(xt+1)exp(fθ(xt;t)12σt+12xtxt+12).p_\theta(x_t \mid x_{t+1}) = \frac{1}{\tilde Z_{\theta,t}(x_{t+1})} \exp\left( f_\theta(x_t; t) - \frac{1}{2\sigma_{t+1}^2}\|x_t - x_{t+1}\|^2 \right).5\5, and initializer compositionality remains underexplored in the main compositional experiment (Zhu et al., 2023). EnergyMoGen relies mostly on inference-time composition rather than a dedicated compositional training loss, and its semantic-aware branch can cause foot sliding and motion jitter (Zhang et al., 2024).

Training remains nontrivial when diffusion and energy are learned jointly. GCD provides a principled minimax objective, but the paper’s evidence is only preliminary and on 5 OR \5D synthetic data; it also requires policy-gradient optimization, nearest-neighbor entropy estimation, and value-function baselines, all of which introduce variance and scaling concerns (Yoon et al., 2023). Adversarial reverse-step EBMs improve markedly when embedded in diffusion, but they still require alternating optimization of an energy, a latent generator, and a variational posterior, and performance degrades when the number of denoising steps is too small or too large (&&&5 OR \57&&&).

In planning, ECD is conservative only for the induced bridge-energy model, not automatically for the true long-horizon data distribution. Its efficient approximation assumes approximately first-order Markov local structure and solves a block-tridiagonal system at each denoising step; the paper notes that this assumption may weaken in settings with delayed dynamics or strong long-range temporal constraints (&&&5query5&&&).

These limitations suggest a stable interpretation of ECD. It is not a single architecture, nor a guarantee that arbitrary expert combinations will be probabilistically exact. It is a design principle: define composition in energy space, preserve a common reverse state space, and use denoising, cross-attention, MCMC correction, or amortized sampling to realize the resulting combined field. The strongest current results arise when the composition mechanism is aligned with training, when the energy or score interface is explicitly shared across experts, and when approximation errors in the reverse process are corrected rather than ignored (&&&5\5&&&, Zhu et al., 2023, &&&5query5&&&).

Topic to Video (Beta)

No one has generated a video about this topic yet.

Whiteboard

No one has generated a whiteboard explanation for this topic yet.

Follow Topic

Get notified by email when new papers are published related to Energy-based Compositional Diffuser (ECD).