---
title: Coevolving Representation Diffusion (CoReDi)
url: https://www.emergentmind.com/topics/coevolving-representation-diffusion-coredi
type: topic
---

# Coevolving Representation Diffusion (CoReDi)

Coevolving Representation Diffusion (CoReDi) is a unified framework for joint image–feature generative modeling in which the semantic representation space co-adapts with the diffusion model, rather than remaining fixed throughout training. CoReDi targets the limitations of previous approaches—where semantic features are projected into a static, low-dimensional space prior to joint diffusion—and demonstrates that allowing this space to evolve yields substantive gains in both convergence speed and sample quality. The methodology is applicable to both VAE latent diffusion and pixel-space diffusion models, and experiments on large-scale benchmarks such as ImageNet256 provide empirical support for its effectiveness [2604.17492].

## 1. Motivation for Adaptive Representations

Standard joint image–feature diffusion models, such as ReDi [2504.16064], operate by projecting pretrained visual encoder features $z_0 \in \mathbb{R}^{L \times D}$ into a lower-dimensional semantic space using a fixed projection $P \in \mathbb{R}^{D \times d}$, frequently computed by PCA. This static mapping produces compressed representations $\tilde z_0 = z_0 P$ that participate in a shared diffusion process with VAE latents $x_0$. The forward process adds noise independently to both streams:
\[
x_t = (1-t)x_0 + t\epsilon_x, \quad \tilde z_t = (1-t)\tilde z_0 + t\epsilon_z,
\]
and the network learns to predict velocities $v^x_\theta, v^z_\theta$ by minimizing the squared errors relative to the noise residuals. However, when $P$ is fixed, the semantic space lacks the capacity to specialize for the generative task, constraining both training efficiency and achievable fidelity. CoReDi removes this bottleneck by making the semantic projection a learnable function $g_\phi$, allowing the representation space to evolve in tandem with the diffusion objective [2604.17492].

## 2. Mathematical Formulation

Let $x_0$ denote either the VAE or pixel latent associated with an image, and $z_0 = \mathrm{VE}(x_0) \in \mathbb{R}^{L \times D}$ the set of high-dimensional features from a frozen encoder (e.g., DINOv2). CoReDi introduces a learnable linear projection $W_\phi \in \mathbb{R}^{D \times d}$, replacing the fixed PCA step:
\[
\tilde z_0 = g_\phi(z_0) = z_0 W_\phi,
\]
followed by channelwise batch normalization without affine parameters:
\[
\hat z_0 = \mathrm{BN}(\tilde z_0).
\]
The forward diffusion dynamics for the joint state are:
\[
x_t = (1-t)x_0 + t\epsilon_x, \quad \hat z_t = (1-t)\hat z_0 + t\epsilon_z.
\]
Velocity heads $v^x_\theta$ and $v^z_\theta$ predict the respective denoising terms. Crucially, CoReDi prevents degenerate evolution (e.g., feature collapse) by constructing the representation loss target using a stop-gradient:
\[
L_\mathrm{rep}(\theta, \phi) = \mathbb{E} \|v^z_\theta(x_t, \hat z_t, t) - [\epsilon_z - \mathrm{sg}(\hat z_0)]\|^2.
\]
The loss for the image stream is:
\[
L_\mathrm{image}(\theta, \phi) = \mathbb{E} \|v^x_\theta(x_t, \hat z_t, t) - (\epsilon_x - x_0)\|^2.
\]
To forestall channel collapse or redundancy, an explicit regularization term $L_\mathrm{reg}(\phi)$ is included, with three investigated forms:

| Name                | Regularization Term                                            | Effect                                                             |
|---------------------|---------------------------------------------------------------|--------------------------------------------------------------------|
| Feature Variance (VF) | $L_\mathrm{var}(\hat z_0)$                                  | Ensures non-trivial channelwise variance per spatial vector         |
| Orthogonality (Ortho)| $L_\mathrm{orth}(W_\phi) = \|W_\phi^\top W_\phi - I_d\|_F^2$ | Promotes orthogonality in $W_\phi$ for decorrelated channels        |
| Covariance (Cov)     | $L_\mathrm{cov}(\hat z_0)$                                   | Minimizes off-diagonal channel covariance across spatial locations  |

The final loss function is:
\[
L(\theta, \phi) = L_\mathrm{image} + \lambda_z L_\mathrm{rep} + \lambda_\mathrm{reg} L_\mathrm{reg}.
\]
Gradients from $L_\mathrm{image}$ and $L_\mathrm{rep}$ affect $\theta$; $L_\mathrm{rep}$ (excluding the stop-gradient term) and $L_\mathrm{reg}$ modulate $\phi$.

## 3. Coevolution Algorithm and Gradient Flow

CoReDi employs a synchronized optimization protocol for simultaneous evolution of the diffusion model ($\theta$) and semantic projection ($\phi$). A high-level pseudocode summary:

```python
Initialize θ (diffusion model), φ (projection W_φ)
Initialize BN statistics μ,σ² via running EMA
for each training iteration do
    # Sample batch, no gradients on visual encoder
    Sample {x₀}, compute z₀ = VE(x₀)
    Compute \tilde z₀ = BN(z₀ W_φ ; μ,σ²)
    Sample t ~ Uniform[0,1], ε_x ~ N(0,I), ε_z ~ N(0,I)
    x_t = (1-t)x₀ + t ε_x
    \hat z_t = (1-t)\tilde z₀ + t ε_z
    [v^x, v^z] = DiffusionUNet_θ(x_t, \hat z_t, t)
    L_image = ||v^x - (ε_x - x₀)||²
    L_rep   = ||v^z - (ε_z - sg(\tilde z₀))||²
    L_reg   = L_var(...) or L_orth(W_φ) or L_cov(...)
    L_total = L_image + λ_z L_rep + λ_reg L_reg
    # Gradient updates
    θ ← θ − η ∇_θ(L_image + λ_z L_rep)
    φ ← φ − η_proj ∇_φ(L_rep + λ_reg L_reg)
    Update BN μ,σ² via EMA on \tilde z₀
end for
```

Batch normalization statistics are maintained via exponentially moving averages. Typical hyperparameters for ImageNet256 include $\lambda_z = 1.0$ (latent space), $\lambda_z = 0.1$ (pixel space), $\lambda_\mathrm{reg} = 1.0$, with equal learning rates for $\theta$ and $\phi$ ($1\text{e}–4$).

## 4. Stability Mechanisms and Theoretical Intuition

Unconstrained learning of $W_\phi$ risks degenerate solutions, such as all-zero or constant channel outputs. CoReDi integrates several stabilizers:

1. **Stop-gradient in representation loss**: Ensures $g_\phi$ cannot simply track its own evolving output, breaking degeneracy.
2. **Batch normalization (zero mean, unit variance)**: Suppresses scale collapse and restricts trivial fixed-point attractors.
3. **Explicit regularization (VF, Ortho, Cov)**: Promotes channelwise diversity and orthogonality.

Empirical ablation demonstrates that omitting any stabilization leads to catastrophic feature collapse or divergent objectives (e.g., FID diverges without batch norm, is $\approx 50$ without stop-gradient, and $\approx 37$ without explicit regularization). This structure is essential for stable coevolution of semantic space and diffusion model [2604.17492].

## 5. Experimental Evaluation

CoReDi is validated on both latent-space and pixel-space diffusion on ImageNet256, using frozen visual encoders such as DINOv2, MOCOv3, SigLIPv2, and MAE.

### Latent-Space Diffusion

A comparison of ReDi versus CoReDi across two scales:

| Model         | Parameters | Iterations | FID↓  |
|---------------|------------|------------|-------|
| SiT-B/2       | 130M       | 400K       | 33.0  |
| ReDi-B/2      | 130M       | 400K       | 21.4  |
| CoReDi-B/2    | 130M       | 200K       | 24.7  |
| CoReDi-B/2    | 130M       | 400K       | 16.4  |
| SiT-XL/2      | 675M       | 7M         | 8.3   |
| REPA-XL/2     | 675M       | 4M         | 5.9   |
| ReDi-XL/2     | 675M       | 4M         | 3.3   |
| CoReDi-XL/2   | 675M       | 2M         | 3.3   |

CoReDi matches or surpasses baseline FID, requiring half the standard training iterations. With classifier-free guidance, CoReDi-XL/2 achieves FID $1.58$ in $400$ epochs (versus $1.80$ for REPA, $1.72$ for ReDi) [2604.17492].

### Pixel-Space Diffusion

For DeCo-L/16 (pixel-space Diffusion model) on ImageNet256:

| Model        | Iter. | Params | FID↓  |
|--------------|-------|--------|-------|
| DeCo-L/16    | 100K  | 426M   | 46.0* |
| DeCo-L/16    | 200K  | 426M   | 31.3  |
| CoReDi-L/16  | 100K  | 426M   | 31.5  |
| CoReDi-L/16  | 200K  | 426M   | 21.5  |

CoReDi achieves an equivalent FID in half the iterations compared to the DeCo baseline. Regularization and loss weight ablations show the results are robust to hyperparameter selection, yielding minor FID variation across $\lambda_\mathrm{reg}$ and optimal $\lambda_z = 0.10$ in pixel space.

### Representation Encoder Variation

Performance improvements of $3$–$6$ FID points over fixed ReDi projections are observed for all tested frozen visual encoders, confirming the robustness of adaptive representation learning.

### Evolution of Representation Structure

Analysis using Local vs. Distant Similarity (LDS), Correlation Decay Slope (CDS), and RMS Spatial Contrast (RMSC) metrics shows that learned representations increasingly develop spatial structure and self-similarity over the course of training, surpassing static PCA projections. This dynamic structuring correlates with improved sample quality and convergence.

## 6. Dynamics of Learned Semantic Space

The adaptive mapping $g_\phi$ specializes the semantic projection channels to complement low-level VAE latents for image synthesis. Early-stage channels are noisy and unstructured; as the objective jointly sculpts $\theta$ and $\phi$, each channel organizes into spatially coherent, semantically meaningful patterns. This coadaptation addresses a major limitation of fixed-projection approaches, making the generative process more responsive to semantic alignment and ultimately achieving faster, higher-quality synthesis.

## 7. Limitations and Future Research

Observed limitations of CoReDi include:
- Sensitivity to loss weights $\lambda_z$ and $\lambda_\mathrm{reg}$, with parameter tuning required for pixel and latent regimes;
- Restriction to linear projections $W_\phi$; incorporating shallow nonlinear mappings or deeper adapters is a potential direction;
- Rigid freezing of the visual encoder; slow momentum-based joint fine-tuning may further enhance coadaptation.

*A plausible implication is* that principles underlying CoReDi—allowing representation spaces to coevolve with generative models—could generalize to other domains such as multimodal and self-supervised diffusion settings, potentially unlocking further advances in representation-aware generation [2604.17492, 2504.16064].

Source: https://www.emergentmind.com/topics/coevolving-representation-diffusion-coredi