EVLF: Early Vision-Language Fusion for Generative Dataset Distillation

Abstract: Dataset distillation (DD) aims to synthesize compact training sets that enable models to achieve high accuracy with significantly fewer samples. Recent diffusion-based DD methods commonly introduce semantic guidance through late-stage cross-attention, where textual prompts tend to dominate the generative process. Although this strategy enforces label relevance, it diminishes the contribution of visual latents, resulting in over-corrected samples that mirror prompt patterns rather than reflecting intrinsic visual features. To solve this problem, we introduce an Early Vision-Language Fusion (EVLF) method that aligns textual and visual embeddings at the transition between the encoder and the generative backbone. By incorporating a lightweight cross-attention module at this transition, the early representations simultaneously encode local textures and global semantic directions across the denoising process. Importantly, EVLF is plug-and-play and can be easily integrated into any diffusion-based dataset distillation pipeline with an encoder. It works across different denoiser architectures and sampling schedules without any task-specific modifications. Extensive experiments demonstrate that EVLF generates semantically faithful and visually coherent synthetic data, yielding consistent improvements in downstream classification accuracy across varied settings. Source code is available at https://github.com/wenqi-cai297/earlyfusion-for-dd/.

• The paper introduces EVLF, a lightweight encoder-stage cross-attention module that fuses visual latents with class text embeddings before diffusion, using MSE for visual preservation and InfoNCE for semantic alignment.
• EVLF consistently improves distilled-data accuracy, including gains of 9.6 percentage points on ImageIDC, 8.1 points on CIFAR-10, and 6.7 points on Tiny-ImageNet at 10 images per class.
• The method broadens synthetic-data coverage and improves transfer learning while remaining plug-and-play, although it currently supports only class-level conditioning and may require denoiser fine-tuning for prior alignment.

Motivation and problem statement

Diffusion-based dataset distillation (DD) methods such as MinimaxDiffusion, D$^4$M, and MGD$^3$ condition generation on textual class prompts through cross-attention layers inside the denoiser. The authors identify a structural weakness in this design: because semantic conditioning is injected only after latent encoding and noise addition, prompt signals act correctively rather than co-evolutionarily during denoising. Textual prompts come to dominate the generative trajectory, weakening the contribution of encoder-derived visual latents and producing "over-corrected" samples that are label-relevant but visually distorted—exhibiting unnatural shapes, text-like textures, and oversimplified silhouettes. This diagnosis is supported qualitatively by sample comparisons on ImageNette and CIFAR-10.

Method

EVLF moves vision-language fusion to the encoder-backbone interface, before the diffusion process begins. A VAE encoder produces a spatial image latent $z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$ and a text encoder produces class embeddings $e_{\text{text}}$. Both are projected into a shared dimension, and a lightweight single-block cross-attention module—image tokens as queries, text tokens as keys/values—fuses them with a residual connection, layer normalization, and a feed-forward layer to yield $z_{\text{fused}}$, which serves as the initialization for the generative process.

The module is trained with a dual objective:

$$\mathcal{L}_{\text{CA}} = \lambda_1 \mathcal{L}_{\text{InfoNCE}} + \lambda_2 \mathcal{L}_{\text{MSE}}$$

where the MSE term anchors the fused latent to the original image latent (visual preservation) and an InfoNCE term with a learnable projector aligns fused latents with same-class text embeddings (semantic alignment). Training is inexpensive: 4 epochs, batch size 16, AdamW, with $\lambda_1 = 0.1$ fixed and $\lambda_2$ linearly warmed from 0.05 to 1.0 over two epochs.

A practical caveat: when the host pipeline reuses a pretrained denoiser without domain adaptation (as in D$^4$M), the fused latent distribution may mismatch the pretrained denoising prior. EVLF therefore includes an optional fine-tuning step using the standard diffusion loss on fused latents; for pipelines like MGD$^3$ that already adapt or do not require adaptation, the denoiser stays frozen. The method requires no changes to training schedules, losses, or denoiser architectures, making it plug-and-play for any encoder-equipped diffusion DD pipeline.

Experimental results

Evaluations span CIFAR-10/100 (32×32), Tiny-ImageNet (64×64), ImageNet subsets (ImageNette, ImageWoof, ImageIDC at 256×256), and full ImageNet-1K (224×224), with results averaged over three seeds. EVLF is applied to both D$^3$0M and MGD$^3$1 and compared against generative baselines (MinimaxDiffusion, DiT backbones) and non-generative methods (SRe$^3$2L, RDED, DM, IDC-1, Herding).

Key quantitative findings:

Setting	Baseline	+EVLF	Gain
ImageIDC, IPC 10	D$^3$3M: 47.7%	57.3%	+9.6%
CIFAR-10, IPC 10	D$^3$4M: 37.6%	45.7%	+8.1%
ImageNette, IPC 10–50 avg	D$^3$5M	—	+4.9% avg
ImageWoof, IPC 100, ResNetAP-10	MGD$^3$6: 64.3%	68.1%	+3.8%
Tiny-ImageNet, IPC 10	D$^3$7M: 42.5%	49.2%	+6.7%

On ImageNet-1K, MGD$^3$8+EVLF reaches 51.3% at IPC 10 and 61.9% at IPC 50, surpassing all compared methods including RDED and MinimaxDiffusion. Transfer learning experiments following the KRR-ST protocol show that models pretrained on MGD$^3$9+EVLF-distilled ImageNet-1K data achieve the best fine-tuning accuracy across CIFAR-10, CIFAR-100, Dogs, and Flowers—for example, 76.03% on Flowers versus 70.45% for KRR-ST.

The largest gains occur precisely where late-fusion over-correction should be most damaging: low IPC budgets and fine-grained datasets (ImageIDC, ImageWoof). This consistency supports the paper's central claim that early fusion mitigates prompt dominance rather than merely adding capacity.

Ablations and analysis

An ablation on ImageIDC within the D$z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$0M pipeline isolates the contributions of the cross-attention module and denoiser fine-tuning. Both components independently improve over the baseline (e.g., at IPC 10: baseline 47.7%, CA-only 51.1%, FT-only 54.1%, combined 57.3%), confirming that early fusion and prior alignment address complementary issues.

t-SNE visualization shows that D$z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$1M and MGD$z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$2 synthetic embeddings occupy narrow regions of the real-data manifold, whereas EVLF variants cover substantially broader regions, indicating improved intra-class diversity—a property prior work links directly to downstream accuracy. A parameter study of $z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$3 shows that any nonzero text-injection weight yields large gains in both validation accuracy and coverage (measured via 20th-nearest-neighbor radii of real samples), while performance is insensitive to further increases; $z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$4 gives the most stable accuracy. Qualitative comparisons on CIFAR-10 birds and ImageWoof beagles show more coherent structure, richer textures, and fewer off-class artifacts than D$z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$5M.

Limitations and open questions

The paper's formulation is restricted to class-level conditioning; instance-level, multi-label, and compositional prompts are not addressed, leaving fine-grained per-sample control unexplored. The optional denoiser fine-tuning introduces a pipeline-dependent design decision—the paper does not provide a general criterion for when it is necessary beyond noting distributional mismatch. Additionally, the coverage metric used in the parameter analysis depends on a chosen neighborhood radius ($z_{\text{img}} \in \mathbb{R}^{H\times W\times C}$6), and the sensitivity of conclusions to this choice is not examined.

Conclusion

EVLF reframes semantic conditioning in diffusion-based dataset distillation as an encoder-stage fusion problem rather than a denoising-stage correction problem. A lightweight residual cross-attention block, trained with MSE and InfoNCE objectives, produces fused latents that preserve visual structure while encoding class semantics, yielding consistent accuracy improvements across architectures, resolutions, and IPC settings—including state-of-the-art results on ImageNet-1K—and improved transfer learning. Its plug-and-play compatibility makes it broadly applicable to existing diffusion DD pipelines, though its scope remains limited to class-level conditioning.