Dataset Distillation with Distribution Constraints

• The paper presents a novel framework that enforces quantitative distribution constraints to ensure synthetic datasets closely mimic the original, enhancing downstream task performance.
• Techniques such as moment matching, optimal transport, and diffusion-based constraints are used to achieve high representativeness and diversity in distilled datasets.
• Empirical results demonstrate improved cross-architecture generalization and robustness, with methods showing significant gains in low images-per-class and long-tail scenarios.

Dataset distillation with distribution constraints refers to the principled compression of large-scale datasets into much smaller, synthetic datasets while explicitly enforcing that the empirical or statistical distribution of the synthetic dataset closely resembles that of the original data. The distinctive feature of these techniques is the design and enforcement of quantitative distribution-level constraints to ensure that the condensed set is both representative (faithful to the original support) and diverse (covering the full range of the data manifold), with tight theoretical and empirical links to generalization in downstream tasks.

1. Formal Definitions and Theoretical Foundations

Dataset distillation is formulated as the search for a compact synthetic dataset $S$ (or, more generally, a synthetic data-generating distribution $Q_\theta$) such that training a model on $S$ yields test performance comparable to training on the full dataset $D$. The core innovation in this area is the explicit enforcement of distributional similarity between the empirical distribution $\hat p_S$ of the synthesized set and the original data distribution $p_D$ according to metrics such as Maximum Mean Discrepancy (MMD), Wasserstein distance, Kullback–Leibler divergence, or specialized kernel spectral distances (Liu et al., 30 Mar 2026, Liu et al., 2023, Wu et al., 2 Mar 2026).

A precise objective can take the following generic form: $S^* = \arg\min_S\, \mathcal D(\hat p_S,\,p_D),$ where $\mathcal D$ is a suitable divergence or distance measure. Under the memorize–generalize assumption, distribution matching is theoretically justified as minimizing an upper bound on downstream risk. Specifically, Liu et al. (Liu et al., 30 Mar 2026) prove the equivalence between optimal dataset distillation and distribution matching, with statistical efficiency determined by the ability of $S$ to cover the modes and tails of $p_D$.

2. Class of Distribution Constraints

2.1. Moment and Kernel-Based Constraints

Early and influential methods employ moment-matching:

• Mean matching: Aligning the mean feature vectors of real and synthetic data (Deng et al., 2024, Ran et al., 2 Dec 2025).
• Covariance matching: Enforcing alignment of higher-order statistics, e.g., feature covariances within classes (Deng et al., 2024).
• MMD: Leveraging universal kernels to match distributions in a Reproducing Kernel Hilbert Space (RKHS), guaranteeing full distributional convergence in the limit (Deng et al., 2024, Ran et al., 2 Dec 2025).
• Spectral alignment: Decomposing alignment into amplitude (diversity) and phase (realism) terms in the frequency domain and weighting these by class size to address long-tail regimes (Wu et al., 2 Mar 2026).

2.2. Optimal Transport (OT) and Wasserstein Constraints

Distribution matching via Wasserstein distance introduces geometric structure beyond first- or second-order moments (Liu et al., 2023, Li et al., 9 Dec 2025). Techniques include:

• Computation of the Wasserstein barycenter of real data features per class, imposing recovery-stage loss functions that align synthetic features to these optimal transport centroids (Liu et al., 2023).
• OT-based losses computed in product manifolds (Euclidean, hyperbolic, spherical) to match intrinsic data geometry (Li et al., 9 Dec 2025).
• Entropic regularization and efficient Sinkhorn approximation for scalability.

2.3. Latent-Space and Diffusion-Based Constraints

Diffusion models and latent generative architectures introduce constraints directly in latent or noise spaces:

• Distributional alignment in the latent domain: Use of forward/inverse diffusion processes (e.g., DDIM inversion) to map data to a high-normality Gaussian space, then selection of distilled images ensures the empirical moments (or higher moments, e.g., skewness) of the distilled latent set match the target latent distribution (Zhao et al., 23 May 2025).
• Minimax objectives with contrastive terms: Simultaneous enforcement of representativeness (pulling toward unexplored regions of the real support) and diversity (pushing synthesized samples apart) via memory banks and minimax cosine similarity terms (Li et al., 26 May 2025).
• Noise optimization ("NOpt"): Optimization of diffusion noise variables at each timestep to align statistics and features of synthesized and real data throughout the denoising trajectory (Liu et al., 30 Mar 2026).

2.4. Distribution Constraints in Cross-modal and Structured Settings

For structured modalities (e.g., audio-visual), decoupled matching of private and common embeddings, sample-level and global alignment via prototype tracking, and cross-modal optimal transport are used to maintain both modality-specific and shared distributional statistics (Li et al., 22 Nov 2025).

3. Optimization Schemes and Algorithmic Architectures

Optimization of distribution constraints is tailored to the chosen metric and generative backbone:

• Bi-level and meta-gradient formulations: Matching model-training trajectories (via "expert trajectory" alignment), often in combination with MMD or cross-entropy constraints, are used to enforce dynamics-level alignment (Ran et al., 2 Dec 2025, Qin et al., 2024).
• Single-level direct matching: Gradient-based minimization of differentiable distributional objectives, e.g., WMDD or D³HR recover-stage latent optimization (Liu et al., 2023, Zhao et al., 23 May 2025).
• Training-free guidance: Incorporation of all guidance at sampling time (not by parameter updates), e.g., DMGD uses classifier-free guidance for semantics and Sinkhorn OT for support coverage (Wang et al., 5 May 2026).

Implementation typically involves:

• Memory banks or prototype caches for targeting unexplored regions (Li et al., 26 May 2025, Su et al., 2024).
• Self-adaptive memory or eviction strategies (removing the most redundant latents to maximize support coverage) (Li et al., 26 May 2025).
• Efficient group, progressive, or batchwise sampling in the latent or feature spaces (Zhao et al., 23 May 2025, Wang et al., 5 May 2026).

4. Empirical Outcomes and Comparative Performance

Quantitative evaluations consistently demonstrate that explicit enforcement of distribution constraints yields state-of-the-art results across a range of tasks:

• On high-resolution benchmarks, e.g., ImageNet-1K, methods such as WMDD (Liu et al., 2023), D³HR (Zhao et al., 23 May 2025), D⁴M (Su et al., 2024), and DsCo (Liu et al., 30 Mar 2026) outperform both classical distribution matching and gradient/trajectory-matching bi-level optimization, especially at low images-per-class (IPC) (Liu et al., 2023, Su et al., 2024, Zhao et al., 23 May 2025).
• Spectral (CSDM) and geometry-aware (GeoDM) matching deliver robustness to long-tail class imbalance and low IPC, with 14 percentage point gains over prior methods in extreme imbalance settings (Wu et al., 2 Mar 2026, Li et al., 9 Dec 2025).
• Cross-architecture generalization is markedly improved when distribution constraints are imposed in latent space with a learned generative prior, avoiding collapse to architecture-specific modes (Su et al., 2024, Tan et al., 13 Jan 2025).

Ablations reveal that each component of distribution matching (e.g., inter-sample class centralization, covariance alignment, high-moment matching) contributes distinct non-redundant improvements (Deng et al., 2024). Representativeness–diversity tradeoffs remain an active area of study, with new minimax objectives (e.g., in self-adaptive memory) showing dramatic accuracy drops when relaxed (Li et al., 26 May 2025).

5. Limitations, Open Challenges, and Extensions

While the use of distribution constraints has advanced the robustness and efficiency of dataset distillation, limitations are evident:

• Most methods rely on empirical feature or latent spaces induced by pretrained encoders or generative models; the quality of constraints depends strongly on the manifold linearization properties of these encoders (Li et al., 26 May 2025).
• Theoretical guarantees (e.g., bounds on excess risk under manifold mismatch) are beginning to emerge (see GeoDM and DsCo), but further refinements are needed for non-stationary or non-shift-invariant metrics (Li et al., 9 Dec 2025, Liu et al., 30 Mar 2026).
• Extremely low-IPC and extreme multimodality regimes challenge simple moment matching; mixture models, adaptive kernel/feature networks, and geometry-aware embeddings are promising future directions (Zhao et al., 23 May 2025, Wu et al., 2 Mar 2026).
• In the training-free setting, especially with diffusion backbones, ensuring global manifold coverage without explicit parameter adaptation remains computationally demanding; advanced sampling and quantization algorithms (e.g., K-means-based distribution approximation, greedy progressive matching) are being actively developed for scalability (Wang et al., 5 May 2026).

6. Implementation and Practical Considerations

Key algorithmic details for practitioners include:

• Precomputing class-wise barycenters or statistical flows can be performed offline and amortized across architectures (Xia et al., 5 Feb 2026, Liu et al., 2023).
• Dynamic memory or prototype management (e.g., self-adaptive eviction, EMA prototypes, or curriculum-based sampling) is crucial for balancing representativeness and diversity without excessive memory overhead (Li et al., 26 May 2025, Li et al., 22 Nov 2025, Zheng et al., 21 Apr 2025).
• In diffusion-based pipelines, alignment can be enforced either in the latent/feature space (via optimal transport between latents) or directly during the noise-optimization steps (e.g., NOpt), with strong gains in both data-accessible and data-free scenarios (Liu et al., 30 Mar 2026, Wang et al., 5 May 2026).
• Geometry and manifold structure can be incorporated into the matching objective by embedding data into a product of spaces with learnable curvature and applying OT (Li et al., 9 Dec 2025).

7. Outlook and Research Directions

Dataset distillation with distribution constraints is converging toward a mature theoretical and practical framework that combines statistical efficiency, representation coverage, and cross-architecture generalization. Recent work has provided theoretical justification for equating distillation with distribution-matching, demonstrated tight generalization bounds when matching in geometry-aware product spaces, and developed scalable, efficient sampling and matching algorithms (Li et al., 9 Dec 2025, Liu et al., 30 Mar 2026, Wu et al., 2 Mar 2026). Ongoing challenges include:

• Extending to non-vision and multi-modal domains, as exemplified in audio-visual dataset distillation (Li et al., 22 Nov 2025).
• Automated adaptation of constraint strength, geometry, and moment order to dataset complexity and class imbalance.
• Fully training-free, data-free distillation workflows that nonetheless guarantee distributional fidelity and downstream utility.

The field continues to advance rapidly, with new algorithmic motifs (noise optimization, prototype caching, spectral and geometric OT, dynamic curriculum) regularly yielding empirical and theoretical enhancements.