Dataset Condensation: Methods & Insights

• Dataset condensation is the synthesis of a small set of synthetic samples that retains the essential learning signal of much larger datasets.
• It employs techniques such as gradient matching and distribution matching to optimize synthetic data for rapid, high-performance training.
• Recent advances demonstrate its potential in reducing computational costs, enhancing privacy, and supporting continual learning and model selection.

Dataset condensation is the process of synthesizing a compact, highly informative synthetic dataset that preserves the essential learning signal of a much larger original dataset. Rather than selecting or compressing real examples, modern dataset condensation methods directly optimize a small set of synthetic samples—often dramatically fewer per class—such that training on these samples enables neural networks to generalize almost as well as training on the full data. The approach has gained importance due to the escalating costs in computation, memory, and privacy risk associated with storing and repeatedly training on large-scale datasets. Recent advances frame dataset condensation as a rigorous optimization problem, bridging statistical matching, gradient-based learning, and knowledge distillation to achieve extreme data compression, fast training, and new applications in model selection, continual learning, and privacy.

1. Core Principles and Definitions

Dataset condensation is formally defined as the synthesis of a small set of data points $\mathcal{S}$ (often, $|\mathcal{S}| \ll |\mathcal{T}|$ for original dataset $\mathcal{T}$) such that training neural networks from scratch on $\mathcal{S}$ achieves test set performance close to that achieved with $\mathcal{T}$. This is achieved by optimizing $\mathcal{S}$ so that models initialized at $\theta_0$ and trained on $\mathcal{S}$ follow similar optimization dynamics—e.g., weight updates, gradient trajectories, or model embeddings—as when trained on $\mathcal{T}$ (Zhao et al., 2020).

Fundamentally, dataset condensation differs from classical coreset selection and compression in that it does not restrict $\mathcal{S}$ to be a subset or mixture of real examples. Instead, $\mathcal{S}$ is free to contain arbitrary synthetic samples, which can be learned directly via backpropagation against task-specific objectives, such as gradient matching, distribution matching in deep feature space, or adversarial proxy losses. The resulting condensed dataset can then serve as a drop-in replacement for the original in downstream training and evaluation pipelines.

2. Methodologies: Objectives and Algorithms

Several categories of dataset condensation objectives have emerged, each reflecting different theoretical and practical considerations:

• Gradient Matching: The predominant framework in early work (Zhao et al., 2020) is to minimize the difference between gradients computed on synthetic and real batches:

$$\min_{\mathcal{S}} \mathbb{E}_{\theta_0 \sim P(\theta_0)} \left[ \sum_t D(\nabla_\theta \mathcal{L}^\mathcal{S}(\theta_t), \nabla_\theta \mathcal{L}^\mathcal{T}(\theta_t)) \right]$$

where $D$ is typically a sum of cosine similarities computed layer- and output-wise.

• Distribution Matching: Later works (Zhao et al., 2023, Zhang et al., 2023) frame condensation as matching the distributions of deep feature representations, often using Maximum Mean Discrepancy (MMD) or other moment-matching kernels. In the M3D method (Zhang et al., 2023), both datasets are embedded into a reproducing kernel Hilbert space, and all orders of moments are aligned:

$$\text{MMD}^2 = \|\mu[\mathbb{P}_\mathcal{T}] - \mu[\mathbb{P}_\mathcal{S}]\|_\mathcal{H}$$

• Generative Model Condensation: Instead of producing explicit images in pixel space, some methods (Zhang et al., 2023, Lee et al., 2022) condense a dataset to the parameters of a generative model, codebook, or decoders in a compressed latent space, sidestepping the parameter explosion seen in direct pixel optimization with increasing resolution or class count.
• Contrastive and Category-aware Losses: Recent objectives improve class discriminability by maximizing the distinction between class-specific gradients (Lee et al., 2022), augmenting with contrastive or interclass losses (Zhang et al., 2023), or incorporating soft category-aware matching via Gaussian Mixture Models (Shao et al., 21 Apr 2024).
• Efficient Parameterization and Regularization: Synthetic data can be parameterized efficiently via multi-formation functions (e.g., upsampling/interpolation, as in (Kim et al., 2022)), or with hierarchical/pruned memory containers (Zheng et al., 2023), allowing for a higher diversity of synthesized examples within the same parameter budget.
• Tailored Objectives for Specific Domains: For recommendation (Wu et al., 2023), text (Wu et al., 2023), and time series (Liu et al., 12 Mar 2024, Ding et al., 4 Jun 2024), condensation objectives are adapted to match task-specific characteristics, such as discrete user–item interactions, semantic text summaries using LLMs, or dual-domain (time/frequency) surrogate objectives.

3. Performance, Evaluation, and Scaling

Performance is assessed by training a neural network from scratch using the condensed set and measuring test accuracy or loss relative to models trained on the original dataset. Key findings across benchmarks include:

• On MNIST, using only 1 image per class, accuracies above 90% are achievable with gradient matching, which is substantially higher than classical coreset selection (Zhao et al., 2020).
• On CIFAR-10, IDC (Kim et al., 2022) and M3D (Zhang et al., 2023) report test accuracies up to 10–20 percentage points better than prior state-of-the-art for 10–50 IPC, and even surpass optimization-oriented methods on high-resolution ImageNet settings.
• On practical large-scale benchmarks, such as ImageNet-1k, generative model condensation (Zhang et al., 2023) and statistical forms of category-aware matching (Shao et al., 21 Apr 2024) yield synthetic datasets of 0.78% of the original size that enable 48.6% ResNet-18 top-1 accuracy (relative to 34.2% using the full downscaled dataset with a synthetic proxy).

Evaluation protocols have been standardized by DC-BENCH (Cui et al., 2022), which exposes critical factors influencing condensation efficacy, including the choice of data augmentation (up to 10% absolute accuracy variation), the condensation compression ratio, and the transferability of synthetic datasets across architectures. Synthesis-based methods show stronger gains at extreme compression (1–10 IPC), but their advantage narrows for higher IPC or across widely different network architectures. Correlation between proxy-set performance (e.g., for neural architecture search) and full-dataset ranking can even be negative for many methods, revealing challenges in transferability.

4. Enhancements, Design Trade-Offs, and Limitations

Several methodological enhancements have addressed earlier bottlenecks:

• Contrastive Matching and Warm-up: DCC (Lee et al., 2022) modifies the loss to sum over class gradients (contrastive signals), with a bi-level warm-up to stabilize optimization, especially on fine-grained data.
• Decomposed Distribution Matching: Recent advances decompose feature distributions into content and style, aligning not only global statistics but also intra-class diversity (using KL-divergence regularization) and higher-order moments (Malakshan et al., 6 Dec 2024).
• Hierarchical Memory and Pruning: HMN (Zheng et al., 2023) organizes condensed representations at dataset, class, and instance levels, supporting redundancy removal via instance-level pruning.
• One-Line and Modular Plugins: For domain specificity, CondTSF (Ding et al., 4 Jun 2024) introduces a one-line additive rule to enforce value matching in time series forecasting, while DConRec (Wu et al., 2023) utilizes probabilistic re-parameterization for discrete recommendation data.
• Calibration for Model Selection: HCDC (Ding et al., 27 May 2024) optimizes synthetic validation sets to align hypergradients (implicit differentiation with Neumann series) so that hyperparameter search or architecture ranking on condensed data mirrors that on the original.

Limitations remain around generalizability: most methods optimize condensation for a specific model architecture and training regime. Transfer to unfamiliar architectures, loss functions, or data augmentations may yield suboptimal performance or incorrect model rankings (Cui et al., 2022). Achieving cross-architecture robustness and reliable proxying for search are open challenges.

5. Applications in Continual Learning, Model Search, and Privacy

Condensed datasets enable several downstream tasks:

• Continual and Incremental Learning: Condensed examples can be replayed for rehearsal, supporting continual learning under memory constraints and outperforming conventional selection-based rehearsal (Zhao et al., 2020, Lee et al., 2022).
• Hyperparameter and Architecture Search: Synthetic proxies dramatically reduce the cost of evaluating candidate models (Ding et al., 27 May 2024), provided the condensation objective addresses validation performance preservation.
• Federated and Edge Learning: Given large communication and storage costs, transmitting or storing condensed data surrogates is more practical than raw datasets or large models (Zhao et al., 2020, Kim et al., 2022).
• Machine Unlearning: Condensed datasets form the backbone of efficient machine unlearning schemes by allowing quick retraining or modular removal of “forgotten” samples while defending against membership inference and inversion attacks (Khan, 31 Jan 2024).
• Domain-Specific Extensions: Dual-domain or plugin methods for time series (Liu et al., 12 Mar 2024, Ding et al., 4 Jun 2024), content-based recommendation (Wu et al., 2023), and graph condensation (Fu et al., 23 Dec 2024) extend the utility of dataset condensation to structured and sequence data, with adapted objectives to preserve domain-specific information.

6. Current Benchmarks, Evaluation Protocols, and Future Prospects

Standardized benchmarks such as DC-BENCH (Cui et al., 2022) provide protocols for isolating the effect of condensation from that of model architecture, data augmentations, and compression ratios. The importance of evaluating over multiple datasets, architectures, and tasks (e.g., NAS, continual learning) is now widely recognized.

Recent trends point to:

• Soft category-aware matching that interpolates between global and per-class statistical alignment via GMM models, validated by theoretical convergence of both KL divergence and landscape flatness (Shao et al., 21 Apr 2024).
• Efficient parameterization techniques scaling to ultra-high resolution and multi-class datasets (e.g., via generative or latent factorization) (Zhang et al., 2023, Lee et al., 2022).
• Decomposed matching strategies, synthesizing by content and style, to ensure both representative prototypes and intra-class diversity (Malakshan et al., 6 Dec 2024).
• Multi-size condensation methods that enable flexible subset selection for fluctuating device resources, mitigating the subset degradation problem (He et al., 10 Mar 2024).
• Unified bi-directional graph condensation based on information bottleneck theory for multi-scale, non-Euclidean data (Fu et al., 23 Dec 2024).

Emerging work is beginning to bridge the gap between practical fast condensation (via distribution or moment matching) and high-fidelity “optimization-oriented” matching. There is a growing emphasis on improving out-of-distribution generalization, privacy-preserving deployment, domain adaptation, and theoretical guarantees.

7. Summary Table: Core Methodological Dimensions

Category	Key Objective	Representative Approaches
Gradient Matching	Match parameter/gradient trajectories	DC (Zhao et al., 2020), IDC (Kim et al., 2022)
Distribution Matching	Match feature statistics (MMD, etc.)	M3D (Zhang et al., 2023), IDM (Zhao et al., 2023)
Generative Condensation	Condense to generator/codebook	GenModel (Zhang et al., 2023), Latent Factorization (Lee et al., 2022)
Content/Style Decoupling	Separate feature and style matching	DDM (Malakshan et al., 6 Dec 2024)
Domain-Specific	Plugin objectives or latent factorization	CondTSF (Ding et al., 4 Jun 2024), TF-DCon (Wu et al., 2023), DConRec (Wu et al., 2023)

Each approach is selected and tuned according to the available computational, memory, and storage resources, the statistical and semantic complexity of the data, and the requirements for transferability, explainability, or privacy. The continued development and empirical evaluation of these strategies signal ongoing progress toward robust, efficient, and general-purpose dataset condensation.