---
title: Dataset Condensation Methods
url: https://www.emergentmind.com/topics/dataset-condensation-dc
type: topic
---

# Dataset Condensation Methods

Searching arXiv for recent and foundational papers on dataset condensation to ground the article in the current literature.
Dataset Condensation (DC) is a data-efficient learning paradigm that synthesizes a small labeled dataset intended to preserve the training utility of a much larger original dataset. Rather than selecting a subset of real examples, DC learns synthetic samples or compact data representations so that models trained from scratch on the condensed set achieve performance approaching that of full-data training while using substantially fewer samples, storage, and compute [2006.05929], [2207.09639]. Across the literature, DC has been formulated through gradient matching, trajectory matching, distribution matching, kernel methods, latent or generative parameterizations, and training-free clustering-based surrogates, with applications spanning computer vision, graph learning, continual learning, neural architecture search, privacy-sensitive settings, and clinical data sharing [2006.05929], [2206.07746], [2602.21717], [2509.10367].

## 1. Conceptual scope and problem formulation

DC targets the setting in which a full training set $T$ is replaced by a much smaller synthetic set $S$ such that training a model on $S$ yields generalization close to training on $T$ [2006.05929]. The canonical goal is to learn a synthetic dataset $S=\{(s,y)\}$ whose samples are directly optimized for downstream training, rather than merely chosen as representative real examples as in coreset selection [2006.05929]. This distinction is central: coreset methods assume useful real exemplars exist and are recoverable by selection heuristics, whereas DC allows the condensed set to be synthetic and explicitly optimized for model training [2006.05929], [2207.09639].

A common bilevel expression of the problem writes the condensed dataset as the solution of an outer optimization that evaluates performance on the real distribution, subject to an inner optimization that trains a model on the condensed set [2503.13935], [2509.10367], [2602.21717]. In parallel, a data-centric formulation replaces explicit inner training with discrepancy minimization between the empirical distributions induced by the real and synthetic datasets [2509.10367]. This discrepancy-based view unifies gradient-space, parameter-space, feature-space, kernel, and optimal-transport formulations, and extends the objective beyond generalization to robustness and privacy [2509.10367].

The practical motivation is computational as much as statistical. Full-data training incurs storage, data-loading, and optimization cost, and these burdens are amplified when many models must be trained, as in hyperparameter search or neural architecture search [2006.05929], [2207.09639]. DC addresses a regime in which one wants a tiny but highly informative proxy dataset that supports training from scratch in minutes, rehearsal under memory constraints, or repeated evaluation across architectures [2006.05929].

## 2. Foundational objective families

The earliest influential modern formulation is gradient matching. In "Dataset Condensation with Gradient Matching" [2006.05929], the objective is to learn $S$ so that gradients of network parameters computed on synthetic data match those computed on real data across random initializations. With network $\phi_\theta$ and losses $\mathcal{L}^T$ and $\mathcal{L}^S$ on real and synthetic batches, a one-step objective is

\[
\min_{\mathcal{S}} \; \mathbb{E}_{\theta \sim p(\theta)} \; \sum_{b \in \mathcal{B}_r} \left\| \nabla_{\theta} \mathcal{L}^T(b; \theta) - \nabla_{\theta} \mathcal{L}^S(\mathcal{S}; \theta) \right\|_2^2.
\]

The method uses class-conditional matching and refines the objective into curriculum gradient matching over several inner steps, avoiding full unrolling of long optimization trajectories while still encouraging parameters trained on $S$ to follow the gradient trajectory of training on $T$ [2006.05929].

A closely related line emphasizes full trajectory alignment. DC-BENCH summarizes "Dataset Distillation by Matching Training Trajectories" as parameter matching between student trajectories on synthetic data and reference trajectories on real data [2207.09639]. Clinical work on mode-connectivity-based surrogates positions MTT, TESLA, FTD, and DATM as prominent trajectory-matching methods and argues that trajectory supervision captures rich global information about optimization flow [2510.05805]. In that clinical setting, raw SGD trajectories are replaced by quadratic Bézier surrogates connecting initial and final states, with segment-based alignment loss
\[
\mathcal{L}_{\text{align}} = \frac{\|\tilde{\theta}_N - \theta_{\text{target}}\|_2^2}{\|\theta_{\text{start}} - \theta_{\text{target}}\|_2^2},
\]
reducing storage and smoothing supervision [2510.05805]. This suggests that trajectory information remains valuable even when the raw trajectory itself is replaced by a lower-curvature surrogate.

Distribution matching (DM) eliminates the inner optimization loop and instead aligns real and synthetic distributions in a feature space [2206.00240], [2307.09742]. In the DM formulation highlighted in the privacy study, the synthetic set minimizes
\[
\min_{S}\ \mathbb{E}_{\psi_\theta,\ \omega\sim\Omega}\Big\|\frac{1}{|T|}\sum_{i=1}^{|T|}\psi_\theta\big(A(x_i,\omega)\big)\ -\ \frac{1}{|S|}\sum_{j=1}^{|S|}\psi_\theta\big(A(s_j,\omega)\big)\Big\|^2
\]
over random feature extractors and augmentations [2206.00240]. In practice, several DM variants use class-wise mean matching in learned embeddings, often interpretable as a linear-kernel MMD [2307.09742], [2406.09860].

Kernel-based formulations constitute another branch. KIP casts condensation as minimizing a kernel ridge regression mismatch under NTK [2206.00240], [2207.09639]. More generally, discrepancy-based work places MMD, Wasserstein distance, IPMs, gradient-space IPMs, parameter discrepancy, value discrepancy, and characteristic discrepancy within a common formalism [2509.10367].

## 3. Efficiency-oriented reformulations and improved matching criteria

A major development after early bilevel methods was the shift toward more scalable surrogates. Improved Distribution Matching (IDM) identifies two shortcomings of naive DM—imbalanced feature numbers and unvalidated embeddings—and addresses them with partitioning-and-expansion augmentation, enriched model sampling through a queue of random and weakly trained feature extractors, and a class-aware regularizer [2307.09742]. The resulting objective combines class-conditional feature mean matching with an accuracy-weighted cross-entropy term on synthetic samples [2307.09742]. This design substantially improves over naive DM on CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet Subset while remaining efficient [2307.09742].

Latent Quantile Matching (LQM) argues that mean matching in latent space has weak matching power and lacks outlier regularization [2406.09860]. Instead of aligning only mean embeddings, LQM matches optimal latent quantiles derived from the Cramér–von Mises criterion. For class $c$, with real embeddings $\mathcal{E}^{\mathcal{T}}_c$ and synthetic embeddings $\mathcal{E}^{\mathcal{S}}_c$, the loss is
\[
\mathcal{L}_c \;=\; \frac{1}{\beta_c}\,\Big\|\,\mathcal{F}_q\big(Q,\mathcal{E}^{\mathcal{T}}_c\big)\;-\;\mathcal{F}_s\big(\mathcal{E}^{\mathcal{S}}_c\big)\,\Big\|^2,
\]
where $Q=\{(2i-1)/(2\beta_c)\}_{i=1}^{\beta_c}$ [2406.09860]. The reported gains are most pronounced under small synthetic budgets and in continual graph learning [2406.09860].

Decomposed Distribution Matching further refines DM by separating style and content [2412.04748]. The paper argues that DM often captures style-like low-level statistics while collapsing content diversity, and therefore introduces style matching via feature-map moments and optional Gram matrices, combined with within-class diversity maximization through symmetric KL divergence on softmax outputs [2412.04748]. The total objective is
\[
L_{\text{total}} = L_{\text{DM}} + \lambda_s L_{\text{style}} - \lambda_d L_{\text{div}},
\]
and the reported improvements reach up to $4.1\%$ on CIFAR10, $4.2\%$ on CIFAR100, $4.3\%$ on TinyImageNet, $2.0\%$ on ImageNet-1K, $3.3\%$ on ImageWoof, $2.5\%$ on ImageNette, and $5.5\%$ in continual learning accuracy [2412.04748].

Several works instead improve the synthetic data parameterization itself. KFS factorizes content and style by learning class-specific latent codes and a small pool of shared decoders, allowing $N=M\times K$ synthetic examples per class while keeping parameter growth modest [2208.10494]. HMN introduces a three-tier hierarchical memory network with dataset-level, class-level, and instance-level memories, then prunes instance memories using AUM-based double-end pruning [2310.07506]. A generative-model formulation learns a conditional generator and shared codebook rather than synthetic pixels, together with intra-class and inter-class feature losses, and reports the first successful condensation on ImageNet-1k [2309.07698]. SCORE addresses a different bottleneck in large-scale realistic condensation: soft-label storage. It formulates condensation as greedy selection under coding-rate objectives balancing informativeness, discriminativeness, and compressibility, and reports that even with $30\times$ compression of soft labels, ImageNet-1K performance decreases by only $5.5\%$ at IPC 10 and $2.7\%$ at IPC 50 [2503.13935].

## 4. Modalities, domains, and application regimes

Although much of the literature is vision-centric, DC has broadened substantially. For graph classification, "Condensing Graphs via One-Step Gradient Matching" replaces expensive multi-step matching with a one-step objective at initialization and models discrete graph structure using Bernoulli logits with a Binary Concrete relaxation [2206.07746]. The method reports reducing dataset size by $90\%$ while approximating up to $98\%$ of original performance and being about $15\times$ faster than multi-step gradient matching on CIFAR10 for synthesizing 500 graphs [2206.07746]. The same paper extends one-step matching to node classification with substantial speedups [2206.07746].

In tabular learning, C$^{2}$TC reformulates condensation as a training-free class-adaptive cluster allocation problem. It replaces gradient-based inner loops with class-wise clustering in a feature space constructed by hybrid categorical feature encoding, optimized by heuristic first-improvement local search [2602.21717]. The method is reported to improve efficiency by at least two orders of magnitude over state-of-the-art baselines across 10 real-world datasets, while achieving superior downstream performance [2602.21717]. This suggests that not all effective condensation requires synthetic pixels or differentiable student models.

Clinical applications motivate both privacy-aware and model-agnostic variants. The mode-connectivity surrogate method targets clinical tabular and time-series tasks, emphasizing AUROC and AUPRC under severe class imbalance and reporting near-lossless performance at low condensation ratios on MIMIC-III [2510.05805]. A later clinical paper extends DC to non-differentiable models such as XGBoost and Cox regression using zero-order optimization and integrates formal differential privacy guarantees, reporting $\varepsilon$ values typically in approximately $[0.5,3.8]$ with $\delta=10^{-5}$ while preserving strong utility across six datasets [2603.09356].

The multi-domain setting introduces another axis of complexity. DAM defines Multi-Domain Dataset Condensation and augments standard DC/DM/MTT pipelines with a training-time Domain-Aware Module that uses frequency-based pseudo-domain labeling and learnable spatial masks [2505.22387]. The module is only active during condensation and keeps IPC unchanged. Reported gains include CIFAR-10 IPC 10 improvements from $47.4$ to $49.8$ for DM and from $50.7$ to $57.9$ for MTT, together with consistent gains on PACS, VLCS, and Office-Home [2505.22387].

Finally, DC has been adapted to deep hashing retrieval. IEM uses feature-embedding matching in hashing space with early-stage model pools and multi-formation parameterization, improving mAP over DM, DSA, and IDC on CIFAR10 and ImageNet subsets while reducing condensation cost relative to gradient-heavy baselines [2305.18076].

## 5. Evaluation practice, empirical patterns, and benchmark findings

DC-BENCH provides the first large-scale standardized benchmark dedicated to dataset condensation and shows that evaluation is highly confounded by augmentation, architecture, initialization, and training budget [2207.09639]. It evaluates methods under multiple augmentations, cross-architecture transfer, a wide IPC range up to 1,000 on CIFAR-10, stability across seeds, and NAS transfer [2207.09639]. One of its main findings is that augmentation alone can drastically change method rankings, and strong selection baselines such as K-Center were previously underestimated [2207.09639].

Several empirical patterns recur across the literature. First, gradient-matching DC is particularly strong at small budgets. In the original gradient-matching paper, on MNIST with 50 images per class and a ConvNet, DC achieves $98.8\pm0.2\%$ versus $99.6\pm0.0\%$ for the whole dataset; on CIFAR-10 with 50 images per class it reaches $53.9\pm0.5\%$ versus $84.8\pm0.1\%$ for full data, outperforming Random, Herding, K-Center, and Forgetting [2006.05929]. Accuracy increases steadily with more images per class, but the gap to full-data training is larger on SVHN and CIFAR-10/100 than on MNIST and FashionMNIST, indicating higher domain complexity [2006.05929].

Second, cross-architecture transfer is real but not uniform. The original DC paper reports that a synthetic set learned with a ConvNet transfers reasonably to LeNet, VGG, and ResNet-18 on MNIST, whereas a set learned with an MLP transfers poorly to ConvNets [2006.05929]. DC-BENCH similarly finds that rankings shift under architecture transfer and that larger models such as ResNet152 perform poorly at very small IPCs but improve with larger budgets [2207.09639]. Some later methods improve transfer substantially; for example, the generative-model approach reports strong gains over pixel-based baselines on AlexNet, VGG11, ResNet18, and MLP using CIFAR-10 synthetic data learned with ConvNet [2309.07698].

Third, distribution-matching methods have become increasingly competitive with optimization-oriented methods. IDM surpasses naive DM and most optimization-based baselines while using fewer resources [2307.09742]. Decomposed DM closes much of the performance gap while scaling to ImageNet-1K [2412.04748]. SCORE targets ImageNet-scale settings and reports strong cross-architecture generalization under realistic-image-plus-soft-label pipelines [2503.13935].

Fourth, certain application settings benefit disproportionately from condensation. In continual learning, the original gradient-matching method reports that replacing rehearsal sample selection with condensed images improves final-stage accuracy by $+2.3\%$ on task-incremental SVHN $\rightarrow$ MNIST $\rightarrow$ USPS without KD [2006.05929]. Decomposed DM reports a $5.5\%$ gain in continual learning accuracy [2412.04748]. In graph continual learning, LQM improves average accuracy while maintaining competitive backward transfer [2406.09860].

A concise summary of benchmark-level observations is useful:

| Setting | Representative finding | Source |
|---|---|---|
| Small IPC | Condensation gains are strongest at very low IPC | [2207.09639] |
| Higher IPC | Gains over strong selection baselines shrink as IPC increases | [2207.09639] |
| Augmentation | Augmentation can drastically alter rankings | [2207.09639] |
| Cross-architecture | Transferability is limited and ranking-sensitive | [2207.09639] |
| NAS proxy | Condensed datasets in modern settings often fail to preserve architecture rankings | [2207.09639] |

One apparent tension in the literature concerns NAS. The original gradient-matching paper reports promising gains, with a CIFAR-10 proxy of 100 images ranking 720 ConvNet architectures with Spearman correlation $0.79$ for top-10 models and evaluating all architectures in $18.8$ minutes versus $8604.3$ minutes on full data [2006.05929]. DC-BENCH, however, reports negative or near-zero correlations for several methods on NAS-Bench-201 and concludes that NAS transfer is currently unsuccessful for condensation datasets in modern settings [2207.09639]. This suggests that NAS utility is highly protocol-dependent.

## 6. Privacy, security, and ongoing controversies

Privacy has become one of the most contested themes in DC. "Privacy for Free: How does Dataset Condensation Help Privacy?" studies DC through the lens of distribution matching and argues that when $m \ll n$, removing one raw sample changes the parameter distribution of models trained on the condensed set by only $O(m/n)$ under its assumptions [2206.00240]. The paper further presents empirical evidence of low membership-inference vulnerability under random initialization of synthetic data and reports that DC-synthesized data can offer stronger utility than several DP-generators at comparable empirical privacy levels on FashionMNIST [2206.00240]. It also emphasizes a key caveat: real-data initialization can leak membership of initialization samples, whereas random initialization reduces visual and membership leakage [2206.00240].

This claim was challenged directly in "No Free Lunch in 'Privacy for Free: How does Dataset Condensation Help Privacy'" [2209.14987]. That critique argues that the theoretical analysis relies on assumptions tantamount to an already private mechanism and that the empirical evaluation uses a confounded baseline. Recomputing the random-subset baseline on the full universe, it reports attack advantage approximately $1.6\%$ versus $1.06\pm1.20\%$ for DM at roughly 1% condensation, concluding that there is no statistically significant privacy improvement over a naive baseline [2209.14987]. It further contrasts this with DP-SGD, which offers formal guarantees and in the cited settings can achieve both better accuracy and lower membership attack success [2209.14987]. Accordingly, a conservative encyclopedia treatment is that DC may attenuate per-sample influence and may be compatible with privacy-preserving workflows, but empirical robustness to membership inference is not a substitute for formal differential privacy [2206.00240], [2209.14987].

More recent work moves from privacy to security vulnerabilities. "InkDrop: Invisible Backdoor Attacks Against Dataset Condensation" studies backdoors in DM-family methods and shows that a malicious provider controlling the condensation pipeline can implant imperceptible instance-dependent perturbations before condensation, preserving clean task accuracy while driving attack success rate to near 1.0 across datasets and downstream architectures [2603.28092]. Reported representative outcomes with a ConvNet synthesizer include CTA approximately $0.6217\pm0.0006$ and ASR approximately $0.9967$ on CIFAR-10 DM, with similarly near-perfect ASR on STL-10, Tiny-ImageNet, and SVHN [2603.28092]. Standard defenses such as PIXEL, ABS, RNP, and PDB either fail or incur severe CTA degradation [2603.28092]. A plausible implication is that condensed datasets should be treated as high-risk artifacts whenever their provenance is uncertain.

Two practical conclusions emerge from these privacy and security studies. First, DC should not be presumed private by default; privacy depends on initialization, threat model, evaluation protocol, and whether formal noise mechanisms are used [2206.00240], [2209.14987]. Second, condensed datasets can encode adversarial intent in ways that transfer across architectures, so provenance, auditing, and defense-in-depth are necessary before deployment [2603.28092].

## 7. Open directions and broader perspective

The field has gradually shifted from asking whether DC works at all toward asking what discrepancy should be minimized, how the synthetic artifact should be parameterized, and which objectives besides accuracy should be preserved. The discrepancy-based perspective makes this shift explicit by treating DC as reduced-support distribution approximation under a chosen discrepancy, with optional regularizers for privacy, robustness, diversity, or other downstream desiderata [2509.10367]. Under this view, gradient matching, feature matching, MMD, Wasserstein, parameter matching, and kernel methods are not isolated ideas but specializations of a single framework [2509.10367].

Several open problems recur across the literature. Scaling to high-resolution, large-scale datasets remains difficult for pixel-parameterized methods, though generative formulations, realistic-image selection pipelines with soft labels, and decomposed DM have advanced the frontier [2006.05929], [2309.07698], [2412.04748], [2503.13935]. Architecture transfer remains imperfect, especially across qualitatively different model families such as ConvNets, ViTs, and non-neural classical models [2207.09639], [2603.09356]. Robust multi-objective optimization—simultaneously preserving accuracy, robustness, privacy, and security—remains underexplored [2509.10367].

Another major open direction is modality expansion. Graph, tabular, and clinical works indicate that DC is not inherently tied to pixel synthesis or differentiable image classifiers [2206.07746], [2602.21717], [2603.09356]. Inference from these works suggests that future progress may depend less on a universal condensation algorithm than on modality-aware abstractions: probabilistic discrete structures for graphs, clustering-based surrogates for tabular data, zero-order optimization for non-differentiable clinical models, and structured latent or hierarchical memories for large-scale vision data [2206.07746], [2602.21717], [2310.07506].

At the same time, standardization remains essential. DC-BENCH shows that without controlled protocols, claims about method superiority can be dominated by augmentation policy, architecture choice, training schedule, or initialization [2207.09639]. This suggests that the maturity of DC as a research area depends not only on better algorithms, but also on stable evaluation methodology, explicit threat modeling, and artifact auditing.

Source: https://www.emergentmind.com/topics/dataset-condensation-dc