Dataset Condensation Methods

Updated 11 July 2026

Dataset Condensation is a data-efficient paradigm that synthesizes small, optimized synthetic datasets to approximate the training utility of larger datasets through techniques such as gradient and distribution matching.
It employs methods like gradient matching, trajectory alignment, and kernel-based formulations to significantly reduce storage, computation, and training time without severely compromising accuracy.
Applications span computer vision, graph learning, continual learning, and privacy-sensitive domains, while challenges remain in cross-architecture transfer and robust evaluation.

Searching arXiv for recent and foundational papers on dataset condensation to ground the article in the current literature. Dataset Condensation (DC) is a data-efficient learning paradigm that synthesizes a small labeled dataset intended to preserve the training utility of a much larger original dataset. Rather than selecting a subset of real examples, DC learns synthetic samples or compact data representations so that models trained from scratch on the condensed set achieve performance approaching that of full-data training while using substantially fewer samples, storage, and compute (Zhao et al., 2020, Cui et al., 2022). Across the literature, DC has been formulated through gradient matching, trajectory matching, distribution matching, kernel methods, latent or generative parameterizations, and training-free clustering-based surrogates, with applications spanning computer vision, graph learning, continual learning, neural architecture search, privacy-sensitive settings, and clinical data sharing (Zhao et al., 2020, Jin et al., 2022, Xu et al., 25 Feb 2026, Chen et al., 12 Sep 2025).

1. Conceptual scope and problem formulation

DC targets the setting in which a full training set $T$ is replaced by a much smaller synthetic set $S$ such that training a model on $S$ yields generalization close to training on $T$ (Zhao et al., 2020). The canonical goal is to learn a synthetic dataset $S=\{(s,y)\}$ whose samples are directly optimized for downstream training, rather than merely chosen as representative real examples as in coreset selection (Zhao et al., 2020). This distinction is central: coreset methods assume useful real exemplars exist and are recoverable by selection heuristics, whereas DC allows the condensed set to be synthetic and explicitly optimized for model training (Zhao et al., 2020, Cui et al., 2022).

A common bilevel expression of the problem writes the condensed dataset as the solution of an outer optimization that evaluates performance on the real distribution, subject to an inner optimization that trains a model on the condensed set (Yuan et al., 18 Mar 2025, Chen et al., 12 Sep 2025, Xu et al., 25 Feb 2026). In parallel, a data-centric formulation replaces explicit inner training with discrepancy minimization between the empirical distributions induced by the real and synthetic datasets (Chen et al., 12 Sep 2025). This discrepancy-based view unifies gradient-space, parameter-space, feature-space, kernel, and optimal-transport formulations, and extends the objective beyond generalization to robustness and privacy (Chen et al., 12 Sep 2025).

The practical motivation is computational as much as statistical. Full-data training incurs storage, data-loading, and optimization cost, and these burdens are amplified when many models must be trained, as in hyperparameter search or neural architecture search (Zhao et al., 2020, Cui et al., 2022). DC addresses a regime in which one wants a tiny but highly informative proxy dataset that supports training from scratch in minutes, rehearsal under memory constraints, or repeated evaluation across architectures (Zhao et al., 2020).

2. Foundational objective families

The earliest influential modern formulation is gradient matching. In "Dataset Condensation with Gradient Matching" (Zhao et al., 2020), the objective is to learn $S$ so that gradients of network parameters computed on synthetic data match those computed on real data across random initializations. With network $\phi_\theta$ and losses $\mathcal{L}^T$ and $\mathcal{L}^S$ on real and synthetic batches, a one-step objective is

$\min_{\mathcal{S}} \; \mathbb{E}_{\theta \sim p(\theta)} \; \sum_{b \in \mathcal{B}_r} \left\| \nabla_{\theta} \mathcal{L}^T(b; \theta) - \nabla_{\theta} \mathcal{L}^S(\mathcal{S}; \theta) \right\|_2^2.$

The method uses class-conditional matching and refines the objective into curriculum gradient matching over several inner steps, avoiding full unrolling of long optimization trajectories while still encouraging parameters trained on $S$ 0 to follow the gradient trajectory of training on $S$ 1 (Zhao et al., 2020).

A closely related line emphasizes full trajectory alignment. DC-BENCH summarizes "Dataset Distillation by Matching Training Trajectories" as parameter matching between student trajectories on synthetic data and reference trajectories on real data (Cui et al., 2022). Clinical work on mode-connectivity-based surrogates positions MTT, TESLA, FTD, and DATM as prominent trajectory-matching methods and argues that trajectory supervision captures rich global information about optimization flow (Nganjimi et al., 7 Oct 2025). In that clinical setting, raw SGD trajectories are replaced by quadratic Bézier surrogates connecting initial and final states, with segment-based alignment loss

$S$ 2

reducing storage and smoothing supervision (Nganjimi et al., 7 Oct 2025). This suggests that trajectory information remains valuable even when the raw trajectory itself is replaced by a lower-curvature surrogate.

Distribution matching (DM) eliminates the inner optimization loop and instead aligns real and synthetic distributions in a feature space (Dong et al., 2022, Zhao et al., 2023). In the DM formulation highlighted in the privacy study, the synthetic set minimizes

$S$ 3

over random feature extractors and augmentations (Dong et al., 2022). In practice, several DM variants use class-wise mean matching in learned embeddings, often interpretable as a linear-kernel MMD (Zhao et al., 2023, Wei et al., 2024).

Kernel-based formulations constitute another branch. KIP casts condensation as minimizing a kernel ridge regression mismatch under NTK (Dong et al., 2022, Cui et al., 2022). More generally, discrepancy-based work places MMD, Wasserstein distance, IPMs, gradient-space IPMs, parameter discrepancy, value discrepancy, and characteristic discrepancy within a common formalism (Chen et al., 12 Sep 2025).

3. Efficiency-oriented reformulations and improved matching criteria

A major development after early bilevel methods was the shift toward more scalable surrogates. Improved Distribution Matching (IDM) identifies two shortcomings of naive DM—imbalanced feature numbers and unvalidated embeddings—and addresses them with partitioning-and-expansion augmentation, enriched model sampling through a queue of random and weakly trained feature extractors, and a class-aware regularizer (Zhao et al., 2023). The resulting objective combines class-conditional feature mean matching with an accuracy-weighted cross-entropy term on synthetic samples (Zhao et al., 2023). This design substantially improves over naive DM on CIFAR-10, CIFAR-100, TinyImageNet, and ImageNet Subset while remaining efficient (Zhao et al., 2023).

Latent Quantile Matching (LQM) argues that mean matching in latent space has weak matching power and lacks outlier regularization (Wei et al., 2024). Instead of aligning only mean embeddings, LQM matches optimal latent quantiles derived from the Cramér–von Mises criterion. For class $S$ 4, with real embeddings $S$ 5 and synthetic embeddings $S$ 6, the loss is

$S$ 7

where $S$ 8 (Wei et al., 2024). The reported gains are most pronounced under small synthetic budgets and in continual graph learning (Wei et al., 2024).

Decomposed Distribution Matching further refines DM by separating style and content (Malakshan et al., 2024). The paper argues that DM often captures style-like low-level statistics while collapsing content diversity, and therefore introduces style matching via feature-map moments and optional Gram matrices, combined with within-class diversity maximization through symmetric KL divergence on softmax outputs (Malakshan et al., 2024). The total objective is

$S$ 9

and the reported improvements reach up to $S$ 0 on CIFAR10, $S$ 1 on CIFAR100, $S$ 2 on TinyImageNet, $S$ 3 on ImageNet-1K, $S$ 4 on ImageWoof, $S$ 5 on ImageNette, and $S$ 6 in continual learning accuracy (Malakshan et al., 2024).

Several works instead improve the synthetic data parameterization itself. KFS factorizes content and style by learning class-specific latent codes and a small pool of shared decoders, allowing $S$ 7 synthetic examples per class while keeping parameter growth modest (Lee et al., 2022). HMN introduces a three-tier hierarchical memory network with dataset-level, class-level, and instance-level memories, then prunes instance memories using AUM-based double-end pruning (Zheng et al., 2023). A generative-model formulation learns a conditional generator and shared codebook rather than synthetic pixels, together with intra-class and inter-class feature losses, and reports the first successful condensation on ImageNet-1k (Zhang et al., 2023). SCORE addresses a different bottleneck in large-scale realistic condensation: soft-label storage. It formulates condensation as greedy selection under coding-rate objectives balancing informativeness, discriminativeness, and compressibility, and reports that even with $S$ 8 compression of soft labels, ImageNet-1K performance decreases by only $S$ 9 at IPC 10 and $T$ 0 at IPC 50 (Yuan et al., 18 Mar 2025).

4. Modalities, domains, and application regimes

Although much of the literature is vision-centric, DC has broadened substantially. For graph classification, "Condensing Graphs via One-Step Gradient Matching" replaces expensive multi-step matching with a one-step objective at initialization and models discrete graph structure using Bernoulli logits with a Binary Concrete relaxation (Jin et al., 2022). The method reports reducing dataset size by $T$ 1 while approximating up to $T$ 2 of original performance and being about $T$ 3 faster than multi-step gradient matching on CIFAR10 for synthesizing 500 graphs (Jin et al., 2022). The same paper extends one-step matching to node classification with substantial speedups (Jin et al., 2022).

In tabular learning, C $T$ 4TC reformulates condensation as a training-free class-adaptive cluster allocation problem. It replaces gradient-based inner loops with class-wise clustering in a feature space constructed by hybrid categorical feature encoding, optimized by heuristic first-improvement local search (Xu et al., 25 Feb 2026). The method is reported to improve efficiency by at least two orders of magnitude over state-of-the-art baselines across 10 real-world datasets, while achieving superior downstream performance (Xu et al., 25 Feb 2026). This suggests that not all effective condensation requires synthetic pixels or differentiable student models.

Clinical applications motivate both privacy-aware and model-agnostic variants. The mode-connectivity surrogate method targets clinical tabular and time-series tasks, emphasizing AUROC and AUPRC under severe class imbalance and reporting near-lossless performance at low condensation ratios on MIMIC-III (Nganjimi et al., 7 Oct 2025). A later clinical paper extends DC to non-differentiable models such as XGBoost and Cox regression using zero-order optimization and integrates formal differential privacy guarantees, reporting $T$ 5 values typically in approximately $T$ 6 with $T$ 7 while preserving strong utility across six datasets (Thakur et al., 10 Mar 2026).

The multi-domain setting introduces another axis of complexity. DAM defines Multi-Domain Dataset Condensation and augments standard DC/DM/MTT pipelines with a training-time Domain-Aware Module that uses frequency-based pseudo-domain labeling and learnable spatial masks (Choi et al., 28 May 2025). The module is only active during condensation and keeps IPC unchanged. Reported gains include CIFAR-10 IPC 10 improvements from $T$ 8 to $T$ 9 for DM and from $S=\{(s,y)\}$ 0 to $S=\{(s,y)\}$ 1 for MTT, together with consistent gains on PACS, VLCS, and Office-Home (Choi et al., 28 May 2025).

Finally, DC has been adapted to deep hashing retrieval. IEM uses feature-embedding matching in hashing space with early-stage model pools and multi-formation parameterization, improving mAP over DM, DSA, and IDC on CIFAR10 and ImageNet subsets while reducing condensation cost relative to gradient-heavy baselines (Feng et al., 2023).

5. Evaluation practice, empirical patterns, and benchmark findings

DC-BENCH provides the first large-scale standardized benchmark dedicated to dataset condensation and shows that evaluation is highly confounded by augmentation, architecture, initialization, and training budget (Cui et al., 2022). It evaluates methods under multiple augmentations, cross-architecture transfer, a wide IPC range up to 1,000 on CIFAR-10, stability across seeds, and NAS transfer (Cui et al., 2022). One of its main findings is that augmentation alone can drastically change method rankings, and strong selection baselines such as K-Center were previously underestimated (Cui et al., 2022).

Several empirical patterns recur across the literature. First, gradient-matching DC is particularly strong at small budgets. In the original gradient-matching paper, on MNIST with 50 images per class and a ConvNet, DC achieves $S=\{(s,y)\}$ 2 versus $S=\{(s,y)\}$ 3 for the whole dataset; on CIFAR-10 with 50 images per class it reaches $S=\{(s,y)\}$ 4 versus $S=\{(s,y)\}$ 5 for full data, outperforming Random, Herding, K-Center, and Forgetting (Zhao et al., 2020). Accuracy increases steadily with more images per class, but the gap to full-data training is larger on SVHN and CIFAR-10/100 than on MNIST and FashionMNIST, indicating higher domain complexity (Zhao et al., 2020).

Second, cross-architecture transfer is real but not uniform. The original DC paper reports that a synthetic set learned with a ConvNet transfers reasonably to LeNet, VGG, and ResNet-18 on MNIST, whereas a set learned with an MLP transfers poorly to ConvNets (Zhao et al., 2020). DC-BENCH similarly finds that rankings shift under architecture transfer and that larger models such as ResNet152 perform poorly at very small IPCs but improve with larger budgets (Cui et al., 2022). Some later methods improve transfer substantially; for example, the generative-model approach reports strong gains over pixel-based baselines on AlexNet, VGG11, ResNet18, and MLP using CIFAR-10 synthetic data learned with ConvNet (Zhang et al., 2023).

Third, distribution-matching methods have become increasingly competitive with optimization-oriented methods. IDM surpasses naive DM and most optimization-based baselines while using fewer resources (Zhao et al., 2023). Decomposed DM closes much of the performance gap while scaling to ImageNet-1K (Malakshan et al., 2024). SCORE targets ImageNet-scale settings and reports strong cross-architecture generalization under realistic-image-plus-soft-label pipelines (Yuan et al., 18 Mar 2025).

Fourth, certain application settings benefit disproportionately from condensation. In continual learning, the original gradient-matching method reports that replacing rehearsal sample selection with condensed images improves final-stage accuracy by $S=\{(s,y)\}$ 6 on task-incremental SVHN $S=\{(s,y)\}$ 7 MNIST $S=\{(s,y)\}$ 8 USPS without KD (Zhao et al., 2020). Decomposed DM reports a $S=\{(s,y)\}$ 9 gain in continual learning accuracy (Malakshan et al., 2024). In graph continual learning, LQM improves average accuracy while maintaining competitive backward transfer (Wei et al., 2024).

A concise summary of benchmark-level observations is useful:

Setting	Representative finding	Source
Small IPC	Condensation gains are strongest at very low IPC	(Cui et al., 2022)
Higher IPC	Gains over strong selection baselines shrink as IPC increases	(Cui et al., 2022)
Augmentation	Augmentation can drastically alter rankings	(Cui et al., 2022)
Cross-architecture	Transferability is limited and ranking-sensitive	(Cui et al., 2022)
NAS proxy	Condensed datasets in modern settings often fail to preserve architecture rankings	(Cui et al., 2022)

One apparent tension in the literature concerns NAS. The original gradient-matching paper reports promising gains, with a CIFAR-10 proxy of 100 images ranking 720 ConvNet architectures with Spearman correlation $S$ 0 for top-10 models and evaluating all architectures in $S$ 1 minutes versus $S$ 2 minutes on full data (Zhao et al., 2020). DC-BENCH, however, reports negative or near-zero correlations for several methods on NAS-Bench-201 and concludes that NAS transfer is currently unsuccessful for condensation datasets in modern settings (Cui et al., 2022). This suggests that NAS utility is highly protocol-dependent.

6. Privacy, security, and ongoing controversies

Privacy has become one of the most contested themes in DC. "Privacy for Free: How does Dataset Condensation Help Privacy?" studies DC through the lens of distribution matching and argues that when $S$ 3, removing one raw sample changes the parameter distribution of models trained on the condensed set by only $S$ 4 under its assumptions (Dong et al., 2022). The paper further presents empirical evidence of low membership-inference vulnerability under random initialization of synthetic data and reports that DC-synthesized data can offer stronger utility than several DP-generators at comparable empirical privacy levels on FashionMNIST (Dong et al., 2022). It also emphasizes a key caveat: real-data initialization can leak membership of initialization samples, whereas random initialization reduces visual and membership leakage (Dong et al., 2022).

This claim was challenged directly in "No Free Lunch in 'Privacy for Free: How does Dataset Condensation Help Privacy'" (Carlini et al., 2022). That critique argues that the theoretical analysis relies on assumptions tantamount to an already private mechanism and that the empirical evaluation uses a confounded baseline. Recomputing the random-subset baseline on the full universe, it reports attack advantage approximately $S$ 5 versus $S$ 6 for DM at roughly 1% condensation, concluding that there is no statistically significant privacy improvement over a naive baseline (Carlini et al., 2022). It further contrasts this with DP-SGD, which offers formal guarantees and in the cited settings can achieve both better accuracy and lower membership attack success (Carlini et al., 2022). Accordingly, a conservative encyclopedia treatment is that DC may attenuate per-sample influence and may be compatible with privacy-preserving workflows, but empirical robustness to membership inference is not a substitute for formal differential privacy (Dong et al., 2022, Carlini et al., 2022).

More recent work moves from privacy to security vulnerabilities. "InkDrop: Invisible Backdoor Attacks Against Dataset Condensation" studies backdoors in DM-family methods and shows that a malicious provider controlling the condensation pipeline can implant imperceptible instance-dependent perturbations before condensation, preserving clean task accuracy while driving attack success rate to near 1.0 across datasets and downstream architectures (Yang et al., 30 Mar 2026). Reported representative outcomes with a ConvNet synthesizer include CTA approximately $S$ 7 and ASR approximately $S$ 8 on CIFAR-10 DM, with similarly near-perfect ASR on STL-10, Tiny-ImageNet, and SVHN (Yang et al., 30 Mar 2026). Standard defenses such as PIXEL, ABS, RNP, and PDB either fail or incur severe CTA degradation (Yang et al., 30 Mar 2026). A plausible implication is that condensed datasets should be treated as high-risk artifacts whenever their provenance is uncertain.

Two practical conclusions emerge from these privacy and security studies. First, DC should not be presumed private by default; privacy depends on initialization, threat model, evaluation protocol, and whether formal noise mechanisms are used (Dong et al., 2022, Carlini et al., 2022). Second, condensed datasets can encode adversarial intent in ways that transfer across architectures, so provenance, auditing, and defense-in-depth are necessary before deployment (Yang et al., 30 Mar 2026).

7. Open directions and broader perspective

The field has gradually shifted from asking whether DC works at all toward asking what discrepancy should be minimized, how the synthetic artifact should be parameterized, and which objectives besides accuracy should be preserved. The discrepancy-based perspective makes this shift explicit by treating DC as reduced-support distribution approximation under a chosen discrepancy, with optional regularizers for privacy, robustness, diversity, or other downstream desiderata (Chen et al., 12 Sep 2025). Under this view, gradient matching, feature matching, MMD, Wasserstein, parameter matching, and kernel methods are not isolated ideas but specializations of a single framework (Chen et al., 12 Sep 2025).

Several open problems recur across the literature. Scaling to high-resolution, large-scale datasets remains difficult for pixel-parameterized methods, though generative formulations, realistic-image selection pipelines with soft labels, and decomposed DM have advanced the frontier (Zhao et al., 2020, Zhang et al., 2023, Malakshan et al., 2024, Yuan et al., 18 Mar 2025). Architecture transfer remains imperfect, especially across qualitatively different model families such as ConvNets, ViTs, and non-neural classical models (Cui et al., 2022, Thakur et al., 10 Mar 2026). Robust multi-objective optimization—simultaneously preserving accuracy, robustness, privacy, and security—remains underexplored (Chen et al., 12 Sep 2025).

Another major open direction is modality expansion. Graph, tabular, and clinical works indicate that DC is not inherently tied to pixel synthesis or differentiable image classifiers (Jin et al., 2022, Xu et al., 25 Feb 2026, Thakur et al., 10 Mar 2026). Inference from these works suggests that future progress may depend less on a universal condensation algorithm than on modality-aware abstractions: probabilistic discrete structures for graphs, clustering-based surrogates for tabular data, zero-order optimization for non-differentiable clinical models, and structured latent or hierarchical memories for large-scale vision data (Jin et al., 2022, Xu et al., 25 Feb 2026, Zheng et al., 2023).

At the same time, standardization remains essential. DC-BENCH shows that without controlled protocols, claims about method superiority can be dominated by augmentation policy, architecture choice, training schedule, or initialization (Cui et al., 2022). This suggests that the maturity of DC as a research area depends not only on better algorithms, but also on stable evaluation methodology, explicit threat modeling, and artifact auditing.