Dataset Distillation for Pre-Trained Self-Supervised Vision Models (2511.16674v1)

Abstract: The task of dataset distillation aims to find a small set of synthetic images such that training a model on them reproduces the performance of the same model trained on a much larger dataset of real samples. Existing distillation methods focus on synthesizing datasets that enable training randomly initialized models. In contrast, state-of-the-art vision approaches are increasingly building on large, pre-trained self-supervised models rather than training from scratch. In this paper, we investigate the problem of distilling datasets that enable us to optimally train linear probes on top of such large, pre-trained vision models. We introduce a method of dataset distillation for this task called Linear Gradient Matching that optimizes the synthetic images such that, when passed through a pre-trained feature extractor, they induce gradients in the linear classifier similar to those produced by the real data. Our method yields synthetic data that outperform all real-image baselines and, remarkably, generalize across pre-trained vision models, enabling us, for instance, to train a linear CLIP probe that performs competitively using a dataset distilled via a DINO backbone. Further, we show that our distilled datasets are exceptionally effective for fine-grained classification and provide a valuable tool for model interpretability, predicting, among other things, how similar two models' embedding spaces are under the platonic representation hypothesis or whether a model is sensitive to spurious correlations in adversarial datasets.

• The paper introduces a novel Linear Gradient Matching approach that synthesizes minimal datasets for linear probing on pre-trained self-supervised vision models.
• It employs multi-scale pyramid regularization and differentiable augmentation to optimize synthetic images that replicate real data gradients.
• Experimental results demonstrate near full-data accuracy and reveal backbone-specific biases, enhancing interpretability and data efficiency.

Dataset Distillation for Pre-Trained Self-Supervised Vision Models

Introduction and Motivation

The proliferation of large-scale pre-trained self-supervised vision models has fundamentally altered how representation learning and downstream task adaptation are performed. Conventionally, dataset distillation strategies have targeted synthesizing a small set of images that allow effective model learning purely from these synthetic instances, typically for training from scratch. However, state-of-the-art workflows increasingly employ linear probing atop fixed, pre-trained feature extractors. "Dataset Distillation for Pre-Trained Self-Supervised Vision Models" (2511.16674) addresses this specific setting: constructing minimal synthetic datasets tuned for maximizing linear probe efficacy on established vision foundation models.

This work proposes a new distillation paradigm where the goal is to find synthetic images that, when passed through a pre-trained feature extractor and used for linear probe training, induce parameter updates – measured by loss gradients – that closely resemble those from the original dataset. The resulting method, termed Linear Gradient Matching, enables the condensation of datasets such as ImageNet-1k to a single synthetic image per class without substantial loss in downstream linear evaluation accuracy.

Figure 1: ImageNet-1k synthetic samples distilled from CLIP, DINO-v2, EVA-02, and MoCo-v3, demonstrating distinct styles correlating with backbone inductive priors and representational biases.

Linear Gradient Matching: Methodology

Distillation Objective

Given a fixed, pre-trained self-supervised backbone $\phi$, and a real labeled dataset $D_\text{real}$, the aim is to synthesize $D_\text{syn}$ such that linear classifiers trained atop $\phi$ yield performance close to full dataset training. At each optimization iteration, a randomly-initialized linear classifier $W$ is sampled, and real as well as synthetic images (post feature-extraction by $\phi$) are fed through $W$ to determine cross-entropy losses. The meta-objective is to minimize the cosine distance between $\frac{\partial \ell_\text{real}}{\partial W}$ and $\frac{\partial \ell_\text{syn}}{\partial W}$, thus directly matching the influence of real versus synthetic data on linear probe optimization.

Figure 2: Diagram of Linear Gradient Matching, depicting how synthetic data is optimized to match real data gradients on top of a pre-trained backbone and randomized classifier.

Implicit Regularization

To address overfitting and adversarial artifact formation, the authors employ a multi-scale pyramid parameterization for synthetic images, inspired by recent work on image inversion. Each image is represented as a sum of upsampled components at varying resolutions, regularized by progressive optimization starting at low resolutions and gradually introducing higher frequency details. Additionally, synthetic samples are optimized in a decorrelated color space, impeding backbone-specific color bias exploitation.

Differentiable Data Augmentation

Performance and generalizability are further improved by extensively augmenting synthetic images during distillation. Multiple rounds of differentiable transformations – flipping, random resized cropping, Gaussian noising – expose the optimization loop to a wider selection of perturbations, promoting invariance and robustness in the learned synthetic representations.

Figure 3: Increasing differentiable augmentation rounds directly improves both single-backbone and cross-backbone performance; PCA visualization reveals that distilled images occupy extremal embedding space positions, often outside real class clusters.

Experimental Results

Superior Linear Probe Performance

On ImageNet-100 and ImageNet-1k, a single distilled image per class achieves linear probe accuracy within a narrow margin (e.g., DINO-v2: 75% on ImageNet-1k, with only seven accuracy points lost relative to full-data probes), and notably outperforms real-image selection baselines, including class centroids, nearest neighbors, and random samples.

Cross-Backbone Generalization

Synthetic datasets distilled from one backbone (e.g., DINO-v2) substantially generalize to others (e.g., CLIP, EVA-02). However, alignment between backbones deeply influences this transfer; for pairs with low representational congruence (e.g., CLIP–MoCo-v3), cross-backbone probe performance degrades.

Critical Role of Regularization and Augmentation

Ablation studies demonstrate the absolute necessity of differentiable augmentation and pyramid representation. Removing either sharply degrades linear probe efficacy, especially in cross-backbone tests, as measured quantitatively and observed visually.

Figure 4: Visual ablation results: the presence of regularization and augmentation yields coherent, semantically meaningful synthetic images; omission causes severe artifacting or loss of structural fidelity.

Spurious Correlations and Model Interpretability

Distilling datasets with known spurious correlations (e.g., "Spawrious" dogs-backgrounds) yields highly interpretable diagnostic outputs. For instance, synthetic images distilled with DINO-v2 display recognizable dog breeds (foreground focus), while those distilled with MoCo-v3 are dominated by background textures, elucidating the latter's failure to generalize under correlation shift scenarios.

(Figure 5)

Figure 5: Distillation on the Spawrious dataset reveals model-specific susceptibility to spurious background; MoCo-v3-determined synthetic images focus exclusively on backgrounds.

Fine-Grained Recognition

On fine-grained datasets (Stanford Dogs, CUB-200), distilled images encode category-discriminative details often exceeding those present in any single real image. The performance gap between distillation and real-image baselines is accentuated in these settings, underscoring the benefit of synthetic image optimization for fine-grained discrimination.

Figure 6: Stanford Dogs classes distilled with DINO-v2; synthetic images encode high-frequency, class-discriminative cues essential for fine-grained identification.

Predicting Model Alignment and Out-of-Distribution Generalization

The degree of cross-model success of distilled datasets aligns strongly with nearest neighbor overlap metrics between backbone feature spaces, supporting the Platonic Representation Hypothesis. Furthermore, foundation models such as DINO-v1, trained exclusively on natural images, can distill entirely out-of-distribution datasets (e.g., ArtBench art styles), and the synthetic images are not merely trivial reconstructions of seen training samples.

Theoretical and Practical Implications

This work establishes that high-performance, ultra-compact synthetic datasets can be generated purely in the context of pre-trained model probing, challenging prior beliefs that distillation is inherently limited in the one-shot-per-class regime for large models. The findings have significant implications for:

• Data-efficient evaluation and privacy: Disclosed distilled sets could replace large datasets for benchmarking and privacy-preserving model sharing.
• Interpretable AI: Synthetic images serve as explicit visual proxies for what features backbone models encode or neglect, opening new avenues for interpretability and bias analysis.
• Foundation model analysis: The Cross-model generalization results offer empirical support for strong latent space homogeneity across independently-trained foundation models.

Future Directions

Potential avenues for subsequent work include extending this methodology to other forms of downstream adaptation (e.g., few-shot learning, non-linear probes), scaling to multimodal representation spaces, or integrating structure-aware augmentations to further improve robustness. Formalizing the theoretical limits of one-image-per-class distillation on arbitrary representation geometries remains an open and critical problem.

Conclusion

Linear Gradient Matching establishes a rigorous, high-performance procedure for distilling vision datasets tailored for linear probing atop pre-trained self-supervised backbones. It unlocks unprecedented data compression rates without significant accuracy loss, delivers clear mechanistic insights into model inductive biases, and helps diagnose representational weaknesses, especially under adversarial or distribution-shifted conditions. This paradigm lays groundwork for broader adoption of distilled datasets in foundation model analysis, model sharing, and the paper of emergent properties in deep representation spaces.