Soft Label Pruning and Quantization for Large-Scale Dataset Distillation

Abstract: Large-scale dataset distillation requires storing auxiliary soft labels that can be 30-40x larger on ImageNet-1K and 200x larger on ImageNet-21K than the condensed images, undermining the goal of dataset compression. We identify two fundamental issues necessitating such extensive labels: (1) insufficient image diversity, where high within-class similarity in synthetic images requires extensive augmentation, and (2) insufficient supervision diversity, where limited variety in supervisory signals during training leads to performance degradation at high compression rates. To address these challenges, we propose Label Pruning and Quantization for Large-scale Distillation (LPQLD). We enhance image diversity via class-wise batching and batch-normalization supervision during synthesis. For supervision diversity, we introduce Label Pruning with Dynamic Knowledge Reuse to improve label-per-augmentation diversity, and Label Quantization with Calibrated Student-Teacher Alignment to improve augmentation-per-image diversity. Our approach reduces soft label storage by 78x on ImageNet-1K and 500x on ImageNet-21K while improving accuracy by up to 7.2% and 2.8%, respectively. Extensive experiments validate the superiority of LPQLD across different network architectures and dataset distillation methods. Code is available at https://github.com/he-y/soft-label-pruning-quantization-for-dataset-distillation.

• The paper introduces LPQLD, a framework that reduces extensive soft label storage via dynamic pruning and quantization.
• It employs class-wise BatchNorm and dynamic teacher-student alignment to enhance intra-class diversity and effective label compression.
• Empirical results demonstrate up to 500x storage reduction on ImageNet datasets with maintained or improved accuracy.

Soft Label Pruning and Quantization for Large-Scale Dataset Distillation: An Analytical Essay

Motivation and Problem Landscape

Large-scale dataset distillation promises to compress vast datasets like ImageNet-1K and ImageNet-21K into compact synthetic surrogates that are storage- and compute-efficient. However, contemporary methods (notably SRe$^2$L and its variants) are hampered by excessive auxiliary soft label storage requirements, often exceeding image storage by orders of magnitude. For ImageNet-1K, soft labels can be up to 40$\times$ larger than the condensed images; for ImageNet-21K, this ratio escalates to over 200$\times$, undermining the fundamental goal of dataset condensation and limiting practical deployment.

The authors dissect this bottleneck into two intertwined root causes:

• Insufficient intra-class image diversity: Synthetic datasets exhibit high within-class similarity, necessitating extensive augmentation and thus ballooning the label storage requirements.
• Limited supervision diversity: Constrained variety in supervisory signals (i.e., soft labels per augmentation per image) leads to poor generalization, especially under high compression regimes.

These discoveries redefine the dataset distillation landscape, suggesting that approaches which do not address both axes of diversity (image and supervision) inevitably pay in performance or efficiency.

Figure 1: LPQLD establishes a new Pareto front for accuracy/storage trade-off in ImageNet-1K distillation, dominating previous baselines by orders of magnitude in compression.

Methodological Advancements: The LPQLD Framework

The paper introduces the Label Pruning and Quantization for Large-scale Distillation (LPQLD) framework, systematically engineered to overcome both image and supervision diversity bottlenecks. LPQLD comprises three synergistic stages:

1. Diverse Synthetic Dataset Generation: The recovery phase is redesigned by batching images within each class and supervising synthesis with class-wise BatchNorm statistics. This facilitates intra-class collaboration and drives lower feature cosine similarity within classes, leading to higher image diversity.

   Figure 2: LPQLD's diverse synthetic dataset generation encourages intra-class collaboration, yielding natural separation among classes.

2. Soft Label Compression: Two orthogonal modules tackle label storage:

• Label Pruning with Dynamic Knowledge Reuse (DKR): Epoch-wise label pruning reduces the set of augmentation-label pairs, but the retained soft labels are reused across multiple training epochs with temperature annealing, maximizing label-per-augmentation diversity without incurring further storage overhead.
• Label Quantization with Calibrated Student-Teacher Alignment (CA): Only top-$k$ logits per label are stored. To address the distributional shift induced by quantization, student temperature is dynamically recalibrated via KL-minimization, ensuring alignment with the sharpened teacher output.

  Figure 3: LPQLD pipeline overview: synthetic data generation (class-wise BN), label pruning/quantization, and compression-aware training with DKR and CA.

1. Compression-Aware Distillation: The distilled datasets and compressed soft labels are used for student model training, leveraging temperature scheduling and grid search for optimal student-teacher alignment.

   Figure 4: MMD comparison showing improved alignment between the feature distribution of synthetic and real datasets in LPQLD.

Empirical Results and Numerical Evaluation

LPQLD is evaluated on ImageNet-1K and ImageNet-21K-P. The method achieves unprecedented compression ratios:

• ImageNet-1K: 78$\times$ label storage reduction, maintaining or exceeding baseline performance. At IPC50, storage drops from 29.7G to 0.38G with no strong accuracy penalty; quantization further boosts accuracy at high compression rates (e.g., +7.2% at 79$\times$ compression).
• ImageNet-21K: 500$\times$ compression, surpassing SRe$^2$L's performance using only a fraction of its label storage.

LPQLD delivers robust performance across student architectures (ResNet-50, EfficientNet-B0, MobileNet-V2, Swin-V2-Tiny) and is compatible with diverse distilled datasets (DWA, DELT, Minimax Diffusion), underscoring its generality. Ablation confirms that the synergy of class-wise BN, batch-level pruning, DKR, quantization, and CA is essential; naïve combinations of pruning and quantization collapse accuracy due to signal sparsity and distribution mismatch.

Figure 5: IPC10 visualization—LPQLD's synthetic images exhibit higher intra-class diversity.

Figure 6: IPC10 visualization—LPQLD yields diverse samples within each class even at low IPC.

Theoretical Contributions and Practical Implications

LPQLD formalizes supervision diversity as the fundamental determinant of performance under extreme compression. Its decomposition into augmentation-per-image and label-per-augmentation diversity yields practical guidance for balancing pruning and quantization (see Pareto front analyses). DKR's use of temperature annealing is theoretically justified via VC-dimension and convergence bounds, extending annealed KD theory to label-pruned regimes. The CA module solves optimal alignment under quantized teacher distributions via temperature grid search, supported by KL divergence landscapes.

Practically, LPQLD enables deployment of large-scale distillation in resource-constrained settings without compromising accuracy, paving the way for model compression, federated learning, and privacy-preserving data minimization at scale.

Future Directions and Limitations

While LPQLD shatters the storage bottlenecks of prior art, training time is not reduced and grid search for CA incurs minimal—but nonzero—computational overhead. Future research should focus on joint optimization of time and storage, adaptive temperature scheduling tailored to architectures, and the extension of supervision diversity principles to data pruning and instance selection regimes.

Conclusion

LPQLD redefines large-scale dataset distillation, combining class-wise BN-driven synthesis, dynamic label pruning, quantization, and calibrated student-teacher alignment into an integrated pipeline that minimizes soft label storage without sacrificing performance. The supervision diversity framework introduced in this work constitutes a principled foundation for future advances in efficient knowledge distillation and dataset condensation (2604.18135).