Latent Quantile Matching in Dataset Condensation

• LQM is a framework that aligns the full spectrum of latent feature distributions between synthetic and real datasets using a statistically principled quantile-based loss.
• It controls all distributional moments—including tails and outliers—thereby offering robustness over traditional mean-based MMD approaches in low-budget and privacy-sensitive settings.
• Empirical studies and theoretical foundations show that LQM consistently outperforms conventional methods on image and graph data, enhancing dataset condensation and continual learning tasks.

Latent Quantile Matching (LQM) is a distribution-matching framework designed to align latent feature distributions of synthetic and real datasets, primarily for dataset condensation (DC). Unlike Maximum Mean Discrepancy (MMD) approaches, which only align means in the latent space, LQM matches the entire quantile spectrum via a statistically principled quantile-based loss. This approach ensures control over all distributional moments, minimizes outlier risk, and is particularly effective for low-budget, privacy-sensitive, and continual learning scenarios. Empirical studies confirm that LQM consistently matches or outperforms MMD-based methods on both image and graph-structured data (Wei et al., 2024). LQM is also grounded in the theoretical framework of $q$-quantile matching experiments, which uniquely characterize the set of all implementable distributions of posterior quantiles (Kolotilin et al., 2024).

1. Problem Setting and Motivation

Dataset condensation aims to construct a compact synthetic dataset $S$, with $|S|\ll|T|$ for a real dataset $T$, such that training a model $\varphi$ on $S$ yields comparable test accuracy to training on $T$. Formally, with $\varphi^T$ and $\varphi^S$ as models trained on $T$ and $S$0, the goal is: $S$1 where $S$2 is typically cross-entropy.

Distribution-matching (DM) based DC sidesteps expensive bi-level optimization by aligning the latent distributions induced by a family of randomly initialized networks $S$3. For a particular $S$4, embeddings are extracted as: $S$5 The synthetic set $S$6 is then optimized to minimize a distance metric $S$7. LQM proposes a quantile-based loss for $S$8, strengthening the expressiveness and robustness of the match versus mean-based criteria.

2. Mathematical Formulation of Latent Quantile Matching

Each data point $S$9 is mapped to a latent embedding $|S|\ll|T|$0. For latent coordinate $|S|\ll|T|$1, define the empirical CDFs over $|S|\ll|T|$2 and $|S|\ll|T|$3: $|S|\ll|T|$4 The corresponding quantile functions are

$|S|\ll|T|$5

The canonical two-sample distributional distance is the Cramér–von Mises (CvM) statistic: $|S|\ll|T|$6 where $|S|\ll|T|$7. Equivalently (by a variable change) the key LQM loss is: $|S|\ll|T|$8 With a budget of $|S|\ll|T|$9 synthetic points per class, the integral is approximated using optimal quantile locations $T$0, minimizing the CvM statistic error for discretization. The discrete LQM loss per feature is: $T$1 Extending to $T$2 classes and $T$3 features, the aggregated LQM loss is: $T$4 This formulation ensures all moments, including tails and outliers, are aligned.

3. Comparison with Maximum Mean Discrepancy-Based Approaches

Maximum Mean Discrepancy (MMD) measures divergence by comparing first moments (means) of mapped features: $T$5 Typically, $T$6 is the identity, limiting discrimination to mean-matching. This presents two main deficiencies:

• Insensitivity to higher-order discrepancies: Distributions with identical means may have arbitrary variance, skewness, kurtosis, or outliers.
• Outlier exposure: Synthetic samples may drift to extreme latent values to match the mean, thus generating harmful support violations.

LQM addresses these by aligning the full quantile sequence, enforcing tighter support and penalizing extremal synthetic points, thereby controlling all distributional moments and mitigating outlier risks (Wei et al., 2024).

4. Algorithmic Implementation and Optimization

Optimization proceeds by stochastic gradient descent on the synthetic samples themselves. A high-level procedure:

• For each class $T$7 and feature $T$8, precompute sorted real embeddings.
• At each epoch:
  • Draw $T$9 afresh.
  • For each class (parallelizable):
  • 1. Extract $\varphi$0, $\varphi$1 via the current feature extractor.
  • 2. For each latent $\varphi$2:
  • Sort $\varphi$3.
  • Compute target quantiles at $\varphi$4 from $\varphi$5.
  • Compute loss as average squared deviations between sorted $\varphi$6 and target quantiles.
  • 3. Backpropagate the per-class average loss to update $\varphi$7.

During backpropagation, the permutation from sorting is treated as fixed, similar to quantile regression backpropagation routines.

Step	Operation	Purpose
Feature extraction	$\varphi$8	Project data to latent space
Sorting	Ascending for each per-class, per-feature	Define empirical quantile structure
Quantile alignment	Match $\varphi$9 to $S$0	Enforce quantile-by-quantile correspondence
Gradient update	Optimize $S$1 via $S$2	Refine synthetic samples

5. Experimental Validation

On image classification (e.g., CIFAR-10/100, TinyImageNet), LQM using an IDM backbone surpasses MMD-based IDM by 2–3% test accuracy on CIFAR-10/100 at low data budgets (1, 10, 50 images/class). For TinyImageNet, these gains persist, especially under stricter (lower) budgets. Empirical CvM distances between synthetic and real latent distributions also decrease significantly under LQM, indicating stronger latent alignment.

On graph classification (e.g., CoraFull, Arxiv, Reddit, Products) with a CaT backbone, LQM outperforms baseline CaT by approximately 3% average accuracy on CoraFull and Arxiv in the one-task setting, and yields +2.0% (CoraFull) and +1.4% (Arxiv) in class-incremental continual graph learning (CGL). Backward transfer is maintained or slightly improved. The improvement diminishes for larger data budgets but LQM always matches or exceeds the previous state of the art (Wei et al., 2024).

6. Hyperparameters, Computational Complexity, and Privacy

LQM inherits learning rate, epoch count, and batch sizes from standard DC backbones (IDM, CaT). The sole replacement is the distance metric (LQM for MMD). Per step, computational cost increases from $S$3 (MMD) to $S$4 (due to sorting and quantile computations over $S$5 features), yet remains practical as $S$6 is typically very small ($S$7).

LQM optimizes for privacy and memory-efficiency:

• Privacy: Synthetic samples do not reveal real data and exhibit robustness to membership inference attacks.
• Memory: LQM’s quantile-wise control yields superior performance especially under stringent memory (small $S$8), making it suitable for continual learning and federated replay settings.

7. Theoretical Foundations: Quantile Matching Experiments

LQM is underpinned by recent theoretical results in quantile matching experiments. For any quantile $S$9, Kolotilin and Wolitzky (Kolotilin et al., 2024) show that all implementable distributions over posterior $T$0-quantiles are realized by a single $T$1-quantile matching experiment. This experiment—pooling data states across the $T$2-quantile with positive assortativity and precise allocation of $T$3 and $T$4 weights—yields a universal construction $T$5 encoding all feasible target distributions for the posterior quantile. The main implications include:

• Implementability: Every distribution within sandwich bounds set by Markov inequalities is attainable via $T$6.
• Uniqueness: $T$7 is the only experiment implementing all $T$8-quantile distributions.
• Optimization: For any continuous functional, pointwise maximization over quantile-intervals from $T$9 yields the optimal induced distribution.

This theoretical foundation validates LQM’s use of quantile alignment, not just as a practical algorithm, but as a statistically and decision-theoretically principled solution for latent distribution matching in DC.

---

Latent Quantile Matching achieves high-fidelity distribution matching via statistical quantile alignment, robustly controls all aspects of latent distributional shape, and meets the rigorous demands of modern dataset condensation, continual, and federated learning tasks (Wei et al., 2024, Kolotilin et al., 2024).