Difficulty-Aligned Trajectory Matching (DATM)

• DATM is a dataset distillation paradigm that creates compact, high-fidelity synthetic datasets by aligning training dynamics with real-data trajectories.
• It uses a dynamic curriculum approach, gradually expanding the matching window from easy to hard trajectory segments to optimize synthetic data quality.
• Empirical results show that DATM nearly matches full-data performance on benchmarks like CIFAR-10/100, demonstrating effective lossless distillation with scalable synthetic sets.

Difficulty-Aligned Trajectory Matching (DATM) is a dataset distillation paradigm designed to synthesize small, high-fidelity datasets that allow neural networks to achieve performance nearly equivalent to training on the entire real dataset. DATM innovates over prior trajectory matching approaches by dynamically aligning the complexity ("difficulty") of learned patterns with the representational capacity of the synthetic data. This alignment is realized by matching synthetic training dynamics to real-data learning trajectories at carefully chosen stages, enabling DATM to scale from very small to large image-per-class (IPC) regimes, ultimately achieving near-lossless distillation as the synthetic set grows in size (Guo et al., 2023).

1. Core Principles of Difficulty-Aligned Trajectory Matching

DATM operates within the trajectory-matching (TM) framework, where the optimization goal is to construct a synthetic dataset $\Dsyn$ such that an identical network, trained solely on $\Dsyn$, will follow a sequence of parameter states closely matching that observed when training with the real dataset $\Dreal$. Formally, for a given teacher trajectory $\tau^*=\{\theta^*_t\}_{t=0}^n$ trained on $\Dreal$, and a student trajectory $\{\hat\theta_{t+i}\}_{i=0}^N$ produced by training on $\Dsyn$:

• The matching loss is defined as

$\cL_{\rm TM}(\Dsyn) = \frac{\|\hat\theta_{t+N} - \theta^*_{t+M}\|_2^2}{\|\theta^*_{t} - \theta^*_{t+M}\|_2^2}$

with $N$ steps of student unroll and $M$ as the matching interval (Guo et al., 2023).

• DATM introduces the principle of difficulty alignment: Rather than statically match synthetic training dynamics to a fixed segment of the real-data trajectory, DATM samples the trajectory stage $\Dsyn$0 for matching according to the synthetic set's size and capacity.
  • Early trajectory segments (small $\Dsyn$1) correspond to acquisition of "easy" or high-support features.
  • Late trajectory segments (large $\Dsyn$2) encode "hard," fine-grained, low-support patterns.

Empirical evidence shows that small synthetic sets excel when matching easy stages, while larger synthetic sets can represent complex, harder patterns. DATM adaptively expands the trajectory-matching window $\Dsyn$3 during distillation, enabling a gradual curriculum from easy to hard (Guo et al., 2023, Zhou et al., 2024).

2. Methodology and Optimization Pipeline

The standard DATM pipeline includes the following steps (Guo et al., 2023, Zhou et al., 2024):

• Expert Trajectory Generation: Train a network $\Dsyn$4 on $\Dsyn$5, storing parameter checkpoints $\Dsyn$6 across epochs.
• Initialization: Form $\Dsyn$7 by sampling from a teacher-correct real-data subset, assigning each synthetic image $\Dsyn$8 both pixels and a soft label parameter $\Dsyn$9 (learned outer variable).
• Iterative Bi-level Optimization:
  • Sample a trajectory segment index $\Dreal$0, defining $\Dreal$1 (start) and $\Dreal$2 (target).
  • Initialize student weights $\Dreal$3.
  • Train on $\Dreal$4 for $\Dreal$5 steps with learning rate $\Dreal$6, using a soft or hard cross-entropy loss.
  • Compute the normalized trajectory-matching loss as above.
  • Backpropagate the loss through both inner (SGD) and outer loops, updating $\Dreal$7, label parameters $\Dreal$8, and learning rates.
  • At periodic intervals, increment $\Dreal$9 to gradually include later (harder) portions of the trajectory.

DATM’s use of both inner (student) and outer (meta) optimization loops and the progressive widening of the matching window constitute its key methodological distinctions. This allows the synthetic set to absorb increasingly difficult patterns as its capacity grows (Guo et al., 2023).

3. Empirical Validation and Results

DATM demonstrates substantial advances in dataset distillation efficacy, particularly in the medium-to-high IPC regime. Tabulated results on canonical benchmarks, using a 3-layer ConvNet unless stated otherwise (Guo et al., 2023, Zhou et al., 2024):

Dataset	IPC	DATM Accuracy (%)	MTT (%)	Random (%)	Lossless*
CIFAR-10	1	46.9	46.2	15.4
CIFAR-10	10	66.8	65.4	31.0
CIFAR-10	50	76.1	71.6	50.6
CIFAR-10	500	83.5	—	73.2	✓
CIFAR-10	1000	85.5	—	78.4	✓
CIFAR-100	1	27.9	24.3	—
CIFAR-100	10	47.2	39.7	—
CIFAR-100	50	55.0	47.7	—
CIFAR-100	100	57.5	—	—	✓
TinyImageNet	1	17.1	—	—
TinyImageNet	10	31.1	—	—
TinyImageNet	50	39.7	—	—	✓

(*"Lossless": DATM synthetic set matches or marginally exceeds full-data accuracy.)

DATM achieves lossless dataset distillation for the default ConvNet on CIFAR-10/100 and TinyImageNet at sufficiently large IPC. Empirical ablations confirm that dynamic, curriculum-style alignment of matching window is crucial; hard-only or easy-only matching underperform the full sequential approach (Guo et al., 2023).

4. Difficulty as a Latent Curriculum and Theoretical Motivation

Difficulty alignment in DATM leverages the empirical observation that deep networks learn easy patterns first—high-support, low-frequency features—then progressively harder, rare, and fine-grained features (Guo et al., 2023). The trajectory stage $\tau^*=\{\theta^*_t\}_{t=0}^n$0 acts as a proxy for difficulty. A small synthetic set covers only the easy modes, so early trajectory matching is appropriate. As set size increases, later (hard) trajectory segments can be included without loss of generality.

Sequential expansion of the matching window ensures the distilled curriculum evolves in lockstep with the synthetic set’s expressivity, bridging between curriculum learning (easy-to-hard at the data level) and parameter-space trajectory matching.

No full convergence guarantee or tight generalization theory is yet available for this approach, though its empirical efficacy is well-established (Guo et al., 2023).

5. Variants and Practical Modifications

The modified DATM (M-DATM) emerged in the context of challenges where label-ordering mismatches could penalize soft-label learning. M-DATM introduces two modifications (Zhou et al., 2024):

• Removal of Soft Labels: Hard class labels are used in the synthetic set to eliminate one-to-one correspondence errors with evaluation scripts, replacing soft cross-entropy with standard cross-entropy for optimization.
• Restriction of Matching Window: In scenarios with hard labels only, matching is confined to early trajectory segments (e.g., $\tau^*=\{\theta^*_t\}_{t=0}^n$1), focusing the synthetic set on learning only easy patterns.

On CIFAR-100 and TinyImageNet (IPC=10), M-DATM achieved 40.61% and 18.31% accuracy, respectively—ranking first in ECCV-2024's Fixed IPC track. Ablations show that removing soft labels alone improves accuracy substantially, but restricting to early trajectories is essential for hard-label-only distillation to avoid collapse on more difficult datasets (Zhou et al., 2024).

6. Limitations and Directions for Advancement

DATM attains lossless distillation only for the backbone model used during set generation. There remains a demonstrable gap when synthetic sets are evaluated on unseen architectures—a target for ongoing research (Guo et al., 2023). Storage and computation costs for maintaining long parameter trajectories are an obstacle to extending DATM to very large datasets such as ImageNet-1K. Integration with memory-efficient trajectory-matching variants and further theoretical study of curriculum-style bi-level objectives are promising directions.

The principle of difficulty alignment, as instantiated by DATM, is foundational for scaling trajectory-matched dataset distillation to larger synthetic datasets without performance degradation, and serves as the basis for recent variants such as M-DATM (Zhou et al., 2024) and approaches leveraging selection-based initialization and partial updates (Lee et al., 2024).