Automatic Training Trajectories (ATT)

• Automatic Training Trajectories (ATT) are adaptive methods that dynamically select training update points to minimize cumulative errors and improve convergence in deep learning and numerical applications.
• ATT techniques address the Accumulated Mismatching Problem by adaptively choosing the optimal iteration based on minimum error metrics, thereby enhancing generalization across diverse architectures.
• In practice, ATT methods boost sample efficiency and reduce residual errors in dataset distillation and PDE solvers, often achieving these gains with negligible additional computational cost.

Automatic Training Trajectories (ATT) refer to a family of adaptive techniques for optimizing the sequence of training updates (“trajectories”) in deep learning, sampling-based numerical methods, and dataset distillation. ATT methods dynamically select training points, steps, or matchings so as to minimize accumulating error and enhance generalization, stability, and sample efficiency across domains such as dataset distillation, PDE solvers, and high-dimensional function approximation. ATT approaches are characterized by their core mechanism: adaptively determining when, where, or how to sample the next training point or match the evolution of weights by minimizing a relevant error metric at every iteration, replacing rigid fixed policies with data-driven, minimum-error matchings (Liu et al., 2024, Chen et al., 2023).

1. Motivation: The Accumulated Mismatching Problem and Adaptive Design

Traditional long-range trajectory-matching in dataset distillation and numerical optimization relies on matching student models to expert trajectories at a fixed number of iterations (Fixed Trajectory Length, FTL). In this setting, the student’s state after $N_S$ synthetic steps is forced to align with the target (teacher) model’s state after $N_T$ real-data steps. However, stochasticity and inherent variability mean that different trajectories naturally arrive at disparate intermediate states at step $N_S$. Thus, FTL approaches distort learning paths, leading to the Accumulated Mismatching Problem (AMP), where semantic drift and trajectory stretching/compression degrade generalization, especially for models or architectures not seen during distillation (Liu et al., 2024).

ATT techniques were introduced to fundamentally address this accumulated mismatch by allowing the matching point or sampled collocation to be adaptively selected—minimizing trajectory error at each step or focusing training on "hard" regions in input/state space (Liu et al., 2024, Chen et al., 2023).

2. Mathematical Formulation

A. Dataset Distillation by ATT

The ATT paradigm in dataset distillation replaces the rigid long-range matching cost with a dynamically optimized alignment:

• Define the trajectory operator for model $f$ and dataset $D$,

$T_{D,f}(\theta_0,N) = \theta_N - \theta_0$

where $\theta_t$ is the model parameter after $t$ steps of SGD.

• FTL-based distillation solves:

$$\min_{D_S} \mathbb{E}_{\theta_0 \sim P_{\theta_0}} \| T_{D_S,f}(\theta_0, N_S) - T_{D_T,f}(\theta_0,N_T) \|_2^2$$

• ATT allows $N_S$ to vary, choosing for each iteration the $N_T$0 (with $N_T$1) minimizing

$N_T$2

and sets

$N_T$3

B. PDE Solvers with Adaptive Trajectories Sampling

In adaptive deep PDE solvers, ATT (also denoted Adaptive Trajectories Sampling, ATS) operates by:

• Maintaining a set of "anchor points" $N_T$4 at epoch $N_T$5.
• For each anchor, generating candidate points by simulating short SDE trajectories,

$N_T$6

• Selecting, for each anchor, the candidate with the largest error indicator (e.g., residual, temporal-difference, Bellman error).
• Updating the anchor set and focusing the loss on new anchors, thereby adaptively concentrating the collocation budget on the most challenging regions (Chen et al., 2023).

3. ATT Algorithms and Pseudocode

A. Dataset Distillation ATT

The core ATT algorithm proceeds by:

1. Initializing the synthetic dataset $N_T$7.
2. Repeating:
   • Sampling expert parameter pairs $N_T$8.
   • Starting from student state $N_T$9.
   • Unrolling student updates for $N_S$0,
     • Updating the state and recording matching errors $N_S$1.
   • Selecting $N_S$2.
   • Back-propagating matching loss at $N_S$3.

This dynamic match-point approach suppresses the accumulation of error characteristic of fixed-step matching (Liu et al., 2024).

B. PDE Solver ATT

The adaptive loop in deep PDE ATT is as follows:

1. At each epoch, from each anchor, generate $N_S$4 SDE-propagated candidate points.
2. Evaluate error indicators at all candidates.
3. Select, for each anchor, the candidate maximizing the indicator as the new anchor.
4. Formulate the loss on these selected anchors and perform a gradient update.
5. Repeat over training epochs, biasing the training set toward regions of highest estimated error (Chen et al., 2023).

A summary table of ATT workflow in both domains:

Domain	Matching Target	Adaptive Step/Point Selection
Dataset Distillation	Student-expert trajectory	$N_S$5
PDE Solvers	PDE residual/error	$N_S$6 across SDE samples

4. Theoretical Properties and Analysis

ATT eliminates the key error accumulation mechanism (AMP) by always selecting the optimal intermediate state for matching, rather than conflating synthetic and real trajectory lengths. This reduces the per-iteration matching error, and by extension, the Accumulated Trajectory Error (ATE). For instance, given

$N_S$7

and update errors propagating additively through distillation, selecting $N_S$8 that minimizes $N_S$9 suppresses the ATE's growth (Liu et al., 2024).

In deep PDE solvers, the theoretical rationale is that trajectories establish temporal (Markov) dependencies, reducing training variance and focusing updates where the error or residual is empirically highest. This empirical targeting leads, under mild conditions, to reductions in integration or residual error—often with comparable or slightly increased computational cost (Chen et al., 2023).

5. Empirical Performance and Impact

ATT has demonstrated improvements in generalization, convergence stability, and accuracy across distinct application domains.

• Dataset Distillation Cross-Architecture Generalization: On CIFAR-10 (10 images-per-class), ATT outperforms FTL methods by 1.3% on ResNet18, 7.2% on VGG11, and 7.7% on AlexNet. On full-dataset settings, its gains range from 0.8% (CIFAR-10) to 1.3% (Tiny-ImageNet) (Liu et al., 2024).
• Parameter Stability: The ATT method is robust to large learning rate multipliers, unlike FTL approaches, which diverge under similar conditions (Liu et al., 2024).
• PDE Solvers: ATT/ATS decreases errors by up to two orders of magnitude in high-dimensional settings. For example, on d=30-dimensional Poisson problems, switching from uniform PINN to ATS-PINN with derivative indicator cuts error by a factor of 100, often at only 5–30% additional computational cost (Chen et al., 2023).
• Convergence Behavior: ATT automatically selects shorter matches early in distillation, lengthening them as learning stabilizes, thereby avoiding catastrophic misalignment at early stages (Liu et al., 2024).
• Wall Time: For dataset distillation, ATT’s per-iteration runtime matches that of vanilla long-range matching due to frequent early stopping in matching steps (Liu et al., 2024).

6. Domain-Specific Integrations and Extensions

ATT/ATS has been adopted and customized for diverse solver architectures:

• Derivative-Free Methods: Adaptive selection using Bellman errors and SDE-driven exploration (Chen et al., 2023).
• Physics-Informed Neural Networks (PINNs): Using Brownian-motion-based proposals and differential-operator residuals as error indicators.
• Temporal-Difference Learning for FBSDEs: Error indicators based on temporal differences, with trajectory proposals via Euler–Maruyama discretizations.

In all cases, the principle is to target the collocation/learning budget on the most error-prone regions or time points, increasing sample efficiency and reducing variance.

7. Connections, Limitations, and Outlook

ATT is closely related to importance sampling and minimum-error matching in bi-level optimization. The approach provably cannot perform worse than fixed-step-matching baselines when the best-matching point is always selected. A plausible implication is that ATT provides a robust principle for any setting where sample efficiency and error-concentration are crucial, though theoretical convergence guarantees remain limited to empirical justification. No closed-form convergence proofs are available, but all published evidence indicates improved stability, generalization, and efficiency (Liu et al., 2024, Chen et al., 2023).

ATT does not maintain explicit sample densities; adaptation is realized implicitly through sequential minimum-error selection. ATT and related schemes have become a foundation for state-of-the-art distilled data generation, operator learning under constraints, and high-dimensional scientific computing.