Hyperparameter Trajectory Inference (HTI)

• Hyperparameter Trajectory Inference (HTI) is a framework that defines training metric trajectories to reduce high-dimensional hyperparameter spaces into a one-dimensional sweep.
• It employs deterministic recursions and context summarizers to simulate, predict, and select hyperparameters efficiently, saving significant computational resources.
• Advanced methods integrate optimal transport and adaptive MCMC to infer conditional hyperparameter dynamics, achieving superior performance in diverse learning settings.

Hyperparameter Trajectory Inference (HTI) encompasses a family of principled methodologies for analyzing, predicting, and optimizing the effect of hyperparameters on learning dynamics. HTI exploits the observation that many hyperparameter choices induce characteristic “trajectories” in training metrics—loss, gradient statistics, optimization outputs—whose structure can be captured, reduced, or transported across hyperparameter space. This enables hyperparameter selection, surrogate modeling, and automated tuning far more efficiently than through naïve high-dimensional grid search. Recent advances formalize HTI for both SGD-based deep learning (Li et al., 29 Sep 2025), conditional output inference (Amad et al., 2 Mar 2026), automated tuning pipelines (Naphade et al., 19 Sep 2025), and adaptive MCMC (Sountsov et al., 2021). HTI is now a central abstraction for understanding and leveraging trajectory invariances in large-scale optimization and machine learning systems.

1. Trajectory Invariance Principles and Dimensionality Reduction

A foundational result for HTI is the trajectory-invariance principle, as established by Li et al. (Li et al., 29 Sep 2025). Empirical studies on large transformers reveal that after an initial regime dominated by the raw learning rate (η), loss curves, gradient-noise scales, and gradient norms all collapse onto a family parameterized by the product φ = ηλ (where λ is the weight decay). Thus, the full two-dimensional (η, λ) hyperparameter space effectively reduces to a one-dimensional sweep along φ.

This reduction proceeds in two regimes:

• Regime I (early training): Trajectories depend primarily on η, independent of λ.
• Regime II (late training): Trajectories collapse when ηλ = η′λ′, i.e., for equal φ.

Beyond loss curves, the collapse extends to the preconditioned gradient-noise scale $\mathcal{B}_{\mathrm{precond}}(t)$ and the preconditioned gradient norm. Quantitative measures of within-φ and between-φ distances (order 10⁻³ vs. an order of magnitude larger, respectively) confirm the strength of this invariance.

2. Statistical Prediction and Low-Dimensional State Trajectories

HTI is not restricted to SGD: in stochastic prox-linear methods for matrix sensing, error trajectories of the algorithm concentrate tightly around the output of a low-dimensional deterministic recursion whose evolution depends exclusively and transparently on step-size (η), batch-size (b), and noise level (σ) (Lou et al., 2024). For rank-1 matrix recovery, this concentrates the full stochastic process into a four-dimensional state, whose transition map predicts (with non-asymptotic accuracy) the contraction rate, statistical error floor, and transient behavior.

Hyperparameters can be chosen without ever running the high-dimensional stochastic algorithm: one simulates the deterministic map for candidate (η, b, σ, d), computes predicted error curves, and minimizes sample complexity or error floor offline.

3. Automated HTI in Hyperparameter Tuning Pipelines

HTI is operationalized in recent automated hyperparameter tuning pipelines using small LLMs (Naphade et al., 19 Sep 2025). The core mechanism is the Trajectory Context Summarizer (TCS), a deterministic pre-processing module producing compact, structured context vectors encoding not only trial-wise hyperparameters and losses but also moving averages, discrete gradients, curvatures, and per-parameter loss histories.

The hyperparameter trajectory is formalized as $T = \{ (t_i, \theta_i, \ell_i) \}_{i=1}^N$, which TCS transforms into a state-like report. This deterministic context enables modestly-sized LLMs (e.g., phi4:reasoning14B) to analyze hyperparameter search spaces and suggest effective trial configurations with reliability comparable to much larger models. Empirically, TCS-enabled pipelines achieve performance within ~0.9 percentage points of GPT-4 across multiple ML tasks, highlighting the power of proper trajectory summarization in HTI.

4. Conditional Hyperparameter Trajectory Inference via Optimal Transport

In settings where model behavior depends nontrivially on a hyperparameter and must be inferred (e.g., evolving reward weights in RL, quantile parameters in regression), one seeks not just a single optimal hyperparameter but to learn how conditional output distributions evolve across hyperparameter space. The CLOT framework (Amad et al., 2 Mar 2026) re-casts this as conditional Lagrangian optimal transport.

Given a family of conditional marginals $\{\mu_k(\cdot|x)\}$ observed at anchor hyperparameters, HTI aims to reconstruct the continuous geodesic trajectory $\lambda \mapsto \mu(\cdot|x,\lambda)$ for unobserved $\lambda$. This is achieved by learning (i) a data-dependent Lagrangian (kinetic and potential terms), (ii) the geodesic interpolations, and (iii) the optimal-transport maps using a min–max learning objective. CLOT surrogates recover the target distribution across $\lambda$ more accurately than direct regression or flow-matching, for RL (reward weighting), quantile regression, and generative modeling, as assessed by MSE, NLL, Wasserstein, and episode reward metrics.

Table 1: Empirical Performance of CLOT-HTI on Key Tasks (Amad et al., 2 Mar 2026)

Setting	Baseline (MSE/NLL/Reward/WD)	CLOT-HTI (Best Variant)
RL, Cancer reward	-38.4 (Direct)	102.5
Quantile regression (MSE)	1.845 (Direct)	0.608
Gen. model, 2-moons (WD)	0.090 (MFM)	0.060

5. Algorithmic Implementations and Pseudocode

HTI methodologies are often instantiated algorithmically as one-dimensional sweeps or deterministic recursions, replacing expensive grid searches. For SGD-based deep models, the typical workflow is:

1. Fix baseline η₀.
2. Define a φ-sweep {φ₁, ..., φ_K}.
3. For each φₖ, set λ = φₖ / η₀ and run a short probe trajectory.
4. Evaluate probe performance (loss, gradient noise, or norm).
5. Select φ* = argminₖ Mₖ; recover (η, λ).

This routine leads to 5× or greater compute savings compared to full η-λ grids (Li et al., 29 Sep 2025). For stochastic prox-linear algorithms, a closed-form recursion predicts optimal hyperparameters directly.

In discrete trial-based HPT settings, the process is formalized as a finite-horizon Markov decision process, and TCS acts as a deterministic expert block generating the state summary fed to downstream optimizer and analysis agents (small LLMs) (Naphade et al., 19 Sep 2025).

6. Adaptive HTI in MCMC: Learning HMC Trajectory Lengths

Trajectory inference within Markov Chain Monte Carlo takes a particular form in adaptive HMC. Here, SNAPER-HMC (Sountsov et al., 2021) infers the optimal HMC trajectory length (τ) online by maximizing a principled, per-gradient effective sample size bound along the most difficult principal component estimated by minibatch Oja’s algorithm.

The adaptation loop samples τ ∼ Uniform(0, 2 \bar τ), computes a cost-normalized effectiveness (ESJD/τ), and adjusts log \bar τ with ADAM. This approach is SIMD-accelerator-friendly, robust across pathological target covariances, and shows long-run ESS/grad and short-run efficiency competitive or superior to NUTS and CheESR, without speculative grid-search over τ.

7. Limitations, Assumptions, and Future Extensions

HTI's effectiveness varies with regime, dataset, and model scale. For trajectory invariance in SGD, regime transitions are driven by iteration count rather than data volume. Most published results target models in the 100–200M parameter range; thresholds and invariance properties may shift at larger scale.

Extensions to higher-dimensional hyperparameter manifolds (e.g., φ = φ(η, λ, β₂, ...)), better integration with batch scheduling, and principled generalization to multi-epoch training or non-SGD optimizers remain open. In conditional inference, injective mapping and feasibility underlie the success of Lagrangian OT approaches.

A plausible implication is that future HTI research may further unify continuous trajectory prediction with discrete trial-based optimization, enabling both efficient offline hyperparameter selection and robust online adaptation in dynamic or post-deployment environments.