Spectrum-Aware Test-Time Steering

• The paper introduces a dynamic, spectrum-aware framework that selects optimal decoding strategies by maximizing a utility function combining accuracy, token cost, and latency.
• The methodology leverages empirical mean cost models and calibrated MLP accuracy predictions, allowing per-query routing among diverse inference strategies.
• Empirical results demonstrate significant improvements in both accuracy and efficiency for LLMs and VLMs, with fast, parameter-efficient test-time adaptation.

Spectrum-Aware Test-Time Steering (STS) denotes a family of dynamically adaptive mechanisms for routing queries or inputs across a finely parameterized “spectrum” of strategies, either at the level of decoding policies in generative models or adaptation shifts in representation space, in order to optimize a utility function that jointly considers accuracy, computational cost, and latency. The unifying feature is continuous or high-resolution steering among possible compute pathways, with joint awareness of spectrum-level trade-offs. This article details two lines of recent research under the STS designation: (1) inference scaling and decoding strategy routing in LLMs (Huang et al., 11 Sep 2025), and (2) principled latent-space steering for test-time adaptation in vision-LLMs (VLMs) (Dafnis et al., 12 Nov 2025).

1. Formal Problem Setting: Dynamic Spectrum Routing

STS in LLMs formalizes the inference-time scaling problem as dynamic, per-query selection from a set $S$ of candidate strategies, $s = (m, \theta_m)$, where $m$ may be best-of-$N$ sampling, beam search, or any other decoding policy, and $\theta_m$ comprises hyperparameters such as $N$ (number of samples), beam width, and depth (Huang et al., 11 Sep 2025). For each query $x$ and strategy $s$:

• $a_s(x) \in [0, 1]$: Predicted accuracy or reward.
• $T_s(x) \geq 0$: Expected output token cost.
• $L_s(x) \geq 0$: Predicted wall-clock latency.

A utility function is defined: $$U_s(x) = a_s(x) - \lambda_T T_s(x) - \lambda_L L_s(x)$$ where $\lambda_T$, $\lambda_L \geq 0$ specify user penalties for token and latency cost, respectively. The optimal strategy is

$s^*(x) = \arg\max_{s \in S} U_s(x).$

Alternatively, with hard constraints, one solves: $$\max_{s \in S} a_s(x), \quad \text{s.t.}~ T_s(x) \leq c,~ L_s(x) \leq \ell.$$

This framework generalizes static approaches, treating the space $S$ as a spectrum over which queries can be routed according to their predicted difficulty and cost profile.

2. Cost Modeling and Prediction Framework

The STS approach circumvents the unavailability of $T_s(x)$, $L_s(x)$ at prediction time by employing empirical mean cost models. For each strategy $s$: $$\mu_T(s) = \mathbb{E}_{x \sim D_{\mathrm{train}}}[T_s(x)], \quad \mu_L(s) = \mathbb{E}_{x \sim D_{\mathrm{train}}}[L_s(x)],$$ which are used in place of per-query estimates. For $a_s(x)$, a two-layer MLP is trained to predict likelihood of correctness, using features comprising both an embedding $e(x)$ of the input and contextual features $\phi(s)$ of the strategy: $f(x, s) = [e(x); \phi(s)].$ Platt scaling is used for improved calibration.

At test time, for user-specified $\lambda_T$, $\lambda_L$, the utility is given by: $$\tilde{U}_s(x) = \hat{a}_s(x) - \lambda_T \mu_T(s) - \lambda_L \mu_L(s).$$ The final chosen strategy $\hat{s}(x)$ maximizes this surrogate utility, after which the model is decoded via the corresponding $(m, \theta_m)$.

Empirical analysis demonstrates that mean cost proxies produce negligible loss ($\sim$1–2%) relative to ground-truth costs, and that the framework is robust to the use of varied embedding backbones (Huang et al., 11 Sep 2025).

3. Algorithmic and Operational Mechanics

A canonical STS routing process for LLMs comprises the following sequence:

1. Feature Extraction: Compute a semantic representation $e(x)$ (e.g., Qwen2.5-Instruct, BERT, etc.) for query $x$; concatenate with strategy features $\phi(s)$.
2. Accuracy Estimation: MLP predicts $\hat{a}_s(x)$, calibrated with empirical soft labels.
3. Cost Retrieval: Look up $\mu_T(s)$ and $\mu_L(s)$ for every $s$.
4. Utility Maximization: For each $s$, compute $\tilde{U}_s(x)$, select $\hat{s}(x)$.
5. Decoding: Apply the selected decoding method $(m, \theta_m)$ to output the result.

Routing is thus data- and spectrum-aware: queries predicted to be hard or ambiguous are steered toward computationally intensive strategies (e.g., deep beam search), while simple queries use lightweight methods (e.g., best-of-2). The spectrum can be arbitrarily enriched with new families of decoding methods or extended to cost axes beyond tokens and latency (e.g., GPU memory, energy).

A similar paradigm is applied in test-time adaptation for VLMs (Dafnis et al., 12 Nov 2025), where the “spectrum” is a spectral subspace extracted from textual prototypes, and steering is performed by learning per-sample shifts in the principal semantic directions.

4. Spectrum-Aware Steering in Latent Space for VLMs

In STS for VLMs (Dafnis et al., 12 Nov 2025), a “spectral subspace” in the semantic embedding space is extracted from the covariance of initial class prototypes $Z_{T_{\mathrm{init}}}$ generated by the frozen text encoder, resulting in a principal basis $U_k \in \mathbb{R}^{D \times k}$ ($k \ll D$). For a test image, a single vector $\boldsymbol{\delta} \in \mathbb{R}^k$ is learned to generate a latent shift $\Delta z_T = U_k \boldsymbol{\delta}$ that is added to all class prototypes and renormalized; this adapted set is used for prediction.

The shift $\boldsymbol{\delta}$ is optimized per-sample at test time to minimize the entropy of predictions across $N$ augmented views of the input, per

$$\mathcal{L}_{\mathrm{STS}} = -\sum_c \bar{p}_c \log \bar{p}_c + \lambda \|\Delta z_T\|_2$$

where $\bar{p}_c$ is the marginal probability for class $c$ across confidence-filtered views.

Key operational properties include:

• Only $k$ parameters are optimized; encoders are frozen.
• No backpropagation through image/encoder weights is required.
• A single gradient step suffices for near-optimal adaptation.
• Typical $k$ is on the order of $10$–$20$, capturing $>$90% feature variance.

5. Quantitative Results and Trade-offs

LLM Decoding (Huang et al., 11 Sep 2025)

STS in LLMs, evaluated on NuminaMath-CoT with Qwen2.5-1.5B-Instruct and a reward model, achieves:

Setting	Max Accuracy	Token Cost (approx)	Latency (approx)
Static Beam Search	~0.45	~2000+	~60s
Static Best-of-N	<0.45	--	--
STS (Adaptive)	0.50	~2000	~40s

• STS dominates both accuracy–cost and accuracy–latency trade-offs across the spectrum $S$.
• At low penalties ($\lambda_T, \lambda_L$ small), most queries route to high-cost strategies; as penalties increase, routing shifts to cheaper configurations without major accuracy loss.
• Dynamic adaptation within a single method family (e.g., only beam search, varying parameters) gives 3–5% accuracy improvements at fixed cost.

Vision-LLM Adaptation (Dafnis et al., 12 Nov 2025)

STS for VLMs, using CLIP-ViT-B/16, demonstrates:

Method	OOD-avg Accuracy	Inference Time (s)	GPU Memory (GB)
Zero-Shot	57.20%	--	--
TPT (Prompt Tuning)	60.71%	0.75	17.6
STS (Single)	62.64%	0.09	1.4
STS_Ensemble	64.96%	--	--

• STS achieves significant speed (8x faster inference) and footprint (12x smaller memory) gains relative to test-time prompt tuning, while offering higher OOD robustness.
• Prompt ensembling further lifts accuracy ceiling to 64.96% OOD-avg over diverse OOD and fine-grained splits.
• Under corruptions (CIFAR10-C), STS matches or exceeds TPT.

6. Extensibility, Generalization, and Practicality

The STS framework offers several extensibility and deployment strengths:

• Spectrum Enrichment: In LLMs, $S$ may admit new decoding paradigms (tree-of-thought, multi-model routing) without altering the routing mechanism. In VLMs, new basis selection or regularization strategies can be swapped in.
• Cost-Axis Generalization: Additional cost axes (GPU memory, energy, or external call delays) can be incorporated as new penalty terms $\lambda$ in the utility function, supporting mixed-objective routing.
• Real-Time Suitability: Mean-cost lookups and low-parameter probes enable practical deployment in real-time agentic and interactive settings, where wall-clock delay is as critical as token usage.
• Empirical Robustness: Predictive proxies for costs and accuracies are reliable; ablations confirm that simple feature choices and single-step adaptation suffice for near-optimal performance.
• Parameter Efficiency: In latent-space steering, only a handful of per-sample parameters need to be optimized, facilitating rapid and scalable adaptation.

7. Significance, Limitations, and Open Directions

STS represents a systematic, spectrum-aware alternative to static or parallel generation methods for test-time strategy selection, providing flexible, data-driven adjustment to per-query computational budget and required response qualities. Numerical results indicate consistent gains in both accuracy and efficiency over baselines, with low operational overhead.

However, limitations include:

• The utility maximization relies on calibration of predictors and cost accuracy; gross misestimation may lead to suboptimal routing.
• For some deployment scenarios, fine-grained latency measurement and cost estimation may require continual recalibration.
• In VLM adaptation, the reliance on entropy minimization with augmentation assumes that OOD and errorful views are filtered; severe distributional shifts not captured by the textual subspace may require deeper adaptation.

A plausible implication is that future research will extend the STS paradigm to multi-modal, multi-agent, or highly dynamic environments, possibly incorporating reinforcement learning for online utility function tuning or integrating richer spectrum structures beyond simple hyperparameter grids.