Locally Adaptive Test-Time Scaling

• Locally Adaptive Test-Time Scaling (LATTS) is a framework that dynamically allocates computational resources during inference based on local difficulty, optimizing efficiency.
• It uses verifier models, confidence heuristics, and learned controllers to focus extra computation on challenging reasoning steps and spatial regions.
• Empirical studies show that LATTS achieves significant accuracy improvements and speedups over uniform scaling methods in chain-of-thought, diffusion, and block diffusion models.

Locally Adaptive Test-Time Scaling (LATTS) is a suite of methodologies for adaptively allocating computational effort at inference time so that it is focused on challenging regions, steps, or samples. In contrast to traditional test-time scaling approaches, which apply uniform compute budgets to all instances or all segments of a generation, LATTS dynamically modulates where and how much additional computation is used. LATTS was first formalized in the context of chain-of-thought reasoning for LLMs, but has since been extended to diffusion models and block diffusion LLMs. Central to LATTS is the concept of local adaptivity: compute resources are concentrated at the spatial, temporal, or structural loci determined to be difficult or unreliable, as identified by (i) verifier models, (ii) confidence heuristics, or (iii) learned controllers. Empirical results across multiple modalities and benchmarks demonstrate that LATTS methods achieve substantially improved accuracy–compute trade-offs over uniform or global scaling baselines (Uscidda et al., 16 Sep 2025, Xu et al., 10 Feb 2026, Ren et al., 25 Nov 2025, Lu et al., 10 Feb 2026).

1. Theoretical Foundations and Motivation

Test-time scaling broadly refers to methods that improve model performance by increasing inference-time computation, such as sampling multiple chains, applying beam search, or post-hoc reranking. However, most traditional TTS approaches scale computation homogeneously across all inputs and output regions, resulting in suboptimal use of resources. LATTS addresses this by reallocating compute in a locally sensitive manner, informed by the predicted or observed difficulty of individual steps (in chain-of-thought models), spatial patches (in diffusion models), or answer consensus (in adaptive sampling).

The foundational principle is the formalization of "local difficulty." In sequence models, for each generation step $t$ given prefix $S_{<t}$ and input $x$, the local difficulty is defined as

$$\Delta(x, S_{<t}) = 1 - \mathbb{E}_{s_t \sim p_\text{model}}[r_f(s_t \mid x, S_{<t})] \in [0,1]$$

where $r_f$ is a modulated verifier score (such as a PRM or a critic). Steps with $\Delta$ near zero are easy, requiring minimal extra effort; those with $\Delta$ near one are hard, justifying additional computation. The expected number of trials per step under LATTS is $E[n_t] = 1/(1 - \Delta(x, S_{<t}))$, focusing compute on segments deemed difficult (Uscidda et al., 16 Sep 2025).

2. LATTS for Chain-of-Thought: Verifier-Guided Generation

In classical verifier-guided chain-of-thought (CoT) generation, candidate solutions are sampled uniformly and then post-hoc filtered or aggregated by a verifier. LATTS introduces step-wise verifier-guided acceptance–rejection (AR) sampling (Uscidda et al., 16 Sep 2025, Tan et al., 2 Apr 2025). The process is as follows:

• At each generation step, candidate outputs are proposed by the model.
• A verifier evaluates each candidate, providing a scalar reliability score.
• Acceptance of a candidate is stochastic, proportional to its verifier score (identity or threshold modulator).
• If no candidate passes, predefined fallbacks are employed: forced selection ("max"), backtracking, or restart.

Algorithmically, this induces computation to be spent on steps where verifier scores are low (highly uncertain or likely incorrect), while high-confidence steps proceed rapidly. LATTS is general over the choice of verifier: it is compatible with process-supervised reward models, LLM-based step critics, or even gradient-based uncertainty proxies.

Extensions such as Adaptive Rectification Sampling (AR-Sampling) further inject adaptivity by issuing trigger sentences only when the PRM detects a step-level error, causing localized rethinking without wasting tokens on already correct reasonings (Tan et al., 2 Apr 2025). Empirically, this leads to 2–5 point accuracy improvements over conventional majority-vote or best-of-N schemes, at a modest token cost.

3. Hypernetwork-Driven Layer-Wise LATTS for LLMs

Recent work has advanced LATTS by introducing fine-grained, unsupervised adaptation at the level of model parameters. In the Unsupervised Layer-Wise Dynamic Test-Time Adaptation framework, a lightweight hypernetwork, ScaleNet, is trained to output per-layer, per-adaptation-step learning rate multipliers for LoRA-injected parameters (Xu et al., 10 Feb 2026). The workflow is:

• For each input prompt $x$, model LoRA factors are reset and a small number of unsupervised gradients steps are performed.
• ScaleNet summarizes the prompt via pooled hidden states, concatenates step information, and outputs scaling factors for each Transformer block's Q/V LoRA parameters.
• Updates have the form $$\Delta W_l \leftarrow -\eta_0 \cdot \alpha_{l,k} \cdot \nabla_{W_l} L_{unsup}(x; \theta^{(k)})$$, where $S_{<t}$0 is a nonnegative multiplier from a parameterized function.
• ScaleNet is meta-trained to minimize supervised loss (e.g., NLL) after $S_{<t}$1 unsupervised adaptation steps, using a first-order approximated meta-optimization.

The approach addresses the instability of fixed learning rate adaptation—per-layer, per-step scaling results in improved stability and generation quality, consistently outperforming both fixed-lr and per-step-only baselines on metrics such as NLL and ROUGE-Lsum. The learned schedules exhibit prompt- and layer-sensitivity, allocating adaptation strength where it is most beneficial (Xu et al., 10 Feb 2026).

4. RL-Guided and Constrained Adaptive Sampling

LATTS principles extend to adaptive sampling for LLMs, where the objective is to balance accuracy, compute, and latency without model internals. Here, the adaptive sampling problem is formulated as a Markov Decision Process (MDP) (Dai et al., 2 Jun 2026):

• The controller observes statistics of the answer pool (e.g., answer counts, entropy) after each sampling round.
• Actions correspond to halting or drawing additional samples; rewards penalize computation and reward correctness.
• The policy is a small MLP trained with Proximal Policy Optimization (PPO) to maximize a Lagrangian objective that encodes accuracy under latency and computation constraints.
• The stopping policy is locally adaptive: each input induces its own policy trajectory depending on the evolving distribution of answers.

Against self-consistency and heuristic early stopping baselines, RL-guided LATTS reduces sampling rounds and total samples for the same or better accuracy. For Qwen-1.7B, RL-guided LATTS achieves 46.3% accuracy on three math datasets with 3.3 rounds and 13.0 samples, outperforming ASC and ESC (Dai et al., 2 Jun 2026).

5. Localized LATTS for Diffusion and Block Diffusion Models

LATTS has been extended to vision generation via the LoTTS paradigm, which adaptively applies test-time scaling spatially within images produced by diffusion models (Ren et al., 25 Nov 2025). Key components:

• Defect localization via cross- and self-attention maps under quality-aware prompts, fused into coherent spatial masks.
• Targeted noise injection and localized denoising, with full-image integration only at the final steps, dramatically reducing GPU cost by 2–4x and improving local and global quality metrics (e.g., HPS, FID, CLIP).

In block diffusion LLMs (BDLMs), Bounded Adaptive Confidence Decoding (BACD) adaptively thresholds which tokens to unmask based on their model confidence, yielding aggressive decoding for easy tokens and slow refinement for difficult ones, bounded by user-specified thresholds (Lu et al., 10 Feb 2026). Combined with the Think Coarse, Critic Fine (TCCF) paradigm—large block sizes for exploration ("think"), small block sizes for checking ("critic")—this results in substantial efficiency-effectiveness gains. Empirical results on TDAR-8B show 2.26× speedup and +11.2 points on AIME24 compared to best prior BDLMs (Lu et al., 10 Feb 2026).

6. Empirical Results and Limitations

The table below summarizes representative empirical results for LATTS variants across domains, as reported in the cited works:

Task/Model	Method	Accuracy/Gain	Compute Cost	Reference
Llama3.3-70B AdaptEval	No TTA	NLL = 2.2114	-	(Xu et al., 10 Feb 2026)
"	LATTS (2stp)	NLL = 1.6692	Stable for K ≤ 5	(Xu et al., 10 Feb 2026)
Qwen-1.7B Math (averaged)	ASC	46.1%	18.9 rounds/samples	(Dai et al., 2 Jun 2026)
"	RL-LATTS	46.3%	3.3 rounds/13.0 samples	(Dai et al., 2 Jun 2026)
MATH500, Llama3.2-1B	BoN	42.2% (at @16)	High	(Tan et al., 2 Apr 2025)
"	AR+BoN	43.4% (at @16)	+~50 tokens/sol.	(Tan et al., 2 Apr 2025)
SD2.1 Diffusion, Best-of-N(9)	Best-of-N	HPS=21.56, FID=13.21	N=9 global samples	(Ren et al., 25 Nov 2025)
"	LoTTS	HPS=24.52, FID=10.89	N=9, 2.8× faster	(Ren et al., 25 Nov 2025)
AIME24, TDAR-8B	TraDo-8B	-	Baseline	(Lu et al., 10 Feb 2026)
"	LATTS	+11.2 pts, 2.26× spd	Enhanced TPF	(Lu et al., 10 Feb 2026)

The consistently superior accuracy–compute tradeoffs observed for LATTS variants reflect the advantage of local, difficulty-aware scaling across sequential, adaptive sampling, and spatial domains.

Limitations of LATTS include: sensitivity to verifier model calibration, necessity for hyperparameter tuning (e.g., acceptance threshold δ, block sizes, fallback strategies), and potential degraded performance where local difficulty cannot be reliably estimated (Uscidda et al., 16 Sep 2025, Ren et al., 25 Nov 2025). In out-of-distribution or poorly verified regions, excessive local adaptation may result in wasted compute or non-convergent behavior.

7. Future Directions and Open Challenges

Several open problems and directions for advancing LATTS methodologies are identified in the literature:

• Verifier learning and adaptation: Robustness to verifier miscalibration, adaptation of the modulator $S_{<t}$2 online, and composition with retrieval or external memories in difficult regions (Uscidda et al., 16 Sep 2025).
• Constrained and preference-conditioned control: Strict budget enforcement via constrained-RL (e.g., CPO), incorporation of user-specified cost-accuracy trade-offs, or multi-objective optimization (Dai et al., 2 Jun 2026).
• Broader domain application: Extending LATTS to code generation, scientific reasoning, and commonsense tasks, including learned triggers and annotation strategies for PRMs in these domains (Tan et al., 2 Apr 2025).
• Hybrid scaling paradigms: Combining step- and block-level adaptivity with beam search, tree-of-thought, or dynamic noise schedules, especially in hierarchical generative models (Lu et al., 10 Feb 2026).
• Theoretical analysis: Elucidation of sample complexity and regret of LATTS versus uniform baselines under realistic noise and uncertainty conditions, particularly in the presence of sparse ground truth (Uscidda et al., 16 Sep 2025).

Locally Adaptive Test-Time Scaling thus redefines inference-time resource allocation by making it instance- and region-aware, yielding robust accuracy gains and efficient compute usage across language, vision, and hybrid generative domains.