Fine-Grained Test-Time Optimization Techniques

• Fine-Grained Test-Time Optimization is a design principle that targets local model errors by replacing global control signals with structured, unit-specific interventions.
• Techniques include decomposing prompts, refining search budgets, and dynamically updating parameter subsets, allowing precise corrections at token, concept, or subgroup levels.
• Empirical results across domains show significant gains in accuracy and efficiency by reallocating test-time compute where model failure occurs.

Fine-grained test-time optimization denotes a family of inference-time methods that replace coarse, global, or monolithic control signals with structured units such as prompt concepts, reasoning directions, verification intervals, subgroup-specific pseudo-labels, token-level faithfulness scores, action-unit prompts, attention-guided regions, expert-routing subsets, or task-local parameter updates. Across recent work, the optimized object may be a prompt, a search budget, a verifier schedule, a prompt embedding, a small parameter subset, a routing decision, or a candidate pool; what unifies these methods is the attempt to spend test-time compute where failure actually occurs rather than at the level of a whole output or a single global score (Sameti et al., 27 Sep 2025, Wang et al., 30 May 2025, Chen et al., 16 May 2025, Wang et al., 17 Dec 2025, Sun et al., 10 Mar 2026, Zeeshan et al., 30 Mar 2026, Hai et al., 19 May 2026, Chen et al., 2 Feb 2026, Gozeten et al., 14 Mar 2025).

1. Conceptual foundations and historical framing

The literature does not present a single canonical definition of fine-grained test-time optimization. Instead, it converges on a common diagnosis: coarse test-time control is often mismatched to the structure of model failure. In compositional text-to-image generation, a model may render the overall scene plausibly while omitting or misbinding a single object, attribute, or relation; in reasoning search, multiple candidates may be near-duplicates of the same latent direction; in test-time reinforcement learning, majority voting may collapse diverse trajectories into one sparse pseudo-label; and in verifier-guided search, step-wise verification or final-only verification may both be suboptimal because verification frequency itself is a control variable (Sameti et al., 27 Sep 2025, Wang et al., 30 May 2025, Wang et al., 17 Dec 2025, Chen et al., 16 May 2025).

A chronological reading of the cited work shows the topic broadening over time. Earlier papers already contained key ingredients: post-hoc hierarchy-aware score correction for fine-grained visual classification without parameter updates (Jain et al., 2023), per-example speech test-time training with selective parameter adaptation (Dumpala et al., 2023), and meta-trained learned optimizers designed for fast per-task self-adaptation (Yang et al., 2023). Later work expanded the space of fine-grained control to learned gradient generators for online TTA (Deng et al., 2024), supervised weight updates for in-context learning (Gozeten et al., 14 Mar 2025), prompt- and search-based reasoning methods (Wang et al., 30 May 2025, Chen et al., 16 May 2025, Wang et al., 17 Dec 2025), compositional multimodal generation (Sameti et al., 27 Sep 2025), and fine-grained personalization or robustness mechanisms in vision-language systems (Zeeshan et al., 30 Mar 2026, Hai et al., 19 May 2026).

This suggests that the term is best understood as a design principle rather than a single algorithmic family. The principle is to expose a more faithful unit of intervention at test time: not merely “improve the answer,” but “improve the missing concept,” “reallocate rollouts across latent directions,” “change verifier cadence,” “update only biases,” “tune only AU prompt embeddings,” or “randomize only the uncertain routing tail.”

2. Axes of granularity

Different papers operationalize granularity at different loci of the inference pipeline. Some methods decompose the input specification; others refine the search frontier, reward signal, verification schedule, internal routing, or adaptation subspace. The following taxonomy is descriptive rather than exhaustive.

Granularity unit	Test-time operation	Representative papers
Semantic concepts, objects, attributes, relations	Prompt decomposition, concept-level scoring, prompt rewriting	(Sameti et al., 27 Sep 2025)
Reasoning directions, verifier intervals, rollout branches	Allocation, pruning, search scheduling	(Wang et al., 30 May 2025, Chen et al., 16 May 2025, Huang et al., 29 May 2026)
Steps, subgroups, tokens	Reward estimation, pseudo-labeling, process verification	(Wang et al., 17 Dec 2025, Sun et al., 10 Mar 2026, Shen et al., 15 Jan 2026)
Temporal windows, AU prompts, semantic regions, expert tails	Prompt tuning, view selection, routing perturbation	(Zeeshan et al., 30 Mar 2026, Hai et al., 19 May 2026, Chen et al., 2 Feb 2026)
Parameter subsets, task-local weights, hierarchical posteriors	Bias-only TTT, in-context weight update, post-hoc correction	(Dumpala et al., 2023, Gozeten et al., 14 Mar 2025, Jain et al., 2023)

Two clarifications recur across the literature. First, “fine-grained” does not necessarily mean token-level. In some papers it means concept-level prompt decomposition, direction-level rollout allocation, subgroup-level pseudo-labels, or window-level personalization (Sameti et al., 27 Sep 2025, Wang et al., 30 May 2025, Wang et al., 17 Dec 2025, Zeeshan et al., 30 Mar 2026). Second, “test-time” does not necessarily mean online weight adaptation. Several central examples are search, reranking, routing, prompt rewriting, or score manipulation with frozen base models (Wang et al., 30 May 2025, Chen et al., 16 May 2025, Sun et al., 10 Mar 2026, Chen et al., 2 Feb 2026, Jain et al., 2023).

3. Methodological families

One major family consists of search, allocation, and routing control without model-weight updates. In this regime, performance gains come from spending fixed inference budget more intelligently. DORA formulates parallel reasoning search as a fixed-budget success-maximization problem and argues that rollout allocation should be direction-oriented rather than solution-level, correcting the bias toward overrepresented reasoning directions (Wang et al., 30 May 2025). VG-Search treats verification granularity $g$ as a continuous control variable between beam-search-like step verification and Best-of-$N$-style final verification, then adds validation-based adaptive policies (Chen et al., 16 May 2025). GeoSolver uses a token-level process reward model both for Best-of-$N$ reranking and step-wise beam pruning in remote sensing VLM reasoning (Sun et al., 10 Mar 2026). Expert-Sample perturbs only the uncertain tail of MoE expert routing while preserving the confident head, thereby moving diversity control from token sampling into internal expert selection (Chen et al., 2 Feb 2026). UniScale places model routing and TTS in one online contextual-bandit action space, making both model choice and search intensity part of a joint inference policy (Huang et al., 29 May 2026).

A second family performs prompt- or output-space optimization. In compositional text-to-image generation, “No Concept Left Behind” decomposes the prompt $P$ into a concept set $\mathcal{C}(P)=\{c_1,\dots,c_k\}$, scores both global and concept-level alignment with FG-CLIP, and feeds those per-concept scores into an LLM-driven prompt rewriting loop (Sameti et al., 27 Sep 2025). CLIP-AUTT selects a minimum-entropy temporal window from a target video and tunes AU prompt embeddings for that video only, keeping the CLIP encoders frozen (Zeeshan et al., 30 Mar 2026). A-TPT updates prompts only, but conditions its view generation and ensemble weighting on refined attention rollout maps so that augmentation is spatially aware in fine-grained recognition (Hai et al., 19 May 2026). TSAN keeps the base LLM frozen and replaces single-candidate critique-revise with a textual self-attention analogue over multiple candidates, using textual keys, values, and attention scores to synthesize a new answer (Mo et al., 10 Nov 2025). HiE is even lighter: it performs no optimization loop and simply reweights fine-class posterior mass by the probability of each leaf’s parent class, yielding hierarchy-aware post-hoc correction (Jain et al., 2023).

A third family performs test-time training or adaptation in parameter space. SCOPE belongs to test-time reinforcement learning: it samples multiple reasoning trajectories, derives subgroup-specific pseudo-labels using step-wise confidence-weighted voting, and updates the model with GRPO during benchmark-specific test-time training (Wang et al., 17 Dec 2025). MGTTA learns a Meta Gradient Generator that transforms noisy unsupervised TTA gradients into refined update directions, with a Gradient Memory Layer storing historical gradient information in parameters (Deng et al., 2024). Speech TTT adapts a masked-autoencoder encoder on each test utterance using masked reconstruction loss and shows that BitFit-style bias updates are more stable than full encoder finetuning (Dumpala et al., 2023). “Test-Time Training Provably Improves Transformers as In-context Learners” studies supervised one-step weight updates on in-context examples and provides a full linear-transformer analysis of when such updates help under distribution shift (Gozeten et al., 14 Mar 2025). M-L2O similarly meta-trains a learned optimizer initialization specifically to support rapid per-task self-adaptation in only a few steps (Yang et al., 2023). LLMdoctor occupies an intermediate position: the patient model stays frozen, but a smaller doctor model is trained offline to provide token-level preference guidance during decoding, so alignment is realized at test time without updating the large model (Shen et al., 15 Jan 2026).

4. Representative domains

In compositional generation, fine-grained control is motivated by local failure modes. “No Concept Left Behind” treats objects, attributes, and relations as separate semantic requirements, computing $S_{\text{global}}(I,P)=\text{CLIP}(I,P)$ alongside per-concept scores $s_i(I,c_i)=\text{CLIP}(I,c_i)$, and optimizes the combined objective $$S_{\text{global}}(I,P)+\frac{1}{k}\sum_i s_i(I,c_i)$$ through iterative prompt search rather than retraining (Sameti et al., 27 Sep 2025). This shifts T2I test-time optimization from a global similarity problem to a concept-aware compositional coverage problem.

In reasoning and alignment, the granularity of the optimization target varies widely. DORA argues that the right primitive is the latent reasoning direction rather than the individual trajectory (Wang et al., 30 May 2025). VG-Search argues that the right primitive can be the verifier call interval $g$, not merely the final answer or each single step (Chen et al., 16 May 2025). SCOPE argues that majority-voted final answers are too coarse for pseudo-reward construction and replaces them with step-wise confidence and subgroup-local consensus (Wang et al., 17 Dec 2025). GeoSolver goes further and trains a token-level process reward model, GeoPRM, on Geo-PRM-2M so that reasoning search can be guided by dense faithfulness signals instead of outcome-only rewards (Sun et al., 10 Mar 2026). LLMdoctor similarly criticizes trajectory-level preference signals and instead extracts sparse token-level rewards from positive/negative behavioral variants of the patient model, training a doctor model with Token-Level Flow-Guided Preference Optimization (Shen et al., 15 Jan 2026). TSAN stays entirely in textual output space but still pursues fine-grainedness by comparing multiple candidate responses on different aspects and synthesizing their complementary strengths (Mo et al., 10 Nov 2025).

In vision, speech, and fine-grained recognition, the granularity often lies in prompts, regions, or parameter subsets. CLIP-AUTT personalizes emotion recognition by adapting AU prompt embeddings on a selected low-entropy temporal window from a single unseen-subject video, then resetting before the next video (Zeeshan et al., 30 Mar 2026). A-TPT uses refined gradient attention rollout to identify semantically meaningful regions that survive adversarial perturbation and uses those regions to control both prompt tuning and multi-view inference (Hai et al., 19 May 2026). Speech TTT treats each test utterance as its own adaptation problem and shows that updating only encoder biases can be preferable to full adaptation under distribution shift (Dumpala et al., 2023). HiE performs hierarchy-aware fine-grained classification correction by multiplying leaf probabilities with parent probabilities at inference time (Jain et al., 2023). For in-context learning, one-step supervised test-time training can trade task-local adaptation for shorter context, reducing the sample size needed for tabular classification by a factor reported as 3 to 5 (Gozeten et al., 14 Mar 2025).

5. Empirical patterns and reported gains

The empirical record is heterogeneous, but several representative results illustrate the range of benefits attributed to fine-grained test-time optimization.

System	Reported result	Setting
“No Concept Left Behind” (Sameti et al., 27 Sep 2025)	GPT4-o improves from 0.717 and 0.744 to 0.810 on T2I CompBench; from 0.719 and 0.765 to 0.827 on DrawBench	FLUX vs MILS vs concept-aware TTO
DORA (Wang et al., 30 May 2025)	Higher accuracy using only 64 rollouts than REBASE at 256 rollouts, with a reported $3.5\times$ reduction in total FLOPs and $N$0 lower inference latency	MATH500, Llama-3.2-1B-Instruct
VG-Search (Chen et al., 16 May 2025)	Adaptive strategies achieve accuracy gains of up to 3.1% over Beam Search and 3.6% over Best-of-N, while reducing FLOPs by over 52%	MATH-500
SCOPE (Wang et al., 17 Dec 2025)	Relative improvements of 13.1% on AIME 2025 and 8.1% on AMC	Qwen3-8B
GeoSolver (Sun et al., 10 Mar 2026)	Self-Consistency gives +4.1%, Best-of-N + GeoPRM gives +15.8%, Beam Search + GeoPRM gives +17.5%	Average gain at generation budget 32
Expert-Sample (Chen et al., 2 Feb 2026)	Pass@32 rises from 85.4% to 91.9%, and accuracy improves from 59.1% to 62.6% with Best-of-N verification	Qwen3-30B-A3B-Instruct on GPQA-Diamond
A-TPT (Hai et al., 19 May 2026)	Average adversarial accuracy 45.7 vs 39.9 for R-TPT on ViT-B/16; clean accuracy 63.0 vs 61.1	Eight fine-grained datasets
MGTTA (Deng et al., 2024)	7.4% accuracy improvement and 4.2 times faster adaptation speed on ImageNet-C	Compared with SAR
TabPFN + TTT (Gozeten et al., 14 Mar 2025)	3 to 5 times fewer samples	Tabular classification

Several cross-paper patterns are notable. First, gains are often largest where the task is explicitly compositional, hard, or diversity-limited: counting, spatial relations, color binding, difficult math reasoning, or fine-grained adversarial recognition (Sameti et al., 27 Sep 2025, Wang et al., 17 Dec 2025, Hai et al., 19 May 2026). Second, many papers emphasize that finer-grained control can improve not just the metric being directly optimized. Concept-aware T2I optimization improves VQA, caption-based alignment, GPT-4o score, and human preference rather than merely CLIP score (Sameti et al., 27 Sep 2025); GeoPRM-based reranking improves not only final accuracy but also search efficiency and cross-model transfer (Sun et al., 10 Mar 2026). Third, compute-aware methods repeatedly argue for better use of budget rather than more budget. DORA, VG-Search, and UniScale all frame the main problem as compute allocation under fixed or constrained inference resources (Wang et al., 30 May 2025, Chen et al., 16 May 2025, Huang et al., 29 May 2026).

This suggests a recurring empirical thesis: fine-grained test-time optimization tends to help when the dominant error is local, redundant, or structurally heterogeneous, and when a coarse global control signal would otherwise smooth over the very distinction that matters.

6. Limitations, misconceptions, and open directions

A first misconception is that fine-grained test-time optimization is synonymous with gradient-based test-time adaptation. The surveyed work includes parameter updates, but also prompt rewriting, routing perturbation, verifier scheduling, reward reconstruction, post-hoc score correction, and online serving policies (Jain et al., 2023, Sameti et al., 27 Sep 2025, Chen et al., 2 Feb 2026, Huang et al., 29 May 2026). A second misconception is that “fine-grained” always means token-level. Some methods operate at the levels of concepts, subgroups, reasoning directions, temporal windows, AU prompts, or verification intervals instead (Sameti et al., 27 Sep 2025, Wang et al., 30 May 2025, Wang et al., 17 Dec 2025, Zeeshan et al., 30 Mar 2026, Chen et al., 16 May 2025).

The limitations are correspondingly diverse. Concept-aware T2I optimization depends on concept extraction quality and CLIP-based semantic judgments (Sameti et al., 27 Sep 2025). DORA’s guarantees rely on assumptions about identifiable directions, equal PRM score within a direction, and effective inverse-cluster-size correction from embedding similarity (Wang et al., 30 May 2025). SCOPE still assigns final-answer-level binary rewards, uses heuristic newline-based step segmentation, and makes subgroup size dynamic rather than subgroup content semantic (Wang et al., 17 Dec 2025). VG-Search adapts $N$1 per task condition rather than online within a single reasoning trajectory (Chen et al., 16 May 2025). GeoSolver requires substantial domain-specific data synthesis and still does not perform true test-time parameter adaptation (Sun et al., 10 Mar 2026). Expert-Sample requires access to internal router scores and is specific to fine-grained MoE architectures (Chen et al., 2 Feb 2026). TSAN remains expensive at inference and its textual attention is metaphorical rather than a normalized numeric kernel (Mo et al., 10 Nov 2025).

Parameter-adaptation methods introduce their own costs. MGTTA needs offline pretraining of the optimizer and updates only a restricted parameter subset at deployment (Deng et al., 2024). Speech TTT remains less scalable than ordinary inference even when BitFit makes batching feasible (Dumpala et al., 2023). Test-time training for ICL depends on labeled in-context demonstrations and a differentiable update path (Gozeten et al., 14 Mar 2025). M-L2O assumes access to optimizee objectives and gradients, and its theory uses strong regularity assumptions (Yang et al., 2023). CLIP-AUTT uses a simple entropy objective on one selected window and the paper does not fully specify the separate test-time update schedule (Zeeshan et al., 30 Mar 2026). A-TPT is not entirely immune to adversarial attacks and pays extra cost for attention extraction over multiple views (Hai et al., 19 May 2026).

The open questions follow directly from these constraints. Several papers explicitly call for richer dependency modeling than flat candidates or fixed granularity, such as dynamic confidence schedules, intra-trajectory verifier adaptation, stronger multimodal scorers, richer process rewards, or learned direction representations (Wang et al., 30 May 2025, Chen et al., 16 May 2025, Sun et al., 10 Mar 2026). Others point toward better integration of structure and efficiency: finer-grained reward signals without unstable dense supervision, more precise but cheaper localization, and online controllers that can adapt granularity continuously rather than by validation-time table lookup (Wang et al., 17 Dec 2025, Huang et al., 29 May 2026). Taken together, the literature suggests that the next phase of fine-grained test-time optimization will likely center on matching the unit of intervention to the unit of error while retaining compute efficiency, verifier reliability, and deployment practicality.