Hybrid Test-Time Scaling

• Hybrid Test-Time Scaling is an inference strategy that combines parallel sampling with sequential refinement, improving performance without altering base model parameters.
• It employs adaptive budget allocation and verifier-guided methods to efficiently distribute compute across sub-problems and reasoning steps.
• The approach is applied across domains—from math to code and multimodal tasks—utilizing self-consistency, memory augmentation, and diverse aggregation techniques.

Searching arXiv for papers on hybrid test-time scaling and closely related test-time scaling frameworks. Searching for the core paper on scaling plateaus and hybrid implications. Searching for hybrid/adaptive test-time scaling papers on self-consistency, selective allocation, and verifier-guided methods. Hybrid test-time scaling denotes inference-time schemes that combine more than one scaling mechanism in order to improve performance without changing the base model parameters. Across recent work, the term covers combinations of parallel sampling and sequential refinement, fast and slow reasoning modes, per-instance budget estimation and within-instance selective allocation, heterogeneous expert aggregation, verifier-guided search, augmentation plus adaptation, and multi-agent or latent-memory inference loops (Wang et al., 26 May 2025, Ji et al., 12 Nov 2025, Xiao et al., 29 Nov 2025, Li et al., 20 Feb 2025, Lyu et al., 8 Dec 2025, Kaya et al., 3 Oct 2025, Wu et al., 11 May 2026, Zhang et al., 29 Sep 2025). The unifying theme is that inference compute is treated as a structured resource: not merely “more tokens” or “more samples,” but a budget to be allocated across modes, steps, sub-problems, trajectories, tools, or experts.

1. Conceptual scope and main axes of hybridization

Hybrid test-time scaling emerged from the observation that the canonical forms of test-time scaling—parallel scaling and sequential scaling—capture only part of the available design space. Parallel scaling samples multiple trajectories or candidates and aggregates them; sequential scaling repeatedly refines or rethinks a single trajectory. Recent systems combine these primitives with routing, verification, search, memory, or system-level specialization, thereby turning inference into a compositional process rather than a single decoding routine (Wang et al., 26 May 2025, Chang et al., 21 Jul 2025).

A useful way to organize the literature is by the axis along which hybridization occurs. Some methods hybridize compute modes, combining parallel exploration with sequential exploitation. Some hybridize reasoning systems, typically a fast “System 1” and a slow “System 2.” Some hybridize granularities, allocating compute per query, per sub-problem, per reasoning step, or per token block. Others hybridize diversity sources, such as architecture heterogeneity and random initialization. Still others hybridize timescales, pairing fast per-instance adaptation with slower cross-instance consolidation (Ji et al., 12 Nov 2025, Xiao et al., 29 Nov 2025, Lyu et al., 8 Dec 2025, Zhang et al., 29 Sep 2025).

Hybrid pattern	Representative mechanism	Paper
Parallel + sequential reasoning	Sampling plus refinement or search	(Wang et al., 26 May 2025, Li et al., 20 Feb 2025, Chang et al., 21 Jul 2025)
System 1 + System 2	Cheap uncertainty estimation plus expensive reasoning	(Ji et al., 12 Nov 2025, Xiao et al., 29 Nov 2025)
Step/sub-problem selectivity	Difficulty-aware routing within one solution	(Xiao et al., 29 Nov 2025, Chang et al., 21 Jul 2025)
Diversity-source fusion	Multi-seed and multi-architecture aggregation	(Lyu et al., 8 Dec 2025)
Ensemble + adaptation	Augmentation followed by test-time adaptation	(Kaya et al., 3 Oct 2025)
Episodic + parametric memory	Retrieval plus consolidation in latent space	(Zhang et al., 29 Sep 2025, Wu et al., 11 May 2026)

This taxonomy suggests that “hybrid” is not a single algorithmic template. Rather, it is an umbrella for inference policies that mix multiple sources of improvement while explicitly managing budget, error propagation, and diversity.

2. Probabilistic foundations and saturation behavior

A central formal result comes from the Test-Time Scaling Performance Model (TTSPM), which shows that parallel scaling and sequential scaling share the same success-probability form under simple probabilistic assumptions. For parallel scaling, with independent candidate correctness probability $p_{\text{sample}}$, the probability of at least one correct answer among $N$ samples is

$1 - (1-p_{\text{sample}})^N.$

For sequential scaling, modeling rethinking as a two-state Markov process with per-round correction probability $p_{\text{rethink}}$, the probability of success within $N$ rounds is likewise

$1 - (1-p_{\text{rethink}})^N.$

TTSPM abstracts both as

$F(N) = F_{\max}\big(1-(1-p_x)^N\big),$

with marginal gain

$\Delta F(N)=F(N+1)-F(N)=F_{\max}p_x(1-p_x)^N,$

and saturation point

$$N^*=\left\lceil \frac{\ln\left(\frac{\epsilon}{F_{\max}\cdot p_x}\right)}{\ln(1-p_x)} \right\rceil.$$

This yields a canonical exponential saturation curve and makes diminishing returns explicit (Wang et al., 26 May 2025).

The same paper argues that this shared structure is the key mathematical rationale for hybrid test-time scaling. The paper does not explicitly derive a hybrid model, but the common form strongly suggests that a hybrid policy can often be approximated in terms of an effective per-unit success probability $p_{\text{hyb}}$ and an effective number of trials $N$0, with performance behaving like

$N$1

This suggests that hybrid design is fundamentally a problem of improving the quality of each “chance” to solve the task while avoiding post-saturation compute (Wang et al., 26 May 2025).

A complementary formalization appears in bandit-based strategic compute allocation. There, the objective is to choose per-query budgets $N$2 under a global budget $N$3:

$N$4

Under a Bernoulli success model with per-query success probability $N$5, the paper shows that adaptive algorithms require

$N$6

whereas uniform allocation requires

$N$7

This provides a theoretical justification for hybrid policies that vary compute across queries rather than using a fixed budget everywhere (Zuo et al., 15 Jun 2025).

3. Budget allocation, stopping rules, and adaptive scheduling

One major branch of hybrid test-time scaling centers on advance budget estimation. Seer Self-Consistency (SeerSC) divides inference into a cheap System 1 and an expensive System 2. System 1 samples $N$8 direct answers, groups them into answer categories, computes category weights

$N$9

normalizes them to $1 - (1-p_{\text{sample}})^N.$0, and then computes confidence-weighted entropy

$1 - (1-p_{\text{sample}})^N.$1

The entropy is mapped to a budget

$1 - (1-p_{\text{sample}})^N.$2

with $1 - (1-p_{\text{sample}})^N.$3 and $1 - (1-p_{\text{sample}})^N.$4. Because $1 - (1-p_{\text{sample}})^N.$5 is known before expensive reasoning starts, all System 2 chains can be launched in parallel. In the reported experiments, SeerSC achieves up to a 47% reduction in token consumption and a 43% reduction in inference latency without significant performance loss (Ji et al., 12 Nov 2025).

A different line of work casts adaptive allocation as online decision-making. The bandit framework of "Strategic Scaling of Test-Time Compute" treats queries as arms, uses elimination to stop allocating to solved queries, and augments this with UCB, gap-based, or entropy-based exploration. The empirical result is that the same total budget can be redistributed toward hard-but-solvable instances: the method reports up to an 11.10% performance improvement on MATH-500 and up to a 7.41% performance improvement on LiveCodeBench relative to uniform allocation (Zuo et al., 15 Jun 2025).

In agentic settings, budget awareness must be defined in wall-clock time rather than token count. "Timely Machine" formalizes total time as

$1 - (1-p_{\text{sample}})^N.$6

and analyzes regimes in which tool latency dominates, generation dominates, or both are comparable. This redefinition matters when tool calls decouple reasoning length from real latency. The reported findings are that smaller models excel with fast feedback through more interactions, while larger models dominate high-latency settings via superior interaction quality, and that existing models generally fail to adapt their reasoning length to time budgets without dedicated training (Ma et al., 23 Jan 2026).

Taken together, these results recast hybrid test-time scaling as a scheduling problem. The budget can be allocated per query, per latency regime, or in advance through cheap uncertainty signals, but the common objective is to spend expensive reasoning only where its marginal value remains positive.

4. Within-query hybrid execution: steps, sub-problems, and verifier-guided pipelines

A second major branch hybridizes computation inside a single solution trajectory. SCALE is the clearest sub-problem-level example. It decomposes a problem $1 - (1-p_{\text{sample}})^N.$7 into sub-problems

$1 - (1-p_{\text{sample}})^N.$8

assigns each sub-problem a difficulty score

$1 - (1-p_{\text{sample}})^N.$9

routes it to System 1 or System 2 via

$p_{\text{rethink}}$0

and executes sequentially with context propagation. The framework is explicitly motivated by the claim that uniform resource allocation across all reasoning sub-problems creates bottlenecks and overthinking. On AIME25, the reported headline result is an accuracy improvement of up to 13.75 percentage points, from 57.50% to 71.25%, while reducing computational costs by 33%-53% (Xiao et al., 29 Nov 2025).

Step-level hybridization appears in "Step-level Verifier-guided Hybrid Test-Time Scaling for LLMs." There the unit of control is not the whole solution but each reasoning step, segmented by \n\n. A process reward model scores partial solutions, step-level self-refinement is triggered conditionally, and parallel expansion is handled with a PUCT-style score

$p_{\text{rethink}}$1

The final hybrid pipeline combines Best-of-$p_{\text{rethink}}$2, MCTS-style exploration, and step-level self-refinement under PRM guidance. On GPQA Diamond, Qwen2.5-3B-Instruct rises from 30.3 at baseline to 51.5 under the hybrid setting, and the paper emphasizes that training-free hybrid TTS can rival or exceed RL-enhanced baselines in some settings (Chang et al., 21 Jul 2025).

Code generation introduces a different within-query hybridization pattern. S* begins with parallel sampling, applies sequential iterative debugging with execution feedback for each sample, and then performs adaptive selection by generating distinguishing test inputs for candidate pairs and grounding judgments in actual execution results rather than predicted outputs. With $p_{\text{rethink}}$3 samples and $p_{\text{rethink}}$4 debugging rounds as the default configuration, the paper reports that DeepSeek-R1-Distill-Qwen-32B with S* achieves 85.7 on LiveCodeBench, and that GPT-4o-mini with S* outperforms o1-preview by 3.7% on the same benchmark (Li et al., 20 Feb 2025).

Another verifier-centered formulation appears in budget-aware discriminative verification. Instead of expensive generative verifiers, it uses a small scalar verifier with Bradley–Terry training and combines it with self-consistency through Weighted Self-Consistency

$p_{\text{rethink}}$5

and Pessimistic Verification

$p_{\text{rethink}}$6

The core empirical claim is budget-sensitive: under a fixed compute budget, the hybrid discriminative approach surpasses state-of-the-art generative verification by up to 15.3% on AIME2025, while discriminative verification itself is only a small fraction of total generation cost (Montgomery et al., 16 Oct 2025).

These systems show that hybrid test-time scaling is increasingly implemented not merely as extra sampling, but as structured control over internal units of reasoning: sub-problems, steps, candidate programs, and answer clusters.

5. Cross-domain extensions and systems-level realizations

Beyond math and code, hybrid test-time scaling has expanded into recommendation, multimodal reasoning, agentic systems, latent control, and specialized inference backbones. In large-scale recommendation, prediction merging provides a non-generative analogue of hybrid scaling: multiple experts produce

$p_{\text{rethink}}$7

with diversity supplied by architecture heterogeneity and random initialization. The paper reports that under the same inference budget, test-time scaling can outperform parameter scaling, and that prediction merging is naturally parallel across servers without increasing user-perceived latency (Lyu et al., 8 Dec 2025).

For small vision-LLMs, hybridization takes the form of ensemble-style augmentation plus adaptation. TTAug generates multiple augmented inputs and aggregates token distributions by

$p_{\text{rethink}}$8

while TTAdapt turns the TTAug consensus into pseudolabels and performs test-time adaptation. On SmolVLM2-2.2B, the paper reports baseline mean performance of about 43.8 over nine tasks, TTAug around 46–48, and TTAdapt sometimes higher still; on an A100, TTAug with $p_{\text{rethink}}$9 moves peak memory from 4.6 GB to 8.75 GB and runtime from 1.43 s to 4.77 s per query (Kaya et al., 3 Oct 2025).

Multi-agent and memory-augmented systems push hybridization further. TMAS organizes inference around solution, verification, summary, experience, and guideline agents, with an experience bank for low-level reusable conclusions and a guideline bank for high-level explored strategies. On IMO-AnswerBench-50, Qwen3-30B under TMAS reaches 40.50 at It19, while on the 4B model Hybrid-RL improves It19 performance from 17.06 to 30.88, indicating that memory and hybrid reward shaping can materially improve iterative scaling (Wu et al., 11 May 2026). LatentEvolve offers a different fast/slow hybrid: daytime scaling performs retrieval-guided latent optimization for each query, while nighttime scaling consolidates latent optimizations into a parametric latent weaver. The reported result is that LatentEvolve surpasses LatentSeek and TTRL by up to 13.33% while showing cross-domain and cross-backbone generalization (Zhang et al., 29 Sep 2025).

Systems-level co-design also matters. Falcon-H1R couples a hybrid Transformer–Mamba backbone with DeepConf-based parallel chain-of-thought filtering. Under DeepConf@512 on AIME25, Falcon-H1R-7B reaches 96.7 at 95.1M tokens, whereas Qwen3-32B reaches 86.7 at 174.8M tokens, illustrating that architecture and inference policy can be jointly optimized for large-scale parallel TTS (Team et al., 5 Jan 2026). In block diffusion LLMs, hybrid scaling appears as Bounded Adaptive Confidence Decoding (BACD) plus Think Coarse, Critic Fine (TCCF): BACD adapts token unmasking using bounded confidence thresholds, and TCCF uses large blocks for exploratory reasoning and smaller blocks for refinement. Applied to TDAR-8B, the paper reports a 2.26x speedup and +11.2 points on AIME24 over TraDo-8B (Lu et al., 10 Feb 2026).

These cross-domain variants demonstrate that hybrid test-time scaling is no longer tied to one modality or one inference primitive. The same design logic—multiple compute modes, explicit budget control, and structured coordination—reappears in recommendation, VLMs, diffusion decoders, agentic time budgets, and memory-augmented reasoning.

6. Evaluation, limitations, and recurrent failure modes

As hybrid TTS diversified, evaluation itself became a research problem. ARISE addresses this by scoring scaling behavior at the sample level:

$N$0

where

$N$1

A correct-to-incorrect transition receives a penalty

$N$2

so negative scaling is explicitly penalized. In the reported results, Claude Opus 4.1 reaches ARISE values of 0.4529 on AIME, 0.4709 on HMMT, 0.4454 on GPQA-Diamond, and 0.4932 on MMLU-Pro, whereas GPT-OSS-20B is negative on all four, including -0.4030 on AIME (Yin et al., 7 Oct 2025). This matters because hybrid systems can improve average accuracy while still degrading some samples; ARISE is designed to surface precisely that pathology.

A related failure mode is strategy-selection bias. The theoretical and empirical analysis of TTS-Uniform argues that naive sampling frequently overconcentrates on one reasoning strategy, and that this can help or harm depending on whether the dominant strategy is low-complexity or high-complexity. TTS-Uniform extracts strategies, allocates budget uniformly across them, and filters unstable strategies via answer entropy. On AIME2025, GPT-4o-mini improves from 3.3% under Self-Consistency Acc@32 to 20.0% under TTS-Uniform variants, while GPT-4.1-mini improves from 56.6% to 70.0% under Uniform-C (F) (Wu et al., 22 Sep 2025). This indicates that some apparent “scaling limits” are in fact failures of exploration policy rather than failures of raw compute.

Across the literature, several limitations recur. TTSPM assumes independence of scaling units and stationary $N$3, assumptions that may break under correlated samples, context fatigue, or strong verifier-guided search (Wang et al., 26 May 2025). SeerSC notes that entropy over answer categories is straightforward in math-like domains but harder in code or open-ended text, where answer clustering is nontrivial (Ji et al., 12 Nov 2025). SCALE depends on the quality of decomposition and difficulty estimation; bad decompositions can misallocate compute (Xiao et al., 29 Nov 2025). Timely Machine shows that token count is insufficient in tool-using environments, where wall-clock time and latency variability dominate (Ma et al., 23 Jan 2026).

A plausible implication is that future hybrid TTS systems will need three things simultaneously: better per-instance budget estimation, stronger local reliability signals for routing or verification, and evaluation metrics that detect not only average improvement but also negative scaling and wasted compute. Recent work already points in that direction, but it also indicates that hybrid test-time scaling is constrained by calibration, answer canonicalization, verifier quality, and the granularity at which compute is allocated.