Thought-Level Beam Search

• Thought-Level Beam Search is an inference method that expands and ranks meaningful reasoning units—such as deductions, equations, tool calls, or proof steps—instead of individual tokens.
• The approach applies verifier-guided scoring, process rewards, lookahead, best-first ordering, and adaptive prune-and-branch scheduling to preserve promising reasoning trajectories while controlling generation and KV-cache costs.
• Applications include mathematical problem solving, deductive reasoning, structured modeling, and uncertainty estimation, with reported systems reducing token use by up to 68.5% and improving accuracy over standard sampling and pruning baselines.

Thought-Level Beam Search (TLBS) is a family of inference procedures that performs beam search over semantically meaningful reasoning units rather than individual tokens. A thought-level hypothesis is a partial reasoning trajectory containing intermediate deductions, subgoals, equations, tool calls, proof steps, modeling components, or other expandable states. At each reasoning stage, multiple candidate continuations are generated, evaluated, and selectively retained. The central objective is to allocate finite test-time computation among partial trajectories: promising prefixes receive additional continuations, while unpromising trajectories are pruned. Recent formulations include verifier-guided deductive search, process-reward-guided search, lookahead and best-first variants, anytime scheduling, and dynamic prune-and-branch allocation under KV-cache constraints (Yang et al., 8 Aug 2026).

1. Conceptual foundations

Conventional beam search operates on token prefixes. Given a beam of partial sequences, each hypothesis is expanded by candidate tokens, scored using cumulative log probability, and reduced to a fixed number of survivors. TLBS changes the search granularity. A single expansion may generate a complete next thought, a subgoal, an equation transformation, a tool invocation and result, a proof lemma, or a modeling stage. The resulting search tree is therefore organized by semantic reasoning depth rather than token position.

A reasoning state can be represented as

$z_t=(x,r_{1:t},\sigma_t),$

where $x$ is the input, $r_{1:t}$ is the sequence of generated thoughts, and $\sigma_t$ may contain scratchpad variables, derived equations, tool outputs, current subgoals, or verifier state. A candidate action $a_t$ induces a transition

$z_{t+1}=F(z_t,a_t).$

At thought depth $t$, the beam is

$B_t=\{z_t^{(1)},\ldots,z_t^{(k)}\}.$

All beam members are expanded, producing

$A_{B_t}=\bigcup_{z\in B_t}\mathcal A(z),$

after which a scorer ranks the candidates and retains the top $k$:

$x$0

The explicit dependence of the scorer on $x$1 distinguishes a beam-aware policy from independent scoring of isolated trajectories. The current beam determines which hypotheses are expanded, which candidates compete, how redundancy is assessed, and which future reasoning distribution is encountered.

The distinction between token-level and thought-level search is substantive. Token-level search provides fine-grained control but may preserve syntactically similar trajectories and cannot readily evaluate whether an individual token is a valid logical consequence. Thought-level search permits step-level evaluation, but thought boundaries are ambiguous, candidate branching is larger, and individual thoughts have variable token lengths and computational costs. A thought-level search can nevertheless contain an inner token decoder that realizes each proposed thought.

2. Learning beam-aware reasoning policies

A foundational formulation treats beam search itself as part of the learned decision process rather than as post-processing applied after training. In “Learning Beam Search Policies via Imitation Learning,” a beam is a state in a secondary search space, and imitation learning trains a policy on the beams generated by the learner rather than solely on ideal prefixes (Negrinho et al., 2018).

For a structured search graph $x$2, each node is a partial output and each terminal node is a complete output. The optimal completion cost of a partial node is

$x$3

where $x$4 is the set of terminal descendants. For a beam $x$5, the beam cost is the cost of its best member:

$x$6

The beam transition cost is

$x$7

A transition has positive cost when the successor beam has discarded every lowest-cost continuation available from the current beam. The terminal cost telescopes across a beam trajectory, which permits local surrogate losses to be collected at visited beams.

The learned policy does not directly select an arbitrary beam as an atomic action. It scores all candidate expansions

$x$8

and retains the top $x$9 candidates:

$r_{1:t}$0

This mechanism makes beam management a ranking problem. Candidate losses include perceptron-first and perceptron-last losses, margin-last loss, cost-sensitive margin loss, beam log loss, neighbor log loss, convex cost-sensitive upper bounds, and weighted pairwise ranking losses. The convex upper bound is particularly significant because it upper-bounds the beam transition cost and is convex in the score vector; for a linear scoring model, it remains convex in the parameters.

The training algorithm has two independent design choices: a surrogate loss and a roll-in strategy. Oracle roll-in generates clean states but creates distribution mismatch. Stop terminates training after the first cost-increasing transition and corresponds to early-update behavior in relevant settings. Reset replaces a failed beam with an oracle state and corresponds to LaSO. Continue allows the learner to proceed through its own errors and is analogous to DAgger.

The associated no-regret analysis is learner-induced: the mixture of policies is compared with the best fixed policy evaluated on the state distribution generated by the learner’s own trajectories. If the surrogate upper-bounds expected transition cost, the average terminal cost is bounded by the initial beam cost, the best-in-class surrogate loss, and the online learner’s regret. These results are conditional on the surrogate assumptions and do not establish global optimality of beam search or correctness of reasoning traces.

For TLBS, this framework implies that training should expose the scorer to corrupted or self-generated reasoning beams rather than only to gold prefixes. A process-aware oracle or cost function is required. Possible costs include outcome error, verifier-based constraint violations, process error, or estimates obtained by a stronger solver, exhaustive solver, verifier-guided planner, or sampled completions. Exact best-completion cost is generally unavailable for open-ended reasoning, making verifier quality and cost estimation central limitations.

3. Verifier-guided and process-aware search

Deductive Beam Search (DBS) provides a direct instantiation of TLBS for chain-of-thought reasoning. It decomposes a rationale into variable-length reasoning steps, samples multiple next steps from every surviving rationale, evaluates their deducibility, and retains the highest-scoring trajectories (Zhu et al., 2024).

For a question $r_{1:t}$1 and rationale

$r_{1:t}$2

the LLM generates one reasoning step conditioned on the question and preceding steps:

$r_{1:t}$3

With beam size $r_{1:t}$4 and $r_{1:t}$5 samples per beam, each iteration produces $r_{1:t}$6 candidate trajectories. A separately trained verifier receives the premises and a candidate step and assigns a deductive score

$r_{1:t}$7

DBS uses a DeBERTa-v3-large NLI-style classifier as its verifier. Candidate trajectories are ranked by cumulative deductive score and reduced to the top $r_{1:t}$8. The generator proposes alternatives; the verifier determines which proposed steps are sufficiently grounded in their premises.

The motivation is error accumulation. A high-probability but invalid intermediate thought can cause all subsequent steps to be locally plausible while globally incorrect. DBS attempts to remove such trajectories at the point where the error is introduced rather than relying solely on final-answer aggregation. It differs from self-consistency, which generates complete rationales and aggregates final answers, and from best-of-$r_{1:t}$9, which selects among completed candidates. DBS repeatedly evaluates intermediate reasoning states.

The verifier is trained using a two-stage labor-free construction procedure. First, gold rationales are corrupted through grounding errors, logic errors, and irrelevant-step insertions. Second, a preliminary verifier and Llama2-7B generate hard negatives. The final verifier is trained with a margin-ranking objective. On the reported evaluation, the verifier achieved accuracy $\sigma_t$0, precision $\sigma_t$1, recall $\sigma_t$2, and F1 $\sigma_t$3. These classification metrics do not guarantee calibrated ranking among near-duplicate candidate steps.

DBS was evaluated on arithmetic, commonsense, and symbolic reasoning datasets with Llama2-7B, Llama2-13B, Llama2-70B, and ChatGPT. Its standard configuration used $\sigma_t$4 and $\sigma_t$5. On GSM8K, reported DBS improvements across model scales were approximately 7.6–16.6 percentage points. For example, Llama2-7B increased from 22.0 under greedy decoding to 31.2 with DBS, while ChatGPT increased from 68.8 to 75.9. DBS combined with self-consistency was generally strongest. The method incurs greater generation and verification cost than greedy decoding, but it can prune invalid prefixes before they generate complete rationales.

BPP-Search applies a related process-aware beam procedure to mathematical modeling. It searches over structured modeling stages rather than free-form token sequences, following the progression

$\sigma_t$6

or the grouped experimental structure

$\sigma_t$7

A policy LLM proposes modeling continuations, a Process Reward Model scores intermediate partial models, and a Preference Model ranks final candidates pairwise (Wang et al., 2024). The experimental beam width is primarily $\sigma_t$8, with an ablation at $\sigma_t$9; each node has at most three children. BPP-Search is principally an inference-time heuristic search-and-ranking method: although its motivation involves reinforcement learning and process supervision, the reported implementation does not contain a policy-gradient or PPO-style policy-optimization objective.

On the reported solvable subsets of StructuredOR, MAMO-ComplexLP, and NL4OPT, BPP-Search achieved correct rates of 0.933, 0.722, and 0.804, respectively, using 15, 21, and 15 reasoning steps. Its structured representation includes sets, parameters, variables, objectives, and constraints, enabling intermediate and final validation through model structure and solver-derived objective values. The results also show that increasing beam width is not uniformly beneficial because PRM scores may be imprecise and nearly identical candidates may include both correct and incorrect trajectories.

4. Lookahead, best-first, and anytime variants

Thought-level beam search need not expand all hypotheses synchronously. Best-First Beam Search changes the ordering of expansion while preserving the standard beam result under monotonic scoring (Meister et al., 2020). Instead of prioritizing shorter hypotheses first, it uses a priority queue ordered by current score. At most $a_t$0 hypotheses of each length are evaluated, and cross-length pruning removes hypotheses that cannot displace established candidates.

For additive log-probability scoring, sequence scores are monotonic because each token log probability is nonpositive. Under this condition, an unfinished hypothesis cannot later improve its current score. Best-First Beam Search can therefore stop when the first complete hypothesis is popped, while returning the same $a_t$1-optimal output as standard beam search. Its full implementation uses a two-level data structure: a priority queue over length-specific beams and a min-max queue within each beam. The full method reduces scoring calls in practice, although it may require $a_t$2 memory instead of standard beam search’s $a_t$3 active-hypothesis memory. A memory-reduced version imposes a total queue capacity and sacrifices exact equivalence for lower memory use.

Lookahead Beam Search (LBS) introduces a separate search-depth variable. For a candidate prefix, it augments the current score with the best continuation over the next $a_t$4 transitions. LBS-0 is ordinary beam search; sufficiently large lookahead becomes exhaustive MAP search (Jinnai et al., 2023). The method therefore defines a continuum between local beam decisions and global model-score optimization. Experiments on machine translation found that LBS generally outperformed vanilla beam search, with LBS-2 usually the strongest setting. LBS-3 often reduced the gain, while exhaustive MAP decoding performed extremely poorly on task metrics despite optimizing model likelihood.

The result illustrates a distinction between search error and model bias. Deeper lookahead reduces search error relative to the model objective but may expose undesirable biases toward short outputs, repetitions, premature termination, or locally predictable wording. The reported results associate intermediate lookahead with a useful balance between these effects. Exact LBS is computationally expensive: LBS-1 has rough per-step cost $a_t$5, whereas the proposed Lookbehind Heuristic Beam Search has ordinary beam-search cost $a_t$6 and uses previous-step information as a proxy for future quality.

For TLBS, lookahead depth should be interpreted as the number of future thought transitions, not the number of generated tokens. A two-thought lookahead may involve many tokens if each thought is long, while ten token transitions may cover only part of one semantic reasoning step. Lookahead is particularly useful when local thought quality is a poor predictor of eventual correctness, but exact expansion becomes infeasible as the thought branching factor grows.

Rectangle Search supplies a different extension: an anytime schedule that repeatedly explores alternatives at multiple depth levels rather than committing to a single fixed-width progression (Lemons et al., 2023). It maintains separate priority queues for each depth, revisits earlier levels, introduces deeper levels incrementally, and updates an incumbent whenever a goal is found. The resulting exploration pattern is rectangular or square-like. Its aspect parameter controls the depth-versus-reconsideration trade-off: small values favor repeated reconsideration of earlier decisions, while large values produce deeper and thinner probes.

With an admissible heuristic, Rectangle Search is complete and returns an optimal final solution after termination. Intermediate solutions are interruptible but need not satisfy a prescribed suboptimality bound. The method is designed to avoid deep heuristic craters in which best-first search spends excessive effort in a misleading region. In TLBS, a node can be a partial reasoning trace, a goal can be a verifier-accepted answer, and a depth queue can contain alternative traces with the same number of thought steps. These guarantees do not transfer automatically because language-model scores, verifier scores, and progress estimates are generally noisy and non-admissible.

5. Dynamic compute allocation with Gambit

The most direct recent formulation of TLBS treats inference as constrained compute allocation. “Thought-Level Beam Search for Reasoning” introduces Gambit, which periodically evaluates partial reasoning traces, prunes low-scoring trajectories, and branches high-scoring prefixes while maintaining a fixed logical population of active traces (Yang et al., 8 Aug 2026).

A trajectory is

$a_t$7

where thought boundaries are identified using double-newline delimiters. A lightweight hidden-state scorer maps the last-layer state at each thought boundary to a quality estimate:

$a_t$8

Gambit uses the running average

$a_t$9

This score is a proxy for the probability that a partial trajectory will eventually yield a correct answer, not a token log-likelihood.

The scheduler distinguishes a logical tree from the physically resident KV-cache traces. Let $z_{t+1}=F(z_t,a_t).$0 be the logical capacity, $z_{t+1}=F(z_t,a_t).$1 the swap size, $z_{t+1}=F(z_t,a_t).$2 the tournament interval, and $z_{t+1}=F(z_t,a_t).$3 the warmup threshold. At a tournament, eligible traces are ranked by $z_{t+1}=F(z_t,a_t).$4. If the active population is below capacity, Gambit branches from the best eligible parents. If it is at capacity, it removes the bottom $z_{t+1}=F(z_t,a_t).$5 traces and branches the top $z_{t+1}=F(z_t,a_t).$6 eligible traces:

$z_{t+1}=F(z_t,a_t).$7

This is a prune-and-branch swap rather than subtractive pruning. A child inherits the parent’s prefix KV cache, so only the continuation after the branch point must be generated. A temperature multiplier may be applied to children to preserve diversity.

The logical–physical separation addresses a cache-management failure mode. A physically evicted trace becomes a ghost trace: it retains logical membership in the search tree but holds no physical KV cache and generates no tokens. Tournament decisions are made using the logical population, preventing cache eviction from being misinterpreted as under-capacity and causing repeated branching from only the top few traces.

The reported standard configuration is

$z_{t+1}=F(z_t,a_t).$8

with a hard-floor memory parameter $z_{t+1}=F(z_t,a_t).$9. Experiments used Qwen3-4B-Thinking-2507, DeepSeek-R1-0528-Qwen3-8B, and Phi-4-reasoning-plus-14B on AIME 2025, AIME 2026, HMMT 2024, HMMT 2025, and GPQA-Diamond, using one NVIDIA B300 GPU and vLLM.

Compared with standard parallel sampling, Gambit reduced generated tokens by up to 68.5%, achieved more than twice the productive trace-completion throughput, and improved accuracy over pruning baselines by up to 6.7 percentage points on HMMT-24 and 3.3 points on AIME-25. On Qwen3-4B, token reductions relative to SC@256 were 49.0% on AIME-25, 40.7% on AIME-26, 60.6% on HMMT-24, 49.1% on HMMT-25, and 19.7% on GPQA. The paper reports that scoring and tree-management overhead accounted for approximately 0.97% of profiled runtime, with GPU forward passes and sampling dominating execution.

Gambit’s runtime example illustrates the purpose of branching. A child inherited 31,404 cached tokens from a parent and generated only 3,649 new tokens before correcting a late reasoning error. The parent produced 203, while the child corrected the answer to 204. This kind of repair is unavailable to purely subtractive pruning when the useful prefix is retained but additional continuations are not generated.

6. Uncertainty, diversity, and evaluation

Beam search can be used not only to produce an answer but also to construct a structured candidate set for uncertainty estimation. “Don’t Throw Away Your Beams” replaces independent multinomial generations with probability-weighted beam candidates for consistency-based uncertainty quantification (Fadeeva et al., 10 Dec 2025). The paper operates at the token-sequence level rather than explicit thought level, but its probability-coverage analysis is relevant to TLBS.

For completed beam candidates $t$0, the paper normalizes their sequence probabilities within the beam:

$t$1

The beam mass is

$t$2

The normalized weights describe the distribution conditional on membership in the beam, whereas $t$3 measures how much of the original model distribution the beam covers. A beam-weighted dissimilarity estimator is

$t$4

where $t$5 is semantic similarity. The method also defines beam-weighted eccentricity, eigenvector dissimilarity, CoCoA, pairwise dissimilarity, and semantic entropy.

The theoretical comparison frames beam estimation as a bias–variance trade-off. Multinomial sampling is unbiased for the full model expectation but has variance $t$6. Beam estimation removes sampling variance but introduces truncation bias from ignoring probability mass outside the beam. Under the stated assumptions, the beam estimator has lower mean-squared error when

$t$7

A distribution-free sufficient condition is

$t$8

For $t$9, this requires approximately $B_t=\{z_t^{(1)},\ldots,z_t^{(k)}\}.$0. The result concerns a bounded consistency statistic and does not prove that beam concentration implies reasoning correctness. A beam can contain highly consistent but jointly incorrect trajectories, particularly when all hypotheses share a systematic early error.

Diversity must therefore be defined semantically rather than merely lexically. Multiple thought trajectories may differ in wording while expressing the same inference, or may be lexically distinct while sharing an invalid premise. Suitable evaluation may include semantic clustering, entailment, structured canonicalization, verifier agreement, and diversity across decompositions or strategies. Probability weighting can prevent many low-probability variants from dominating an uncertainty estimate, but it can also suppress low-probability yet correct alternatives.

A comprehensive TLBS evaluation should separate task correctness from model score, verifier score, and search behavior. Relevant criteria include answer accuracy, best-of-beam accuracy, survival of teacher- or verifier-approved trajectories, cost-increase frequency, process validity, semantic diversity, generated tokens, model calls, verifier calls, latency, KV-cache use, calibration, and robustness to initially incorrect or misleading states. For dynamic allocators, productive trace-completion throughput and GPU utilization are also relevant.

7. Limitations and research directions

TLBS depends on a representation in which reasoning states are meaningfully expandable and evaluable. Thought segmentation is difficult: semantic units have variable token lengths, may contain multiple implicit operations, and may depend on non-Markovian context. A fixed-depth formalism therefore requires explicit termination, a maximum thought count, or an expanded state representation.

Oracle supervision is another unresolved issue. Exact best-completion costs are generally intractable. Learned verifiers can be weak, biased, or overconfident, and a noisy scorer may amplify rather than suppress incorrect reasoning. In deductive search, a verifier may reject a valid but unconventional inference. In mathematical modeling, a process reward model may score structurally plausible but objectively incorrect models. In Gambit, active branching can concentrate computation on confidently wrong prefixes.

Non-monotonic objectives weaken safe pruning. Length normalization, delayed verifier rewards, tool outcomes, proof completion, and final-answer checks can increase a partial trajectory’s score after extension. Best-first or branch-and-bound pruning is mathematically safe only when future improvement is bounded by a valid optimistic estimate. Otherwise, prioritization may remain useful, but exact equivalence with standard beam search and optimality guarantees are lost.

Beam width and search depth are distinct controls. Width determines how many alternatives remain at a frontier; lookahead depth determines how many future transitions influence a pruning decision; token budget determines the amount of generated text and computation. Larger width or deeper lookahead is not uniformly better because broader search can reduce search error while increasing exposure to model bias, verifier error, shortness bias, or reward hacking.

The principal systems challenge is computational allocation. Thought-level branching may require many language-model calls, verifier evaluations, tool executions, and KV-cache blocks. Prefix sharing reduces recomputation, but long inherited contexts remain expensive. Memory-reduced best-first methods and logical–physical separation address different parts of this problem: the former sacrifices search exactness for bounded memory, while the latter maintains a stable logical population despite physical cache eviction.

Future TLBS systems are likely to combine several mechanisms: learner-induced training trajectories, process or deductive verification, finite adaptive lookahead, best-first ordering, rectangle-style reconsideration of earlier decisions, semantic deduplication, residual sampling, and dynamic prune-and-branch allocation. The common abstraction is

$B_t=\{z_t^{(1)},\ldots,z_t^{(k)}\}.$1

The unresolved question is not whether a larger search can be performed, but how computation should be allocated among partial reasoning states whose quality is uncertain, whose rewards may be delayed, and whose shared prefixes may contain both reusable progress and unrecoverable errors.