Long-Term Multipath Decoding for LLM Inference

• Long-Term Multipath (LTM) decoding is a novel inference strategy for LLMs that uses a dynamic tree search mechanism to explore multiple reasoning paths.
• It evaluates full sequence probabilities with long-range scoring, allowing the model to recover from local errors and select globally coherent outputs.
• Empirical results demonstrate significant accuracy gains on benchmarks like GSM8K and HumanEval, with efficient integration into self-correction frameworks.

Long-Term Multipath (LTM) decoding is a novel inference strategy for LLMs designed to address the “short-sightedness” of conventional next-token prediction. Unlike standard autoregressive decoding techniques such as greedy decoding, beam search, or nucleus sampling—which make token-level decisions based on immediate likelihoods—LTM views decoding as a dynamic tree search maintaining multiple partial hypotheses, evaluating them using long-range sequence scores, and pruning only those paths that fall below a tunable cumulative probability threshold. This approach enables systematic exploration of multiple reasoning trajectories, allowing the model to recover from local missteps and select globally coherent and correct outputs over the entire sequence (Li et al., 9 Sep 2025).

1. Formal Framework and Algorithm

Let $x$ denote the input prompt, and $s_i = (y_0, …, y_i)$ represent a partial decoded sequence of length $i$. Under an autoregressive model $M$, the probability of $s_i$ is

$P(s_i) = \prod_{k=0}^i P(y_k \mid y_{0:k-1}, x).$

Standard decoding methods select $y_{i+1}$ to maximize $P(y_{i+1} \mid s_i, x)$ or maintain a fixed number of beams with the highest $P(s_{i+1})$. LTM, in contrast, maintains a variable-width set of $k_i$ candidates at each timestep, ensuring the retained sequences together cover at least a fraction $s_i = (y_0, …, y_i)$0 of the total probability mass, and enforces an upper bound $s_i = (y_0, …, y_i)$1. At each step, every survivor expands to all possible vocabularies $s_i = (y_0, …, y_i)$2, and candidate sequences are sorted by their cumulative probability. The minimal $s_i = (y_0, …, y_i)$3 satisfying

$s_i = (y_0, …, y_i)$4

is chosen. Final hypotheses are selected by lowest perplexity,

$s_i = (y_0, …, y_i)$5

Key components:

• Path generation: Each $s_i = (y_0, …, y_i)$6 partial sequence spawns $s_i = (y_0, …, y_i)$7 children by appending every possible next token.
• Delayed evaluation: Scoring is performed on the entire partial sequence, deferring pruning until after expansion.
• Trajectory selection: Selected by probability mass threshold $s_i = (y_0, …, y_i)$8 and capped by $s_i = (y_0, …, y_i)$9.

2. Algorithmic Procedure

The LTM algorithm proceeds as follows:

$M$6

This procedure ensures systematic exploration of paths, with dynamic pruning and adaptive allocation of computational resources.

3. Mathematical Properties and Complexity

The LTM scoring function for a sequence $i$0 is its full joint probability,

$i$1

or equivalently, its perplexity $i$2. At step $i$3, expansion and sorting of $i$4 candidates are necessary; with the cap $i$5, per-step complexity becomes $i$6. $i$7 tends to be small when token distributions are peaky, but can grow in regions of high uncertainty—precisely when long-range reasoning is most valuable.

When used within Feedback-Triggered Regeneration (FTR), LTM decoding is only applied to outputs flagged as negative. If $i$8 is the fraction requiring regeneration and each call averages beam width $i$9, total inference time is $M$0, empirically measured at $M$1–$M$2 baseline for common benchmarks, versus $M$3 for naïve two-pass schemes.

4. Comparative Analysis with Traditional Decoding

LTM decoding is contrasted with several established strategies:

Method	Beam/Tuning	GSM8K Llama2-7B	GSM8K Llama2-13B	Remarks
Greedy Decoding	N/A	24.3%	(not given)	Baseline
Beam Search	width tuned	26.1%		Modest improvement
Nucleus/adaptive	tuned per model	25.7–26.1%		Comparable to beam search
LTM Decoding	dynamic	27.6%		+1.5% absolute over best baseline

LTM’s dynamic beam width focuses computation where output uncertainty is greatest, and long-range scoring allows recovery from local optima. Trade-offs include increased per-step cost in flat distributions and greater implementation complexity (Li et al., 9 Sep 2025).

5. Empirical Performance and Evaluation

Integrated with FTR, LTM decoding achieves substantial improvements on mathematical reasoning (GSM8K, MultiArith) and code generation (HumanEval) benchmarks across a spectrum of LLMs: Llama2-7B, Llama2-13B, Llama3-1B, Llama3-3B, Qwen-1.5B, and Qwen-3B. Under ground-truth feedback (“Protocol 1”) on Llama2-7B:

• Initial zero-shot GSM8K accuracy: 20.6%
• • Critic Prompt: 17.1%
• • IoE Prompt: 13.6%
• • FTR (with LTM): 36.0%

Absolute gains of 10–20% are observed on other tasks and larger models, for both simulated and ground-truth feedback regimes. This demonstrates that deep multipath search, applied at the correction stage, systematically enhances logical/mathematical reasoning and code-generation pass rates (Li et al., 9 Sep 2025).

6. Case Study: MultiArith and Error Recovery

In a representative MultiArith instance, traditional beam search (width 3) prematurely prunes a low-probability token that leads to the correct answer, retaining beams that become dead-ends in subsequent steps. LTM with $M$4 expands to temporary width $M$5, preserving the low-probability yet promising trajectory and ultimately recovering the solution. This ability to look ahead and flexibly increase beam width underpins LTM’s advantage in complex, multi-step reasoning scenarios.

7. Integration within Self-Correction Frameworks

LTM serves as a core component of Feedback-Triggered Regeneration (FTR), where it is activated only following negative user (or simulated) feedback. This design avoids blanket recomputation, preserves correct initial outputs, and focuses recomputation on genuinely problematic responses. FTR + LTM is demonstrated to be more efficient than naïve re-decoding, with empirical inference times 1.3×–3.9× those of vanilla inference, as opposed to a fixed 2× increase for two-pass self-correction approaches. This efficiency—combined with superior accuracy—distinguishes the approach within the self-correction literature for LLMs.

In summary, Long-Term Multipath decoding provides a principled, adaptive tree-search mechanism for LLM inference, prioritizing long-range sequence quality and efficiently allocating computational effort. Its empirical superiority and modular integration with feedback-driven correction frameworks make it a substantive advancement in the decoding methodology landscape for LLMs (Li et al., 9 Sep 2025).