Caterpillar of Thoughts: The Optimal Test-Time Algorithm for Large Language Models

Abstract: LLMs can often produce substantially better outputs when allowed to use additional test-time computation, such as sampling, chain of thought, backtracking, or revising partial solutions. Despite the growing empirical success of such techniques, there is limited theoretical understanding of how inference time computation should be structured, or what constitutes an optimal use of a fixed computation budget. We model test-time computation as an algorithm interacting with a Markov chain: at any point, the algorithm may resume generation from any previously observed state. That is, unlike standard Markov chains where the states are drawn passively, we allow the algorithm to backtrack to any previously observed state of the Markov chain at any time. Many of the existing test-time algorithms, such as Chain-of-Thought (CoT) (Wei et al., 2023), Tree-of-Thoughts (ToT) (Yao et al., 2023), or Best-of-$k$ (Brown et al., 2024) could be seen as specific algorithms in this model. We prove that while backtracking can reduce the number of generations exponentially, a very limited form of backtracking is theoretically sufficient. Namely, we show that the optimal algorithm always generates a caterpillar tree. That is, if we remove the leaves of the state tree generated by the optimal algorithm, we obtain a path. Motivated by our characterization of the optimal algorithm, we present Caterpillar of Thoughts (CaT), a new test-time computation algorithm, reducing the number of token/state generations. Our empirical evaluation shows that CaT, compared to ToT, achieves a better success rate while also reducing the number of token generations.

• The paper demonstrates that optimal test-time inference for LLMs is achieved through a caterpillar tree strategy that leverages selective rewinding to minimize token usage.
• It introduces a recursive formulation of expected hitting time O(x) and employs noisy Laplacian approximations to efficiently navigate large Markov chains.
• Empirical evaluations on Game of 24 and crosswords confirm that the CaT algorithm improves success rates and reduces token consumption compared to beam-based methods.

Caterpillar of Thoughts: Optimal Test-Time Algorithms for LLMs

Problem Formulation and Theoretical Framework

"Caterpillar of Thoughts: The Optimal Test-Time Algorithm for LLMs" (2603.22784) rigorously investigates the optimal deployment of test-time computation for LLMs. The problem is formalized as a search over a fully observable Markov chain, where each state is a partial solution and transitions correspond to the stochastic generation of new tokens. Unlike standard Markovian inference, the algorithm may rewind to any previously observed state, enabling flexible exploration strategies. This abstraction captures a wide spectrum of test-time methods, including Chain-of-Thought (CoT), Tree-of-Thoughts (ToT), self-consistency, iterative refinement, and verifier-guided backtracking.

The authors demonstrate that allowing arbitrary rewinding can reduce the expected steps to reach a target state exponentially compared to non-rewinding methods. However, they then prove that the optimal solution generates a "caterpillar" tree: all rewinds and expansions occur along a single path, with leaves attached at distance 1, obviating the need for full tree expansion or arbitrary branching. Formally, the optimal algorithm always rewinds either to the minimizer state (the current best partial solution) or to the most recent state, maintaining only two states at any time. They derive recursive formulas for the optimal expected hitting time to the target, $O(x)$, for any state $x$.

Algorithmic Contributions

The principal algorithm, Caterpillar of Thoughts (CaT), operationalizes this theoretical insight. At each iteration, CaT rewinds to the state whose $O(x)$ is minimal among observed states and continually samples from it until a "better" child is found, i.e., a descendant with strictly lower $O(g) < O(x^*)$. This minimizes the expected number of steps to the solution and ensures all observed states are within distance 1 of the central path.

When $O(x)$ cannot be computed exactly—an inevitable scenario for massive Markov chains induced by real-world LLMs—the authors propose using noisy Laplacian approximations and demonstrate that polynomially many steps suffice for efficient search. They also prove a sharp lower bound: if estimators of $O(x)$ are adversarially (rather than randomly) noisy, any algorithm requires exponentially many steps, ruling out robust polynomial-time solutions without probabilistic guarantees.

Empirical Evaluation

The authors empirically validate CaT on two established reasoning benchmarks: the Game of 24 (arithmetic-based sequence construction) and $5\times5$ crosswords (structured combinatorial completion). For both tasks, CaT is implemented with softmax-based rewinding, using model-estimated $O(x)$ values as scores, and compared against beam-based ToT.

On Game of 24, CaT improves success rates from 74% (ToT) to 81% while reducing average token usage by more than 20%. On crosswords, CaT exceeds truncated ToT across word, letter, and game-level accuracy, despite a matched token budget. These results empirically support the claim that caterpillar expansion, guided by accurate or stochastic evaluations of partial solution quality, is markedly more efficient in token generation and solution discovery.

Figure 1: Log-linear relationship between the number of valid expressions and number of steps in Game of 24, illustrating efficient scaling by CaT.

Theoretical Implications

The primary theoretical result—that the optimal search tree is always a caterpillar—contrasts with prevalent approaches favoring broader tree or graph-based explorations. By showing that depth-oriented, non-branching rewinding is universally optimal under fully observable transition probability regimes, the paper overturns the heuristic basis for many current test-time computation strategies. The recursive characterization of $O(x)$ supplies a concrete metric for decision-making at inference-time, which, if tractably estimated, can guide practical algorithms in both symbolic and generative reasoning contexts.

Moreover, the exponential separation between rewinding and non-rewinding search underscores the critical importance of state revisitation. Algorithms constrained to breadth-first or fixed-depth strategies incur infeasible computational costs in problems with deep solution states or low transition success probabilities, as detailed in the theoretical lower bounds.

Practical and Future Directions

Practically, CaT enables substantial reductions in inference compute and generative steps, directly translating to lower cost and faster solution times in resource-constrained deployments. The framework is agnostic to the underlying LLM; any model with an accessible transition function and evaluative oracles can instantiate CaT.

Theoretical extensions include the study of partially observable or adversarially perturbed Markov chains, where Laplacian noise models facilitate robust polynomial algorithms but adversarial noise demands exponential search. This delineates the boundary between tractable and intractable test-time optimization, suggesting that future progress depends on improving estimator reliability, perhaps via meta-evaluation or auxiliary verification.

One open avenue is exploiting structured domain knowledge, such as symbolic state decompositions or graph-based priors, to further improve estimator accuracy for $O(x)$ and thereby accelerate caterpillar-based reasoning. Integration with verifier-guided backtracking schemes, as discussed in the comparison section, implies broad compatibility with existing techniques. The empirical superiority of CaT across arithmetic and combinatorial reasoning hints at generalizability to planning, program synthesis, and multi-step logical inference tasks.

Conclusion

"Caterpillar of Thoughts" (2603.22784) rigorously resolves the structure of optimal test-time computation for LLMs in terms of caterpillar tree expansion. Empirical and theoretical evidence demonstrates that depth-focused, minimal-branch rewinding strategies, grounded in recursive evaluation of expected hitting time, far outperform breadth-first or fixed-trajectory methods under a fixed compute budget. The framework not only illuminates principled algorithmic foundations for inference-time search but also sets the stage for future developments in sample-efficient and robust reasoning for both current and next-generation LLMs.