Caterpillar of Thoughts: Optimal Rewinding for LLMs

• Caterpillar of Thoughts (CaT) is a test-time computation algorithm that formalizes LLM inference as adaptive exploration of a finite Markov chain with rewinding.
• It establishes that a minimal, non-branching backtracking strategy—yielding a 'caterpillar' spine—is sufficient to achieve optimal inference with exponential token gains.
• Practical implementations replace the exact oracle with a softmax value estimator, improving success rates and reducing token consumption on reasoning benchmarks.

Searching arXiv for the main paper and closely related test-time reasoning work. Caterpillar of Thoughts (CaT) is a test-time computation algorithm for LLMs introduced in "Caterpillar of Thoughts: The Optimal Test-Time Algorithm for LLMs" (Azarmehr et al., 24 Mar 2026). It formalizes inference-time reasoning as adaptive exploration of a finite, fully observable Markov chain with rewinding, in which the algorithm may resume generation from any previously observed state. Within that model, CaT is characterized as an optimal rewinding strategy: although backtracking can reduce the number of generations exponentially, the paper proves that only a very limited form of backtracking is theoretically sufficient. The resulting exploration tree is a caterpillar: if the leaves are removed, the remaining structure is a single path (Azarmehr et al., 24 Mar 2026).

1. Position within test-time computation

CaT addresses a general phenomenon in LLM inference: models can often produce substantially better outputs when allowed additional test-time computation, including sampling, chain of thought, backtracking, or revising partial solutions (Azarmehr et al., 24 Mar 2026). The paper treats these procedures not as ad hoc prompting heuristics but as algorithmic policies over partial generations. This reframing is intended to answer two questions that the abstract identifies as under-theorized: how inference-time computation should be structured, and what constitutes an optimal use of a fixed computation budget (Azarmehr et al., 24 Mar 2026).

Within this perspective, existing methods such as Chain-of-Thought (CoT) prompting (Wei et al., 2022) and Tree-of-Thoughts (ToT) (Yao et al., 2023) can be seen as specific algorithms in the same rewinding model (Azarmehr et al., 24 Mar 2026). The paper also places Best-of-$k$ in that family (Azarmehr et al., 24 Mar 2026). A common misconception in this area is that stronger test-time reasoning necessarily requires increasingly elaborate branching search. Under the paper’s formal assumptions, that is not the case: no complex branching is ever needed, because the optimal exploration tree is non-branching in a precise sense (Azarmehr et al., 24 Mar 2026).

2. Formal model: Markov chains with rewinding

The formal object is a finite Markov chain $M=(\Omega,P)$, where each state $x \in \Omega$ corresponds to a partial solution, such as a token prefix, and $P(x,y)$ is the probability that the LLM extends $x$ into $y$ in one step, interpreted as one token or one block of tokens (Azarmehr et al., 24 Mar 2026).

A rewinding algorithm is defined adaptively. At step $t$, the algorithm chooses any previously visited state $x_{t'}$ with $0 \le t' \le t$, rewinds to it, and then the chain takes one transition from $x_{t'}$ to $M=(\Omega,P)$0 according to $M=(\Omega,P)$1. The parent relation is written as $M=(\Omega,P)$2 (Azarmehr et al., 24 Mar 2026). This differs from a standard Markov chain trajectory, in which states are drawn passively without intervention.

The optimization target is a hitting-time objective. Given an initial state $M=(\Omega,P)$3 and a designated target $M=(\Omega,P)$4, the goal is to design a rewinding strategy that minimizes the expected total number of transitions until $M=(\Omega,P)$5 is observed (Azarmehr et al., 24 Mar 2026). The notation is as follows: if $M=(\Omega,P)$6 is the history and $M=(\Omega,P)$7 is the set of distinct observed states, then $M=(\Omega,P)$8 denotes the optimal expected remaining steps to $M=(\Omega,P)$9 starting from the set $x \in \Omega$0; in particular, $x \in \Omega$1 (Azarmehr et al., 24 Mar 2026). The theory minimizes expectation rather than enforcing a strict hard cap, although the summary notes that one may also impose a hard budget $x \in \Omega$2 on the total number of token generations (Azarmehr et al., 24 Mar 2026).

A central recursion is given in Corollary 2.4. For any $x \in \Omega$3, define

$x \in \Omega$4

Then

$x \in \Omega$5

The intended interpretation is explicit: from $x \in \Omega$6, one keeps sampling one step until a strictly “better” state $x \in \Omega$7 is reached, and then pays its remaining cost (Azarmehr et al., 24 Mar 2026).

3. Structural theorem: the caterpillar characterization

The paper’s main structural claim is stated through the notion of a non-branching algorithm. A rewinding strategy is non-branching if, at each step $x \in \Omega$8, it only ever rewinds to the most recent state $x \in \Omega$9 or its parent $P(x,y)$0. Equivalently, the exploration tree has exactly one spine and leaves only at distance at most $P(x,y)$1 from it (Azarmehr et al., 24 Mar 2026).

Theorem 2.5 states that, given the initial state $P(x,y)$2, Algorithm 1 (CaT) is a non-branching rewinding strategy and reaches $P(x,y)$3 in expected $P(x,y)$4 steps, i.e. optimally (Azarmehr et al., 24 Mar 2026). The theorem is supported by three ingredients given in the proof sketch. First, Lemma 2.2 shows that $P(x,y)$5 depends only on the observed set $P(x,y)$6, not on the tree shape. Second, Theorem 2.3 proves by induction that for any observed set $P(x,y)$7 there is some $P(x,y)$8 that achieves $P(x,y)$9 under a non-branching strategy; equivalently,

$x$0

Third, the optimal policy is therefore to track the current minimizer $x$1, sample single steps from $x$2 until a state $x$3 with $x$4 is found, and then update $x$5 (Azarmehr et al., 24 Mar 2026).

Together, these statements yield the paper’s defining geometric image. If the leaves of the generated state tree are removed, what remains is a path. The root-to-leaf path is the “thought spine,” and any side child is pruned immediately if it does not improve the value (Azarmehr et al., 24 Mar 2026). This suggests that, under the Markov-chain-with-rewinding model, the essential role of test-time computation is not to maintain a broad search frontier, but to identify a sequence of strictly improving states.

4. Algorithmic form and computational properties

Algorithm 1, the exact form of CaT, is succinct. It takes as input an initial state $x$6, a target $x$7, and an oracle $x$8. It initializes $x$9, repeatedly draws one successor $y$0, and replaces $y$1 by $y$2 only if $y$3. The loop terminates when $y$4 (Azarmehr et al., 24 Mar 2026). The exact algorithm therefore requires access to the optimal value function.

Because exact $y$5 is not available in practice, the paper replaces that oracle with an estimator $y$6, such as a learned critic or LLM score, inside a softmax-rewinding subroutine (Algorithm 5) (Azarmehr et al., 24 Mar 2026). This introduces a small amount of branching for robustness, but the backbone remains a caterpillar (Azarmehr et al., 24 Mar 2026). The practical guidance specifies that parent states are sampled from the observed set $y$7 with probability proportional to

$y$8

with temperature $y$9 reported as working well (Azarmehr et al., 24 Mar 2026).

The paper’s illustrative example considers a linear chain $t$0 with “death” arcs of probability $t$1 leading to an absorbing sink (Azarmehr et al., 24 Mar 2026). Without rewinding, one almost surely dies. CaT instead rewinds to $t$2 until it samples $t$3 in expected $t$4 steps, then shifts the spine to $t$5 and repeats, for a total expected cost of $t$6 steps to reach $t$7 (Azarmehr et al., 24 Mar 2026).

A stronger separation appears in Lemma 2.6. In the chain of Figure 1 with $t$8, any non-rewinding or limited-backtracking strategy needs $t$9 samples to hit $x_{t'}$0 with constant probability, whereas CaT uses only $x_{t'}$1 samples in expectation (Azarmehr et al., 24 Mar 2026). The paper therefore attributes exponential gains to rewinding.

The finite-state value function $x_{t'}$2 can itself be computed exactly. Algorithm 2 is described as “Dijkstra-like” and computes $x_{t'}$3 exactly in $x_{t'}$4 or $x_{t'}$5 time by repeatedly adding to $x_{t'}$6 the next state with minimal provisional distance

$x_{t'}$7

This is a dynamic-programming or shortest-path-like computation over the Markov chain (Azarmehr et al., 24 Mar 2026).

5. Empirical results on reasoning benchmarks

The empirical comparison is against Tree-of-Thoughts, using GPT-4 as the base LLM and identical prompting for child generation and evaluation (Azarmehr et al., 24 Mar 2026). The reported implementation uses the softmax variant of CaT rather than the exact oracle-based form.

On the Game of 24 benchmark, the task is to build an expression evaluating to $x_{t'}$8 from four numbers by sequentially picking two numbers and an operator (Azarmehr et al., 24 Mar 2026). The evaluation uses 100 “hard” games, indices 901–1000, and reports success rate and average total tokens, defined as prompt plus generated tokens (Azarmehr et al., 24 Mar 2026). With a budget of 15 iterations and best of 2 runs, ToT (beam 5, 15 iters) attains a success rate of 74 with average tokens 19.2k, whereas CaT (best of 2, 15 iters) attains a success rate of 81 with average tokens 15.3k (Azarmehr et al., 24 Mar 2026). Figure 2 is described as showing a roughly log-linear correlation between number of steps and number of valid expressions generated, illustrating CaT’s efficiency (Azarmehr et al., 24 Mar 2026).

On 5×5 Crosswords, the task is to fill a 5×5 crossword grid from 10 clues, consisting of 5 across and 5 down clues (Azarmehr et al., 24 Mar 2026). The dataset comprises 20 held-out puzzles from GooBix, and the reported metrics are word accuracy, letter accuracy, puzzle success, and average tokens (Azarmehr et al., 24 Mar 2026). At 20 iterations, ToT achieves word accuracy 39.5, letter accuracy 64.8, games solved 5, and average tokens 73.4k, while CaT achieves word accuracy 50.0, letter accuracy 68.6, games solved 15, and average tokens 66.8k (Azarmehr et al., 24 Mar 2026).

The ablation notes identify several trade-offs. Softmax rewinding introduces occasional branching for robustness, with hyperparameters including temperature $x_{t'}$9 and number of children per parent (Azarmehr et al., 24 Mar 2026). Best-of-2 runs mitigate being stuck on a suboptimal spine at small extra cost (Azarmehr et al., 24 Mar 2026). The summary reports that, empirically, CaT both raises success rates and reduces token consumption by approximately 20–30% (Azarmehr et al., 24 Mar 2026).

6. Robustness, implementation guidance, and limitations

The robustness analysis distinguishes between statistical and adversarial noise in the value estimates. Under Laplace-noise on $0 \le t' \le t$0, defined in Definition 4.1, the paper gives a stable variant, Algorithms 3–4, that attains a $0 \le t' \le t$1-approximation to the optimal hitting time at cost polynomial in $0 \le t' \le t$2 (Azarmehr et al., 24 Mar 2026). By contrast, Theorem 4.5 shows that adversarial multiplicative noise can force exponential blowup, so some statistical assumption is necessary (Azarmehr et al., 24 Mar 2026). In other words, the non-branching optimality claim is exact when $0 \le t' \le t$3 is known, but approximate inference depends critically on the quality of $0 \le t' \le t$4.

The practical guidance follows directly from that dependency. For the oracle, the summary recommends prompting the LLM, or a small verifier, to score the likelihood of success from a partial state $0 \le t' \le t$5 and then using the negative log-score as $0 \le t' \le t$6 (Azarmehr et al., 24 Mar 2026). For softmax rewinding, it recommends sampling parent states from $0 \le t' \le t$7 with probability proportional to $0 \le t' \le t$8, with $0 \le t' \le t$9 (Azarmehr et al., 24 Mar 2026). For the iteration budget, it suggests fixing $x_{t'}$0–$x_{t'}$1 for medium-difficulty tasks and using Best-of-$x_{t'}$2 with $x_{t'}$3–$x_{t'}$4 to improve robustness (Azarmehr et al., 24 Mar 2026). For child proposals, it recommends sampling only a few continuations, $x_{t'}$5–$x_{t'}$6, from the selected parent rather than using a large beam (Azarmehr et al., 24 Mar 2026).

The limitations are explicit. The accuracy of $x_{t'}$7 matters: overly noisy estimates can still mislead the spine, although softmax rewinding and multiple runs help (Azarmehr et al., 24 Mar 2026). Very deep or highly branching problems may require larger $x_{t'}$8 or hybrid search, for example combining CaT with local tree expansions (Azarmehr et al., 24 Mar 2026). This suggests that CaT is best understood not as a universal replacement for all search procedures, but as a theorem-backed statement about the minimal branching structure required for optimality under the paper’s rewinding model (Azarmehr et al., 24 Mar 2026).