---
title: 'Caterpillar of Thoughts: Optimal Rewinding for LLMs'
url: https://www.emergentmind.com/topics/caterpillar-of-thoughts-cat
type: topic
---

# Caterpillar of Thoughts: Optimal Rewinding for LLMs

Searching arXiv for the main paper and closely related test-time reasoning work.
Caterpillar of Thoughts (CaT) is a test-time computation algorithm for large language models (LLMs) introduced in "Caterpillar of Thoughts: The Optimal Test-Time Algorithm for Large Language Models" [2603.22784]. It formalizes inference-time reasoning as adaptive exploration of a finite, fully observable Markov chain with rewinding, in which the algorithm may resume generation from any previously observed state. Within that model, CaT is characterized as an optimal rewinding strategy: although backtracking can reduce the number of generations exponentially, the paper proves that only a very limited form of backtracking is theoretically sufficient. The resulting exploration tree is a *caterpillar*: if the leaves are removed, the remaining structure is a single path [2603.22784].

## 1. Position within test-time computation

CaT addresses a general phenomenon in LLM inference: models can often produce substantially better outputs when allowed additional test-time computation, including sampling, chain of thought, backtracking, or revising partial solutions [2603.22784]. The paper treats these procedures not as ad hoc prompting heuristics but as algorithmic policies over partial generations. This reframing is intended to answer two questions that the abstract identifies as under-theorized: how inference-time computation should be structured, and what constitutes an optimal use of a fixed computation budget [2603.22784].

Within this perspective, existing methods such as Chain-of-Thought (CoT) prompting [2201.11903] and Tree-of-Thoughts (ToT) [2305.10601] can be seen as specific algorithms in the same rewinding model [2603.22784]. The paper also places Best-of-$k$ in that family [2603.22784]. A common misconception in this area is that stronger test-time reasoning necessarily requires increasingly elaborate branching search. Under the paper’s formal assumptions, that is not the case: no complex branching is ever needed, because the optimal exploration tree is non-branching in a precise sense [2603.22784].

## 2. Formal model: Markov chains with rewinding

The formal object is a finite Markov chain $M=(\Omega,P)$, where each state $x \in \Omega$ corresponds to a partial solution, such as a token prefix, and $P(x,y)$ is the probability that the LLM extends $x$ into $y$ in one step, interpreted as one token or one block of tokens [2603.22784].

A rewinding algorithm is defined adaptively. At step $t$, the algorithm chooses any previously visited state $x_{t'}$ with $0 \le t' \le t$, rewinds to it, and then the chain takes one transition from $x_{t'}$ to $x_{t+1}$ according to $P(x_{t'},\cdot)$. The parent relation is written as $p(x_{t+1})=x_{t'}$ [2603.22784]. This differs from a standard Markov chain trajectory, in which states are drawn passively without intervention.

The optimization target is a hitting-time objective. Given an initial state $x_0$ and a designated target $z \in \Omega$, the goal is to design a rewinding strategy that minimizes the expected total number of transitions until $z$ is observed [2603.22784]. The notation is as follows: if $X=(x_0,\ldots,x_t)$ is the history and $S=\mathrm{distinct}(X)$ is the set of distinct observed states, then $\mathcal O(S)$ denotes the optimal expected remaining steps to $z$ starting from the set $S$; in particular, $\mathcal O(\{x\})=\mathcal O(x)$ [2603.22784]. The theory minimizes expectation rather than enforcing a strict hard cap, although the summary notes that one may also impose a hard budget $B$ on the total number of token generations [2603.22784].

A central recursion is given in Corollary 2.4. For any $x \ne z$, define
$$
L(x)=\{y : P(x,y)>0 \text{ and } \mathcal O(y)<\mathcal O(x)\}.
$$
Then
$$
\mathcal O(x)=\frac{1+\sum_{y \in L(x)} P(x,y)\cdot \mathcal O(y)}{P(x,L(x))}.
$$
The intended interpretation is explicit: from $x$, one keeps sampling one step until a strictly “better” state $y \in L(x)$ is reached, and then pays its remaining cost [2603.22784].

## 3. Structural theorem: the caterpillar characterization

The paper’s main structural claim is stated through the notion of a non-branching algorithm. A rewinding strategy is non-branching if, at each step $t$, it only ever rewinds to the most recent state $x_t$ or its parent $p(x_t)$. Equivalently, the exploration tree has exactly one spine and leaves only at distance at most $1$ from it [2603.22784].

Theorem 2.5 states that, given the initial state $x_0$, Algorithm 1 (CaT) is a non-branching rewinding strategy and reaches $z$ in expected $\mathcal O(x_0)$ steps, i.e. optimally [2603.22784]. The theorem is supported by three ingredients given in the proof sketch. First, Lemma 2.2 shows that $\mathcal O(S)$ depends only on the observed set $S$, not on the tree shape. Second, Theorem 2.3 proves by induction that for any observed set $S$ there is some $x \in S$ that achieves $\mathcal O(S)=\mathcal O(x)$ under a non-branching strategy; equivalently,
$$
\mathcal O(S)=\min_{x \in S}\mathcal O(x).
$$
Third, the optimal policy is therefore to track the current minimizer $x^*$, sample single steps from $x^*$ until a state $g$ with $\mathcal O(g)<\mathcal O(x^*)$ is found, and then update $x^* \leftarrow g$ [2603.22784].

Together, these statements yield the paper’s defining geometric image. If the leaves of the generated state tree are removed, what remains is a path. The root-to-leaf path is the “thought spine,” and any side child is pruned immediately if it does not improve the value [2603.22784]. This suggests that, under the Markov-chain-with-rewinding model, the essential role of test-time computation is not to maintain a broad search frontier, but to identify a sequence of strictly improving states.

## 4. Algorithmic form and computational properties

Algorithm 1, the exact form of CaT, is succinct. It takes as input an initial state $x_0$, a target $z$, and an oracle $\mathcal O(\cdot)$. It initializes $x^* \gets x_0$, repeatedly draws one successor $g \sim P(x^*,\cdot)$, and replaces $x^*$ by $g$ only if $\mathcal O(g)<\mathcal O(x^*)$. The loop terminates when $x^*=z$ [2603.22784]. The exact algorithm therefore requires access to the optimal value function.

Because exact $\mathcal O(\cdot)$ is not available in practice, the paper replaces that oracle with an estimator $\widehat{\mathcal O}(x)$, such as a learned critic or LLM score, inside a softmax-rewinding subroutine (Algorithm 5) [2603.22784]. This introduces a small amount of branching for robustness, but the backbone remains a caterpillar [2603.22784]. The practical guidance specifies that parent states are sampled from the observed set $S$ with probability proportional to
$$
\exp(-\widehat{\mathcal O}(x)/T),
$$
with temperature $T \approx 1$ reported as working well [2603.22784].

The paper’s illustrative example considers a linear chain $x_0 \to x_1 \to \cdots \to x_n$ with “death” arcs of probability $1-p$ leading to an absorbing sink [2603.22784]. Without rewinding, one almost surely dies. CaT instead rewinds to $x_0$ until it samples $x_1$ in expected $1/p$ steps, then shifts the spine to $x_1$ and repeats, for a total expected cost of $n/p$ steps to reach $x_n$ [2603.22784].

A stronger separation appears in Lemma 2.6. In the chain of Figure 1 with $p=\tfrac12$, any non-rewinding or limited-backtracking strategy needs $\Omega(2^n)$ samples to hit $x_n$ with constant probability, whereas CaT uses only $O(n)$ samples in expectation [2603.22784]. The paper therefore attributes exponential gains to rewinding.

The finite-state value function $\mathcal O(x)$ can itself be computed exactly. Algorithm 2 is described as “Dijkstra-like” and computes $\mathcal O(x)$ exactly in $O(n^2)$ or $O((m+n)\log n)$ time by repeatedly adding to $T$ the next state with minimal provisional distance
$$
d_x=\frac{1+\sum_{y \in T} P(x,y)\, d_y}{P(x,T)}.
$$
This is a dynamic-programming or shortest-path-like computation over the Markov chain [2603.22784].

## 5. Empirical results on reasoning benchmarks

The empirical comparison is against Tree-of-Thoughts, using GPT-4 as the base LLM and identical prompting for child generation and evaluation [2603.22784]. The reported implementation uses the softmax variant of CaT rather than the exact oracle-based form.

On the Game of 24 benchmark, the task is to build an expression evaluating to $24$ from four numbers by sequentially picking two numbers and an operator [2603.22784]. The evaluation uses 100 “hard” games, indices 901–1000, and reports success rate and average total tokens, defined as prompt plus generated tokens [2603.22784]. With a budget of 15 iterations and best of 2 runs, ToT (beam 5, 15 iters) attains a success rate of 74 with average tokens 19.2k, whereas CaT (best of 2, 15 iters) attains a success rate of 81 with average tokens 15.3k [2603.22784]. Figure 2 is described as showing a roughly log-linear correlation between number of steps and number of valid expressions generated, illustrating CaT’s efficiency [2603.22784].

On 5×5 Crosswords, the task is to fill a 5×5 crossword grid from 10 clues, consisting of 5 across and 5 down clues [2603.22784]. The dataset comprises 20 held-out puzzles from GooBix, and the reported metrics are word accuracy, letter accuracy, puzzle success, and average tokens [2603.22784]. At 20 iterations, ToT achieves word accuracy 39.5, letter accuracy 64.8, games solved 5, and average tokens 73.4k, while CaT achieves word accuracy 50.0, letter accuracy 68.6, games solved 15, and average tokens 66.8k [2603.22784].

The ablation notes identify several trade-offs. Softmax rewinding introduces occasional branching for robustness, with hyperparameters including temperature $T$ and number of children per parent [2603.22784]. Best-of-2 runs mitigate being stuck on a suboptimal spine at small extra cost [2603.22784]. The summary reports that, empirically, CaT both raises success rates and reduces token consumption by approximately 20–30% [2603.22784].

## 6. Robustness, implementation guidance, and limitations

The robustness analysis distinguishes between statistical and adversarial noise in the value estimates. Under Laplace-noise on $\mathcal O(x)$, defined in Definition 4.1, the paper gives a stable variant, Algorithms 3–4, that attains a $(1+\epsilon)$-approximation to the optimal hitting time at cost polynomial in $\mathcal O(x)$ [2603.22784]. By contrast, Theorem 4.5 shows that adversarial multiplicative noise can force exponential blowup, so some statistical assumption is necessary [2603.22784]. In other words, the non-branching optimality claim is exact when $\mathcal O$ is known, but approximate inference depends critically on the quality of $\widehat{\mathcal O}$.

The practical guidance follows directly from that dependency. For the oracle, the summary recommends prompting the LLM, or a small verifier, to score the likelihood of success from a partial state $x$ and then using the negative log-score as $\widehat{\mathcal O}(x)$ [2603.22784]. For softmax rewinding, it recommends sampling parent states from $S$ with probability proportional to $\exp(-\widehat{\mathcal O}(x)/T)$, with $T \approx 1$ [2603.22784]. For the iteration budget, it suggests fixing $B=15$–$20$ for medium-difficulty tasks and using Best-of-$k$ with $k=2$–$3$ to improve robustness [2603.22784]. For child proposals, it recommends sampling only a few continuations, $2$–$5$, from the selected parent rather than using a large beam [2603.22784].

The limitations are explicit. The accuracy of $\widehat{\mathcal O}(x)$ matters: overly noisy estimates can still mislead the spine, although softmax rewinding and multiple runs help [2603.22784]. Very deep or highly branching problems may require larger $B$ or hybrid search, for example combining CaT with local tree expansions [2603.22784]. This suggests that CaT is best understood not as a universal replacement for all search procedures, but as a theorem-backed statement about the minimal branching structure required for optimality under the paper’s rewinding model [2603.22784].

Source: https://www.emergentmind.com/topics/caterpillar-of-thoughts-cat