Stochastic Tree-of-Thoughts in LLM Inference

• Stochastic Tree-of-Thoughts is a framework that organizes LLM reasoning as a stochastic search over intermediate states for reliable debugging and code repair.
• It uses a softmax-based heuristic to balance exploration and exploitation, preserving partial dialogue and code states for controlled backtracking.
• Empirical results in RTL repair and other domains demonstrate high pass rates and practical improvements in automated reasoning tasks.

Stochastic Tree-of-Thoughts (SToT) is a test-time search framework that treats large-language-model reasoning as a stochastic traversal over explicitly represented intermediate states rather than as a single linear decoding trajectory. In its canonical RTL-repair formulation, introduced within R3A, each “thought” is an intermediate dialogue-and-code state, and the search process repeatedly samples which state to expand according to a heuristic that balances exploration and exploitation, with the objective of finding a patch that passes all RTL testbenches within a time or token budget (Luo et al., 25 Nov 2025). Subsequent work places this idea within broader theories of test-time computation, heuristic classification, neural-symbolic repair, and entropy-regularized search, indicating that SToT is both a concrete repair algorithm and a member of a larger class of controlled backtracking methods for LLM inference (Azarmehr et al., 24 Mar 2026).

1. Concept and scope

In R3A, Stochastic Tree-of-Thoughts is defined in the context of automated patch generation for RTL (Verilog) designs, where “thoughts” are intermediate agent states in a conversation or search. Its key idea is to organize the LLM’s trial-and-error dialogue into a search tree over states and to guide tree expansion not by pure BFS or DFS but by stochastic sampling of tree nodes according to a heuristic. The stated objectives are to turn the LLM’s inherent randomness into a structured search process, to balance exploration against exploitation, to systematically record and re-invoke partial dialogue and code states from a precise decision point, and to reliably find a patch that passes all RTL testbenches within a time or token budget (Luo et al., 25 Nov 2025).

This formulation differs from ordinary retry-based prompting in that the search state is persistent and explicitly revisitable. The method does not merely sample multiple full solutions independently; it records partial progress and branches from earlier decision nodes. This suggests that SToT is best understood as test-time control over stateful reasoning rather than as simple output diversification.

A broader theoretical account appears in “Caterpillar of Thoughts,” which models test-time computation as an algorithm interacting with a finite, fully observable Markov chain whose states encode partial LLM solutions. In that model, the algorithm may resume generation from any previously observed state, and Chain-of-Thought, Tree-of-Thoughts, and Best-of-$k$ can all be viewed as special cases. This theoretical framing is not specific to RTL repair, but it clarifies the underlying computational primitive that SToT exploits: controlled rewinding to prior partial states (Azarmehr et al., 24 Mar 2026).

2. State-space formulation and search procedure

The canonical SToT state in R3A is $s=(c,h)\in C\times H$, where $c$ is the current code revision and $h$ is the agent chat history up to the last decision node. Each code revision lives in its own Git-like branch, and the search tree is therefore simultaneously a tree over dialogue states and a tree over code states. When a state is sampled, the system checks out the corresponding code branch, restores the exact dialogue history, and invokes the agent to continue debugging from that decision point. The resulting child state $s'=(c',h')$ is then added to the active set unless $c'$ passes all testbenches, in which case the search terminates successfully (Luo et al., 25 Nov 2025).

The sampling rule is a softmax over heuristic scores:

$$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$

After a state is sampled, R3A subtracts a small freshness penalty from its score to discourage repeatedly selecting the same node and thereby drive exploration. The algorithm can also impose hard bounds on the number of live states or prune states whose score falls below a minimum threshold, although the R3A description emphasizes time, token, and budget limits as the primary effective bounds (Luo et al., 25 Nov 2025).

The same structural pattern is visible in Clover, which maintains a frontier of states $s=(c,h)$, recomputes heuristic scores over that frontier, samples one node by softmax, restores code and dialogue, invokes the main agent until it proposes a new hypothesis, applies the patch, verifies it by simulator or linter, and either returns the first valid repair or adds the new node to the tree. Clover states explicitly that each expansion generates exactly one new child node, that there is no explicit beam-pruning in the basic loop, and that context $(c,h)$ is fully restored before each call to the main agent (Luo et al., 19 Apr 2026).

A practical consequence of this formulation is that backtracking is stateful rather than implicit. The agent does not merely “rethink”; it resumes from a serialized decision point with a concrete code snapshot and a bounded local history.

3. Heuristic scoring, stochastic control, and search bias

R3A defines a scalar heuristic $f(c,h)$ whose components include the number of passed testbenches $s=(c,h)\in C\times H$0, the total number of testbenches $s=(c,h)\in C\times H$1, the number of LLM queries $s=(c,h)\in C\times H$2, the count of unresolved compile or simulation errors $s=(c,h)\in C\times H$3, the tokens consumed $s=(c,h)\in C\times H$4, the instability penalty $s=(c,h)\in C\times H$5 for invalid tool commands, the number of patches $s=(c,h)\in C\times H$6, a base value $s=(c,h)\in C\times H$7, and tunable coefficients $s=(c,h)\in C\times H$8. The intended preference is explicit: states that pass more tests and incur fewer errors, fewer wasted tokens or queries, fewer patches, and fewer invalid tool calls receive higher scores (Luo et al., 25 Nov 2025).

Clover uses a closely related score in which $s=(c,h)\in C\times H$9, $c$0, $c$1, $c$2, $c$3, $c$4, $c$5, and coefficients $c$6 determine the promise of a hypothesis. Its reported default settings are $c$7, $c$8, $c$9, $h$0, $h$1, and $h$2, with $h$3 when an explicit softmax temperature is used. The paper states the qualitative role of these coefficients directly: increasing $h$4 makes the sampler more exploitative, increasing $h$5, $h$6, or $h$7 penalizes compile errors, long token use, or too many patches, and changing $h$8 adjusts the exploration–exploitation trade-off (Luo et al., 19 Apr 2026).

In this family of methods, stochasticity is not treated as noise to be eliminated; it is regulated. The softmax policy gives high-scoring nodes more probability mass while preserving non-zero probability for less-promising states. R3A motivates this as a way to avoid both “myopic dead-ends” and “shallow broad guesswork” (Luo et al., 25 Nov 2025). Clover makes the same claim in different terms, describing S-ToT as a mechanism that turns the LLM’s intrinsic randomness into a reliable combinatorial search (Luo et al., 19 Apr 2026).

An important misconception is that “stochastic” here implies unstructured randomness. The defining feature of SToT is the opposite: randomness is subordinated to a recorded state space, a heuristic ranking, and explicit verification.

4. Initialization, guidance, and hybrid variants

In R3A, SToT is preceded by a multi-agent anomaly detection stage for fault localization. The full RTL hierarchy is partitioned into approximately 20–50 small, self-contained code fragments; each fragment is paired with lint and waveform mismatch summaries; an independent LLM instance scores the fragment and highlights suspicious lines; these scores are aggregated; low-confidence fragments are filtered; and the top-$h$9 fault candidates are selected. In practice, the system produces two initial states: one with no hint and one seeded with the top candidates in the initial history. The search tree therefore begins as multiple roots—one “blind” and one “focused”—which preserves flexibility while steering early expansions toward high-value regions (Luo et al., 25 Nov 2025).

Clover generalizes this architecture into a neural-symbolic harness. Its inner MainAgentProcedure() may spawn sub-agents such as a context agent or lint-fix agent, and may call an SMT-based symbolic solver when appropriate. The paper’s central claim is that different repair operations favor distinct strategies and that the orchestration layer dynamically dispatches tasks to specialized LLM agents or symbolic solvers, while S-ToT manages the main agent’s context as a search tree (Luo et al., 19 Apr 2026). This suggests that SToT can function as a control layer above heterogeneous reasoning modules rather than as a monolithic generation algorithm.

A different extension appears in “Heuristic Classification of Thoughts Prompting,” which recasts stochastic Tree-of-Thoughts as a search tree $s'=(c',h')$0 over partial reasoning states. It introduces a heuristic classifier $s'=(c',h')$1 that labels thoughts, scores them, prunes those below a threshold, and reweights effective branching probabilities so that

$s'=(c',h')$2

In that account, SToT becomes a special case of a broader heuristic-control framework in which sampled children are immediately classified, reweighted, and pruned according to a learned or prompt-based heuristic (Lin et al., 14 Apr 2026).

These variants share a common design pattern: the LLM proposes candidate continuations, but a secondary control mechanism—fault localization, heuristic classification, symbolic dispatch, or verification feedback—shapes which continuations remain live.

5. Empirical behavior across domains

The empirical literature presents SToT primarily as a reliability mechanism for RTL repair, but later work also situates it within more general search-and-reasoning settings.

Work	Domain	Reported result
R3A	RTL repair	29/32 bugs fixed; average pass@1 $s'=(c',h')$3, pass@5 $s'=(c',h')$4
Clover	RTL repair	31/32 bugs fixed; pass@1 $s'=(c',h')$5
CaT	Game of 24 / 5×5 Crossword	better success than ToT with fewer token generations
HCoT	24-Game / 1D-ARC / List-Function	outperforms several prompting baselines in reported settings
Protein Thoughts	PPI discovery	mean best-binder rank $s'=(c',h')$6 vs $s'=(c',h')$7; Micro-F1 $s'=(c',h')$8

For RTL repair, R3A reports that it fixes 90.6% of bugs in the RTL-repair dataset within a given time limit, covering 45% more bugs than traditional methods and other LLM-based approaches, and achieving an 86.7% pass@5 rate on average (Luo et al., 25 Nov 2025). Its detailed benchmark description states 29/32 bugs fixed, a 45% absolute gain over the prior SMT-based state of the art, and average pass@1 $s'=(c',h')$9, pass@5 $c'$0 under a $c'$1 retry budget. On the hard subset, naïve single-shot prompting yields $c'$2 pass@1, BFS-only Tree-of-Thoughts hovers around $c'$3, and SToT is approximately $c'$4 (Luo et al., 25 Nov 2025).

Clover reports stronger RTL-repair results under a different resource regime. On the standard RTL-Repair benchmark of 32 RTL bugs, its full system with S-ToT fixes 31/32 bugs under a 30 min / 2 M token constraint and achieves pass@1 $c'$5. The comparison figures given are 16/32 bugs and pass@1 $c'$6 for traditional SMT-only RTL-Repair, 12/32 bugs and pass@1 $c'$7 for MEIC, and 20/32 bugs and pass@1 $c'$8 for UVLLM. On the 15 most complex designs, “no ToT” gives pass@1 $c'$9, “no SMT” gives pass@1 $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$0, and the full system gives pass@1 $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$1 (Luo et al., 19 Apr 2026).

Beyond RTL repair, “Caterpillar of Thoughts” reports that on the Game of 24, Tree-of-Thoughts with beam 5 attains 74% success at 19.2k average tokens, whereas CaT with 15 steps attains 81% at 15.3k tokens and CaT with 10 steps attains 78% at 14.2k tokens. On 5×5 Crossword, Tree-of-Thoughts at 20 steps yields 39.5% word accuracy, 64.8% letter accuracy, 5% games solved, and 73.4k average tokens, while CaT yields 50.0%, 68.6%, 15%, and 66.8k, respectively (Azarmehr et al., 24 Mar 2026).

“HCoT” reports 24-Game, 1D-ARC, and List-Function results. On the 24-Game with 1362 instances, ToT-BFS $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$2 attains 55.65% accuracy with 7.28 M generated tokens, while HCoT configurations range from 24.3% to 36.71% accuracy with 2.50 M to 3.10 M generated tokens. On 1D-ARC with GPT-4o, HCoT reports 47.61% accuracy, 58.16% matching, and 70.04% match-and-solve, compared with 44.06% for IO-Prompt, 38.40% for Auto-CoT, and 31.19% for ToT-Prompt. On List-Function with DeepSeek-R1, HCoT reports up to 81.2% accuracy, compared with 76.0% for IO-Prompt, 73.2% for Auto-CoT, and 54.4% for ToT-Prompt (Lin et al., 14 Apr 2026).

“Protein Thoughts” is not an RTL-repair system, but it demonstrates that stochastic thought search can also be instantiated as a hypothesis-guided, entropy-regularized ToT in scientific discovery. On SHS148k, it reports mean best-binder rank of 11.2 versus 47.7 for an entropic tree search baseline, described as a 76% improvement, and a trained value function with $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$3 Micro-F1 (Yeon et al., 19 May 2026).

6. Theory, limitations, and interpretive issues

A central theoretical question is whether broad tree search is intrinsically necessary. “Caterpillar of Thoughts” answers this negatively within its formal model: for every finite set of observed states $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$4, there exists an $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$5 such that the optimal algorithm can be chosen non-branching and only ever backtracks to the one state that minimizes the single-state value $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$6. The resulting optimal history is a caterpillar tree: if one removes the leaves of the generated state tree, one obtains a path (Azarmehr et al., 24 Mar 2026). In that framework, broad tree search is not required for optimality.

That result should not be overextended. The same paper assumes a finite, fully observable Markov chain and an objective of minimizing expected hitting time or maximizing target-hitting probability under a budget. By contrast, the RTL-repair papers explicitly state that they make no formal convergence guarantees because the search space of code patches is unbounded (Luo et al., 19 Apr 2026). R3A likewise reports empirical reliability rather than closed-form reliability bounds, noting that pass@$$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$7 is measured empirically (Luo et al., 25 Nov 2025). A plausible implication is that the caterpillar result is best read as a theoretical reference point for test-time computation, whereas SToT in repair systems remains a practical heuristic under verification feedback.

The computational profile is similarly pragmatic. R3A states that each iteration samples one state in $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$8 via softmax, checks out code in $$\Pr[s]=\frac{\exp(f(s))}{\sum_{s'\in S}\exp(f(s'))}.$$9, and runs an LLM dialogue of cost $s=(c,h)$0, where $s=(c,h)$1 is approximately tokens per round. In the worst case it generates $s=(c,h)$2 states and performs $s=(c,h)$3 LLM calls, while practical budgets keep execution to 20–100 calls (Luo et al., 25 Nov 2025). Clover notes that frontier size grows with the number of expansions and identifies more aggressive beam-pruning as an open question (Luo et al., 19 Apr 2026).

Several open limitations recur across the literature. Clover lists dependence on hand-tuned heuristic coefficients, frontier growth, the omission of semantic code-structure signals such as AST-based distances, and the absence of theoretical bounds on the number of expansions needed for success (Luo et al., 19 Apr 2026). HCoT frames a related issue in different terms: heuristic filtering can improve token efficiency, but its behavior depends on the quality of the classifier and the abstraction used to label partial reasoning states (Lin et al., 14 Apr 2026). Protein Thoughts introduces adaptive temperature, directive-conditioned weighting, and hypothesis-aware pruning, indicating one route toward more structured stochastic control; however, that framework is domain-specific and tied to a learned value function over biological signals (Yeon et al., 19 May 2026).

The overall picture is therefore twofold. On one side, SToT is an empirical strategy for reliability under bounded compute, especially in verified repair workflows. On the other, it belongs to a broader research program on test-time computation, where stochastic sampling, backtracking, heuristic selection, and verification are treated as algorithmic design choices rather than incidental prompt engineering.