Entropy-fork Tree Majority Rollout (ETMR)

• ETMR is an entropy-aware rollout strategy that refines test-time reinforcement learning by targeting high-uncertainty token positions.
• It employs a tree-structured approach combining prefix reuse with selective branching to reduce token cost and improve exploration.
• The method aggregates leaf responses through majority voting to generate accurate pseudo-labels, balancing exploration and exploitation.

Entropy-fork Tree Majority Rollout (ETMR) is an entropy-aware rollout strategy for test-time reinforcement learning (TTRL) with LLMs. It was introduced as the rollout component of the broader ETTRL framework and is presented as a replacement for the standard fully parallel rollout plus majority-vote procedure used in vanilla TTRL. ETMR samples a small number of root trajectories, identifies high-entropy token positions as fork points, branches only at those positions, completes the resulting continuations to full responses, and selects a pseudo-label by majority voting over extracted answers. In the paper’s framing, ETMR is intended to improve the exploration–exploitation balance during pseudo-label estimation by reusing low-entropy prefixes and allocating branching only to uncertain parts of the reasoning trajectory (Liu et al., 15 Aug 2025).

1. Definition and motivation

ETMR is introduced to address two weaknesses attributed to ordinary TTRL. The first is high inference cost: reliable pseudo-labels often require “tens to hundreds of rollouts,” and fully parallel sampling repeatedly regenerates redundant low-entropy tokens. The second is early-stage estimation bias: majority-voted pseudo-labels may be inaccurate early in training, yet the policy may still overfit to them, which can reduce diversity and push optimization toward local optima. ETMR directly targets the first issue and is described as indirectly helping exploration and diversity by branching specifically at uncertain parts of a trajectory rather than resampling entire responses from scratch (Liu et al., 15 Aug 2025).

The method’s name reflects its structure. “Entropy-fork” refers to branching at high-entropy token positions. “Tree” refers to a rollout geometry in which one initial sampled response acts as a trunk and multiple continuations are spawned from selected fork points. “Majority rollout” refers to answer aggregation at the leaves: each completed branch is converted to a candidate answer, and the final pseudo-label is the majority answer over those leaves. The paper explicitly contrasts this with standard parallel rollouts, which it characterizes as suffering from prohibitive inference cost, insufficient exploration for a given token budget, early estimation bias, overconfidence, reduced diversity, and poor exploration–exploitation balance (Liu et al., 15 Aug 2025).

The central intuition is that low-entropy regions can be exploited through prefix reuse, whereas high-entropy regions should receive exploration budget. ETMR therefore does not treat every decoding position symmetrically. It assumes that high-entropy tokens “significantly influence reasoning quality” and diversity, while low-entropy tokens “have minimal impact on final outcomes” and can be reused. This suggests an allocation policy that is local in token space rather than global at the sequence level.

2. Rollout construction and tree structure

For a prompt $x$ or $q$, ETMR first samples $M$ initial complete responses from the current policy $\pi_\theta$. Each sampled response becomes one tree root. Along each root trajectory, token uncertainty is computed for every token position. The pseudocode uses

$$H(y_t) \leftarrow -\log \pi_\theta(y_t \mid x, y_{<t}),$$

whereas elsewhere the paper formally defines token entropy as Shannon entropy over the full next-token distribution,

$$H_t = -\sum_{v\in\mathcal{V}} \pi_\theta(v\mid c)\log \pi_\theta(v\mid c), \qquad c=(q,o_{<t}).$$

The paper notes this discrepancy only implicitly; it does not resolve it explicitly (Liu et al., 15 Aug 2025).

After uncertainty estimation, the top-$N$ token positions with highest entropy or uncertainty are selected as fork points for each root trajectory: $B_{i,l} \leftarrow \text{Top-}N\ H(\cdot \mid x).$ At each selected position $t$, ETMR samples $B$ new continuations from the prefix $q$0. Every branch is then continued to a complete response. The resulting tree is shallow rather than recursively expanding: for each root rollout there is one base trajectory, and at each of $q$1 selected fork points the algorithm creates $q$2 additional branches. The total number of leaf rollouts is modeled as

$q$3

This means the implemented structure is one root plus $q$4 extra leaves per tree, not branch-on-branch recursion (Liu et al., 15 Aug 2025).

The branching rule is therefore fixed by three structural hyperparameters: $q$5, the number of trees; $q$6, the number of fork points per tree; and $q$7, the number of branches per fork point. The paper does not define an entropy threshold for branching, a pruning criterion, a confidence-based stopping rule, or a dynamic adaptive branch factor. All branches proceed to leaf nodes, and all leaf answers are eligible for majority aggregation.

This shallow-fork structure distinguishes ETMR from recursive tree-search procedures. A closely related inference-time method, “Fork-Think with Confidence,” uses a single seed path, selects low-confidence positions, samples continuations from those prefixes, and aggregates answers by majority vote, but it is likewise not a recursively expanding tree search in the full sense (Al-Khalili et al., 30 Jun 2026). ETMR differs in being embedded in TTRL and in using entropy as the canonical fork signal.

3. Formalization and budget equations

Within the ETTRL paper, ETMR is formalized as a rollout-generation and pseudo-label-estimation strategy rather than as a new RL loss. The preliminaries retain the usual TTRL pseudo-label reward: $q$8 where $q$9 is the majority-voted answer over sampled outputs. ETMR modifies how the sampled set is constructed, but not the basic majority-vote reward principle (Liu et al., 15 Aug 2025).

The paper defines token entropy as

$M$0

and response entropy as

$M$1

Although response entropy is used mainly for EAR rather than ETMR, the token-level entropy is ETMR’s primary branching statistic (Liu et al., 15 Aug 2025).

A key contribution of the paper is an explicit token-cost model. Assuming fork points are uniformly distributed along the generation, the token cost for one tree-based rollout set is

$M$2

Hence

$M$3

Relative to fully parallel sampling, the token ratio is

$M$4

For $M$5 and $M$6, this gives

$M$7

which the paper rounds to “60%.” In the experimental setting $M$8, $M$9, $\pi_\theta$0, the number of rollout leaves is

$\pi_\theta$1

and the token ratio is

$\pi_\theta$2

These formulas make ETMR a prefix-sharing rollout generator with an explicit budget model rather than a purely qualitative tree heuristic (Liu et al., 15 Aug 2025).

The same fixed-budget perspective is echoed in later tree-search work on GRPO, which argues that rollout construction should maximize rollout-set informativeness rather than merely sample independently from the root. That work does not implement majority voting, but it formalizes why budget-aware intermediate-state selection can matter under sparse group-relative rewards (Hu et al., 6 May 2026). This suggests a broader interpretation of ETMR as one instance of budget-aware rollout design.

4. Majority voting, pseudo-labels, and integration with TTRL

ETMR’s aggregation rule remains the standard pseudo-labeling mechanism of TTRL: extract a final answer from every complete leaf response and choose the majority answer as the pseudo-label. The rollout tree therefore changes candidate generation rather than label semantics. After pseudo-label estimation, the policy is updated using GRPO on the sampled responses, after downsampling to the training subset if desired (Liu et al., 15 Aug 2025).

The ETTRL paper states that the downstream RL update uses GRPO: $\pi_\theta$3 and

$\pi_\theta$4

ETMR affects this objective indirectly by altering the sampled rollout set and the resulting pseudo-label $\pi_\theta$5, which changes the binary rewards assigned to trajectories (Liu et al., 15 Aug 2025).

The majority-vote mechanism is central but also limited. The ETTRL paper associates ETMR with better exploration and improved pseudo-label quality, yet it does not provide a separate formal calibration analysis of majority accuracy. This limitation becomes central in later work. ECHO, for example, explicitly retains the same majority-vote pseudo-label construction,

$\pi_\theta$6

but argues that majority-derived rewards can be noisy and biased early in training and therefore adds confidence-aware rollout control, pruning, and confidence-adaptive policy updates (Zhao et al., 2 Feb 2026). This places ETMR in a lineage of methods where majority voting is retained but increasingly buffered by uncertainty or confidence signals.

A separate but related line of inference-time reasoning work shows that majority vote can also be effective outside TTRL when branching occurs from internal prefixes rather than from the prompt root. “Fork-Think with Confidence” uses a single greedy seed path, identifies low-confidence positions, samples multiple continuations, and aggregates by majority voting over extracted final answers (Al-Khalili et al., 30 Jun 2026). This supports ETMR’s core structural assumption that majority aggregation need not require independent full-sequence resampling.

5. Empirical results, budget efficiency, and limitations

The ETMR experiments use Qwen2.5-Math-1.5B, Qwen2.5-Base-3B, and Llama-3.1-8B on AIME 2024, AMC, and MATH-500, with Pass@1 under greedy decoding as the evaluation metric. The general rollout protocol uses 64 sampled responses per prompt for voting in the base protocol, then downsamples to 32 responses per prompt for training, with maximum generation length 3072 tokens and training episodes of 10 on MATH-500, 30 on AMC, and 80 on AIME 2024. In the ETMR setting, $\pi_\theta$7, $\pi_\theta$8, and $\pi_\theta$9, giving 60 leaf rollouts compared with 64 fully parallel baseline rollouts, while reducing average token consumption to 60% of the fully parallel baseline (Liu et al., 15 Aug 2025).

On Qwen2.5-Math-1.5B, ETMR improves over TTRL from 15.8 to 21.0 on AIME, from 48.9 to 50.8 on AMC, and from 73.0 to 76.9 on MATH-500, raising the average from 45.9 to 49.6. On Qwen2.5-Base-3B, ETMR changes 7.9 to 9.2 on AIME, 40.7 to 41.7 on AMC, and 72.2 to 71.7 on MATH-500, with average 40.3 to 40.9. On Llama-3.1-8B, ETMR changes 10.0 to 16.9 on AIME, 32.3 to 35.4 on AMC, and 63.7 to 59.5 on MATH-500, with average 35.3 to 37.3 (Liu et al., 15 Aug 2025).

The paper’s headline claim that the approach enables Llama-3.1-8B to achieve a 68 percent relative improvement in Pass@1 on AIME 2024 while consuming only 60 percent of the rollout token budget corresponds numerically to the ETMR table’s 10.0 to 16.9 improvement, reported there as a 69.0 percent relative gain. The token-budget reduction is therefore clearly attributable to ETMR, while the broader ETTRL framing combines ETMR with EAR (Liu et al., 15 Aug 2025).

The paper also states several limitations. First, wall-clock speedup is lower than theoretical token savings because parallel sampling uses batched execution efficiently, ETMR uses a tree-structured pipeline-style rollout, and the current RL framework lacks native support for such hybrid scheduling. Second, ETMR is pronouncedly sensitive to the temperature parameter; too high a temperature can cause training collapse. Third, the visible text does not provide explicit ETMR ablations over entropy thresholds, variable tree depth, adaptive stopping, or dynamic branch allocation (Liu et al., 15 Aug 2025).

Subsequent work sharpens these limitations. ECHO argues that entropy-only branching can induce rollout collapse, in which the branching budget concentrates on a few trajectories with consecutive high-entropy segments, rapidly reducing the number of effective branches. ECHO therefore replaces pure entropy-driven forking with an entropy–confidence hybrid branch-width rule and adds online pruning (Zhao et al., 2 Feb 2026). This suggests that ETMR’s entropy fork rule is effective but not necessarily sufficient when local uncertainty is persistent rather than isolated.

6. Related formulations and broader context

ETMR occupies an intermediate position between flat self-consistency and full recursive tree search. It is more structured than fully parallel majority rollouts because it shares low-entropy prefixes, but less general than recursive search because it builds a shallow fork structure attached to sampled roots. In this respect it resembles later confidence-based prefix branching methods and differs from search procedures with node-selection policies, recursive expansion, or backup operators (Al-Khalili et al., 30 Jun 2026).

Its most direct descendants are methods that retain the same broad pipeline—multiple candidate rollouts, majority-based pseudo-labeling, and online updates—but refine the forking rule. ECHO is explicitly framed as an improvement over ETMR-style TTRL: it keeps tree-structured rollouts and majority pseudo-labeling, but jointly uses local entropy and group-level confidence to control branch width, prunes persistently low-confidence branches, and modifies GRPO-style optimization with confidence-adaptive clipping and entropy-confidence hybrid shaping (Zhao et al., 2 Feb 2026). This makes ETMR a baseline architecture for entropy-guided tree majority rollout, with later work adding anti-collapse control.

A different extension concerns the source of rollout diversity. Entropy-Gated Latent Recursion introduces a second, deterministic diversity axis indexed by the layer span $$H(y_t) \leftarrow -\log \pi_\theta(y_t \mid x, y_{<t}),$$0 at which top decoder layers are recursively re-applied at high-uncertainty tokens. It is not itself a majority-tree search method, but it is compatible with majority-vote aggregation and expands the set of candidate rollouts available to downstream procedures that consume rollouts, including self-consistency and GRPO (Bhattacharjee et al., 15 Jun 2026). This suggests that ETMR’s token-space branching need not be the only axis along which branching can occur.

Theoretical work on budgeted tree search for GRPO provides another contextual lens. InfoTree formalizes rollout informativeness under a fixed budget and argues that independent samplers can suffer persistent collapse on hard prompts. It recasts intermediate-state selection as monotone submodular maximization and derives Uncertainty-aware Upper Confidence Bound terms in which token-level entropy appears as part of a marginal-gain approximation (Hu et al., 6 May 2026). Although InfoTree does not implement majority voting, its fixed-budget analysis supports the broader ETMR intuition that branching should be targeted where it produces informative divergence rather than spent uniformly at the root.

There is also a mathematically distinct but conceptually related use of “tree majority” outside LLM inference. A stochastic learning model on rooted regular trees with majority update policy reduces an infinite-dimensional process to a scalar recursion,

$$H(y_t) \leftarrow -\log \pi_\theta(y_t \mid x, y_{<t}),$$1

and exhibits convergence and phase transitions determined by the fixed points of $$H(y_t) \leftarrow -\log \pi_\theta(y_t \mid x, y_{<t}),$$2 (Podder et al., 2024). That model does not define ETMR and does not use entropy, but it provides a clean tree-majority prototype in which local stochastic evidence is aggregated upward. This suggests a useful distinction: ETMR is an algorithmic rollout policy for pseudo-label estimation in TTRL, whereas rooted-tree majority dynamics supplies an abstract model of how local aggregation on trees can exhibit stable and unstable regimes.

Taken together, these neighboring formulations locate ETMR as a specific design pattern: entropy-guided, prefix-sharing, shallow tree rollout with majority-vote pseudo-labeling inside TTRL. Its defining claims are token-budget reduction through prefix reuse, exploration focused on high-entropy positions, and preservation of majority-vote training semantics. Its main caveats are shallow tree depth, incomplete specification of dynamic branching policies, and sensitivity to entropy-only control in difficult regimes (Liu et al., 15 Aug 2025).