Entrocraft: Controlled Entropy in LLM RL

• Entrocraft is a rejection-sampling method for reinforcement learning that treats the entropy curve as a customizable training hyperparameter to prevent model performance saturation.
• It filters rollout samples based on advantage estimates, aligning with targeted entropy schedules and integrating with policy-gradient updates like PPO, GRPO, or GSPO.
• Empirical results demonstrate that Entrocraft improves pass@K metrics and held-out accuracy on benchmarks by effectively managing entropy collapse during training.

Searching arXiv for the primary paper and closely related context papers on RL, entropy control, and policy-gradient methods mentioned in the source block. Entrocraft is a rejection-sampling method for reinforcement learning (RL) of LLMs that treats the entropy curve as a user-customizable training schedule rather than a passive diagnostic. It was introduced to address performance saturation in LLM RL, a regime in which pass@K, generalization to held-out math problems, and output diversity stop improving as training scales. The central claim is that entropy collapse is the operative mechanism behind this saturation, and that precise control of the entropy trajectory can delay or avoid it. In the reported experiments, Entrocraft requires no objective regularization, is advantage-estimator-agnostic, and is applied on top of standard policy-gradient updates such as PPO, GRPO, or GSPO (Li et al., 29 Apr 2026).

1. Problem setting and motivating observation

Entrocraft originates from the observation that, under policy-gradient RL for LLMs, token-prediction entropy tends to collapse. In the reported account, entropy functions as a proxy for exploration: once entropy falls below a moderate level, the model “locks in” on a narrow set of reasoning paths, after which additional data or compute no longer improves pass@K, held-out mathematical generalization, or output diversity (Li et al., 29 Apr 2026).

The work positions this phenomenon as a diagnosis of performance saturation rather than merely an optimization artifact. Existing interventions are described as including entropy bonuses, clipping large importance ratios, and decoupling positive and negative updates. These methods can raise entropy, but only coarsely; the reported failure mode is long-term instability, including entropy oscillations and gradient noise, which in turn limits further gains. Within this framing, Entrocraft recasts the entropy curve as a first-class hyperparameter, analogous to a learning-rate schedule.

A plausible implication is that the method is less concerned with maximizing entropy per se than with controlling its temporal profile. The distinction matters because the reported objective is not uniformly high entropy, but a schedule that preserves useful exploration early and relaxes that constraint later in training.

2. Theoretical relation between advantage and entropy change

The theoretical basis of Entrocraft is a relation between per-update entropy change and the advantage distribution under assumptions described as realistic for LLM policies, specifically small step size and validity of a Taylor expansion. Writing the policy at step $t$ as a softmax over logits, and denoting the vocabulary distribution by $p = (p_1,\dots,p_{|V|})$, a single unconstrained policy-gradient update induces a probability shift $\delta p$. The reported first-order change in Shannon entropy,

$H(p) = -\sum_i p_i \log p_i,$

is

$$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$

The update is then stated to be proportional to the advantage-weighted score function:

$\delta p_k \propto \hat{A}_k.$

Under mild lower bounds on $p_k$, this yields the token-level statement

$$\text{Theorem 1 (Token-Level):} \quad \hat{A}_k \cdot \Delta H \le 0.$$

The interpretation given is direct: positive advantages almost always decrease entropy, because they make the model more confident, while negative advantages increase entropy by flattening the distribution (Li et al., 29 Apr 2026).

The analysis is extended from token-level entropy to sequence-level entropy by aggregating across token positions in a rollout $y$. The reported result is

$$\text{Theorem 2 (Sequence-Level):} \quad \hat{A}(x,y) \cdot \Delta H \le 0,$$

provided

$p = (p_1,\dots,p_{|V|})$0

which is described as empirically always holding for LLM rollouts under standard advantage estimators. The stated takeaway is that uncontrolled policy gradients implicitly force entropy collapse whenever good rollouts dominate.

This suggests that the intervention target is not the policy objective itself, but the empirical distribution of advantages presented to the optimizer. In that sense, Entrocraft operates by modulating which samples are allowed to influence an otherwise standard policy-gradient update.

3. Entropy-guided rejection sampling mechanism

Entrocraft implements entropy control through a rejection-sampling filter applied at rollout time. For each RL step and prompt $p = (p_1,\dots,p_{|V|})$1, a candidate pool $p = (p_1,\dots,p_{|V|})$2 is sampled from the old policy $p = (p_1,\dots,p_{|V|})$3. Each rollout is assigned an advantage $p = (p_1,\dots,p_{|V|})$4, and the batch-average entropy $p = (p_1,\dots,p_{|V|})$5 is computed (Li et al., 29 Apr 2026).

An out-of-range indicator is then defined as

$p = (p_1,\dots,p_{|V|})$6

where $p = (p_1,\dots,p_{|V|})$7 is the current target window. The semantics are asymmetric in sign but simple in effect. If entropy is too low, then $p = (p_1,\dots,p_{|V|})$8, so $p = (p_1,\dots,p_{|V|})$9 and the desired update should have $\delta p$0; the method therefore favors negative-advantage samples. If entropy is too high, then $\delta p$1, so $\delta p$2 and the method keeps positive-advantage samples in order to obtain $\delta p$3.

Each candidate rollout $\delta p$4 is accepted with probability

$\delta p$5

with $\delta p$6, and the reported experiments use $\delta p$7. Only accepted samples enter the usual PPO, GRPO, or GSPO gradient. The pseudocode given in the source can be summarized as follows:

$\delta p_k \propto \hat{A}_k.$7

The method is described as objective-agnostic, because it operates on the sampled rollout set rather than through an additional loss term. It is also described as introducing zero additional loss terms. The reported functional consequence is rapid movement of the entropy into a user-specified window within a few updates, because the filter directly reshapes the advantage distribution.

4. Entropy schedules and curve design

A central feature of Entrocraft is that entropy targets can be scheduled over training in the same way as a learning-rate decay. The study reports experiments with three schedules: a fixed target, cosine decay, and linear decay (Li et al., 29 Apr 2026).

The fixed target uses $\delta p$8. It is reported to be initially stable, but around 200K samples the filter must rely on increasingly scarce negative or positive samples, which leads to large fluctuations and eventual collapse or explosion. The cosine schedule is

$\delta p$9

It reportedly keeps training smooth but slightly underperforms the linear schedule. The linear schedule decays $H(p) = -\sum_i p_i \log p_i,$0 from $H(p) = -\sum_i p_i \log p_i,$1 to $H(p) = -\sum_i p_i \log p_i,$2 over the RL curriculum. In the reported ablations, this schedule is the winner: it prevents late-stage instabilities, sustains continual improvement beyond 300K samples, and yields the highest held-out accuracy.

The source further reports a contrast between fixed-target and linear schedules. Under a fixed target, entropy begins to oscillate after 200K samples, KL-penalty spikes, and held-out mean@32 on MATH-500 drops. Under the linear schedule, the entropy decline is smooth and monotonic, KL loss remains constant, and mean@32 on MATH-500, AIME-25, and AIME-26 rises steadily even up to 400K samples.

A plausible implication is that schedule design matters because the availability of sign-conditioned advantages changes as the policy sharpens. Early in training, aggressive rejection can maintain a high-entropy regime; later, the same target becomes expensive in terms of sample selection and can destabilize the update statistics. The linear decay can therefore be read as a mechanism for easing off the hardest rejection steps as training proceeds.

5. Empirical behavior and reported performance

The empirical claims center on improved generalization, output diversity, and long-term training, with performance measured on mathematical reasoning benchmarks and pass@K metrics (Li et al., 29 Apr 2026). The reported results include model-size comparisons, long-horizon training behavior, and held-out evaluation.

Setting	Reported metric	Reported value
Baseline GRPO on Qwen3-4B	MATH-500 mean@32	$H(p) = -\sum_i p_i \log p_i,$3
Baseline GRPO on Qwen3-4B	MATH-500 pass@32	$H(p) = -\sum_i p_i \log p_i,$4
Qwen3-4B + Entrocraft	MATH-500 mean@32	$H(p) = -\sum_i p_i \log p_i,$5
Qwen3-4B + Entrocraft	MATH-500 pass@32	$H(p) = -\sum_i p_i \log p_i,$6
Standard GRPO on Qwen3-8B	MATH-500 mean@32	$H(p) = -\sum_i p_i \log p_i,$7
Standard GRPO on Qwen3-8B	MATH-500 pass@32	$H(p) = -\sum_i p_i \log p_i,$8
AIME-25 baseline to Entrocraft	pass@32	$H(p) = -\sum_i p_i \log p_i,$9

These figures support the paper’s statement that a 4B model with Entrocraft outperforms an 8B baseline. The same section reports that on AIME-25, pass@32 rises from $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$0 to $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$1, described as a $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$2 improvement. The pass@K curves for $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$3 are said to grow faster than the baseline, and this is interpreted in the source as evidence that the method avoids collapse to a single “best” answer.

The long-term behavior is presented as a central result. Standard GRPO is reported to saturate after 100K samples, with MATH-500 mean@32 plateauing at approximately $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$4. Entrocraft with linear annealing is reported to sustain improvement for up to $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$5 more samples, reaching approximately $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$6 mean@32 and thereby delaying saturation by a factor of four. On unseen benchmarks, specifically AMC-23 and AIME-24, AIME-25, and AIME-26, the paper reports consistent gains of $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$7 to $$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$8 points in mean@32. Training is further described as numerically stable, with bounded KL penalties and no gradient explosions or implosions.

Taken together, these results motivate the paper’s broader claim that entropy control can function as a lever for both sample efficiency over long horizons and retained output diversity. This suggests a coupling between exploration geometry and pass@K scaling that is not captured by single-sample or low-$$\Delta H = H(p+\delta p)-H(p) \approx - \sum_{i\in V} \delta p_i \log p_i.$$9 evaluation alone.

6. Relation to prior entropy-preserving interventions

The source positions Entrocraft against earlier attempts to prevent entropy collapse through regularization or clipping. The interventions named are entropy bonus, clipping large importance ratios, and decoupling positive and negative updates. Their common limitation, as characterized in the paper, is that the resulting entropy curves often exhibit long-term instability, including oscillations, and that these instabilities hinder performance gains (Li et al., 29 Apr 2026).

Entrocraft differs in two stated respects. First, it requires no objective regularization; it acts through rollout filtering instead of augmenting the loss. Second, it is advantage-estimator-agnostic, because the mechanism depends only on the sign and magnitude structure of estimated advantages rather than on a specific estimator construction. The method is also described as working on top of any policy-gradient method, with PPO, GRPO, and GSPO named explicitly.

A plausible implication is that the method occupies a distinct point in the design space of entropy control. Rather than shaping the objective to indirectly influence entropy, it shapes the sample set so that the gradient receives the desired entropy-inducing bias. In the terminology of the paper, this amounts to directly reshaping the advantage distribution.

The paper also provides a theoretical explanation for the behavior of existing RL and entropy-preserving methods through the sign relation between $\delta p_k \propto \hat{A}_k.$0 and $\delta p_k \propto \hat{A}_k.$1. Within that framing, standard policy-gradient improvement naturally drives confidence up and entropy down whenever high-advantage rollouts dominate the update. Entrocraft does not overturn that dynamic; it exploits it by altering which rollouts are admitted.

7. Significance, interpretation, and boundaries of the claim

The reported contribution of Entrocraft is summarized by the paper as transforming entropy from a passive diagnostic into an actively controlled training hyperparameter. The operative components are explicitly enumerated: linking per-step $\delta p_k \propto \hat{A}_k.$2 to $\delta p_k \propto \hat{A}_k.$3 theoretically, biasing the advantage distribution through rejection sampling, and scheduling the target entropy, with simple linear decay identified as the most effective schedule in the reported ablations (Li et al., 29 Apr 2026).

Several boundaries are also clear from the description. The theoretical statements are first-order and depend on assumptions such as small step size and Taylor-expansion validity. The sequence-level theorem includes an additional condition on $\delta p_k \propto \hat{A}_k.$4 and token probabilities, which the paper states holds empirically for LLM rollouts under standard advantage estimators. The empirical results are concentrated on mathematical reasoning benchmarks and on metrics such as mean@32 and pass@K. Accordingly, any broader claim beyond those settings would be an inference rather than a directly established result.

A common misconception would be to treat Entrocraft as merely another entropy-maximization heuristic. The source does not support that characterization. The method does not prescribe uniformly higher entropy; instead, it prescribes precise curve control, and the preferred schedule is not flat but annealed from approximately $\delta p_k \propto \hat{A}_k.$5 to approximately $\delta p_k \propto \hat{A}_k.$6. Another misconception would be to read the method as replacing PPO, GRPO, or GSPO. The paper instead describes it as a filter layered on top of those policy-gradient methods.

In the paper’s own terms, the significance of Entrocraft lies in addressing performance saturation by making the entropy trajectory explicitly controllable. This suggests a broader methodological perspective in which exploration schedules for LLM RL may be tuned with a granularity comparable to learning-rate schedules, rather than delegated to indirect regularization mechanisms alone.