On-Policy Entropy Flow Optimization

• The paper demonstrates that reweighting token gradients based on first-order entropy change effectively mitigates entropy collapse in RLVR.
• OPEFO is a strict on-policy method that resizes entropy-increasing and entropy-decreasing updates using an adaptive, batch-specific coefficient.
• Empirical results show that OPEFO stabilizes entropy trajectories and improves reasoning benchmarks by up to +2.3 points compared to baseline methods.

Searching arXiv for the primary OPEFO paper and closely related entropy-control / on-policy RL papers to support the article. On-Policy Entropy Flow Optimization (OPEFO) is a strict on-policy modification of the policy-gradient update in reinforcement learning with verifiable rewards (RLVR) that balances token-level entropy flow by adaptively reweighting entropy-increasing and entropy-decreasing updates within each batch. In the formulation introduced in “Understanding and Preventing Entropy Collapse in RLVR with On-Policy Entropy Flow Optimization,” OPEFO is motivated by the observation that widely used RLVR algorithms, such as GRPO, often suffer from entropy collapse, leading to premature determinism and unstable optimization; OPEFO addresses this by estimating each token’s first-order contribution to entropy change, partitioning tokens into entropy-increasing and entropy-decreasing sets, and rescaling their policy-gradient contributions with a batch-specific coefficient chosen to approximately enforce zero net entropy change (Xu et al., 12 May 2026).

1. RLVR setting and the target of optimization

The method is defined in the RLVR setting, where a policy LLM $\pi_\theta$ (denoted $T_\theta$) generates a response $y=(y_1,\dots,y_T)$ to a prompt $x$, and a verifier computes an automatic reward $r(x,y)$. Unlike RLHF, there is no human preference model; rewards are objective and automatically verifiable. The underlying objective is

$$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$

Using the policy-gradient theorem, the paper writes

$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$

with GRPO-style group-relative advantages to avoid a separate value network: $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$ For each prompt, $K$ responses are sampled, rewards are computed, and all tokens in a trajectory share the same normalized advantage (Xu et al., 12 May 2026).

In this context, “strict on-policy” means that each update uses fresh rollouts from the current policy $\pi_\theta$, and no importance ratios or reference policy are used in the loss. OPEFO leaves this base RLVR objective intact at the level of rewards and advantages; its intervention is narrower and more specific. It modifies the weighting of token-level gradients in the base on-policy objective to control entropy flow (Xu et al., 12 May 2026).

This restriction is important because the paper positions OPEFO against two classes of remedies for entropy collapse: entropy bonuses, which act at the objective level, and PPO-style clipping heuristics, which rely on reference policies and importance ratios. OPEFO instead operates directly on current-policy token gradients while preserving strict on-policy sampling and optimization. A plausible implication is that the method is best understood not as a separate RL algorithm, but as an entropy-balancing layer placed inside an existing GRPO-style RLVR training loop.

2. Entropy collapse and the token-level entropy-flow diagnosis

The central diagnosis begins with token-level entropy. For a state $T_\theta$0, token entropy is

$T_\theta$1

and policy entropy over a dataset is the average token entropy

$T_\theta$2

Low entropy corresponds to a near-deterministic next-token distribution; high entropy corresponds to more exploratory behavior (Xu et al., 12 May 2026).

The paper analyzes entropy collapse through a first-order approximation of entropy change under a policy-gradient step. Under a tabular-softmax, parameter-independent assumption in which each state-action pair $T_\theta$3 has its own logit parameter $T_\theta$4, Theorem 1 states that with learning rate $T_\theta$5,

$T_\theta$6

The expectation is strictly on-policy, and the quantity is used as a diagnostic for entropy dynamics rather than as a direct optimization target (Xu et al., 12 May 2026).

Using Qwen2.5-Math-7B on the DAPO-17K RLVR dataset, the paper reports that GRPO and its strict on-policy variant start around $T_\theta$7, then drop sharply in the first $T_\theta$8 updates, and converge near zero. The corresponding net entropy change per update, $T_\theta$9, is strongly negative in the first $y=(y_1,\dots,y_T)$0 steps and later approaches zero as entropy saturates at a very low level. The paper defines this pattern as entropy collapse: rapid early decline in $y=(y_1,\dots,y_T)$1 together with persistent negative net entropy flow (Xu et al., 12 May 2026).

To sharpen this account, tokens are partitioned into entropy-increasing and entropy-decreasing sets: $y=(y_1,\dots,y_T)$2 The net entropy change of an update is then

$y=(y_1,\dots,y_T)$3

Empirically, the paper finds that in early training

$y=(y_1,\dots,y_T)$4

so entropy-decreasing updates consistently outweigh entropy-increasing ones. This token-level imbalance is presented as a unified explanation of entropy collapse in RLVR with GRPO: the optimization process sharpens the token distribution more strongly than it broadens it (Xu et al., 12 May 2026).

A broader entropy-oriented reading is reinforced by adjacent work. “Understanding the impact of entropy on policy optimization” argues that higher entropy can smooth optimization landscapes and connect local optima in some environments (Ahmed et al., 2018), while “Policy Optimization Reinforcement Learning with Entropy Regularization” formalizes on-policy maximum-entropy policy gradients through the soft policy gradient theorem (Liu et al., 2019). These works do not define OPEFO, but they support the general premise that entropy dynamics are not merely exploratory side effects; they alter optimization geometry and update behavior.

3. The OPEFO update rule

OPEFO is defined by a simple modification of the on-policy gradient. After computing $y=(y_1,\dots,y_T)$5 for each token and partitioning tokens into $y=(y_1,\dots,y_T)$6 and $y=(y_1,\dots,y_T)$7, OPEFO introduces a balancing coefficient $y=(y_1,\dots,y_T)$8 and rescales token contributions as follows: $y=(y_1,\dots,y_T)$9 Entropy-increasing tokens are amplified by $x$0; entropy-decreasing tokens are down-weighted by $x$1. When $x$2, the update favors preserving or increasing entropy; the paper reports that $x$3 throughout training in practice (Xu et al., 12 May 2026).

The coefficient is chosen adaptively at the batch level through a zero-entropy-flow condition: $x$4 Let

$x$5

The paper gives

$x$6

In implementation, this is written with positive magnitudes: $K$3 and per-token policy-gradient terms are rescaled by $K$4 The paper emphasizes that OPEFO is just a scalar rescaling of policy-gradient terms per token, based on the sign of $x$7 (Xu et al., 12 May 2026).

This formulation distinguishes OPEFO from entropy bonuses and clipping heuristics. Entropy regularization adds

$x$8

with $x$9 in the reported experiments, but the paper argues that this acts coarsely and can over-correct, producing late-stage entropy explosion and excessive response lengths. PPO-style clipping and variants such as Clip-higher indirectly influence entropy by modifying importance-ratio thresholds, but they remain approximate rather than strict on-policy because they rely on a stale reference policy and importance ratios. OPEFO differs by directly balancing entropy flow without a reference model and without leaving the strict on-policy regime (Xu et al., 12 May 2026).

A useful contrast is provided by other entropy-control formulations in the literature. “PolicyFlow” uses a Brownian Regularizer that shapes the velocity field of a continuous normalizing flow so that policy density spreads without explicit entropy computation (Yang et al., 1 Feb 2026). “Flow Matching Policy with Entropy Regularization” derives an explicit entropy term through vector-field divergence for ODE-based flow policies (Gao et al., 18 Mar 2026). These are distinct mechanisms and different policy classes, but they illustrate the same broader research direction: modifying optimization to regulate how policy uncertainty evolves, rather than only adding static entropy bonuses.

4. Algorithmic integration and computational properties

Within a single on-policy RLVR update, OPEFO proceeds in eight steps. Prompts are sampled from training data $r(x,y)$0, $r(x,y)$1 responses per prompt are generated from current $r(x,y)$2, verifier rewards are computed, group-normalized advantages are assigned to tokens, standard per-token policy-gradient terms $r(x,y)$3 are formed, token-level entropy changes $r(x,y)$4 are estimated, tokens are partitioned into $r(x,y)$5 and $r(x,y)$6, the batch-level $r(x,y)$7 is computed, and the rescaled loss

$r(x,y)$8

is backpropagated. The paper presents this as a drop-in replacement for the plain policy-gradient step in GRPO-style RLVR; the rest of the training loop is unchanged (Xu et al., 12 May 2026).

The practical estimator for $r(x,y)$9 is left implementation-specific, but the code assumes per-token values are already available. The paper describes $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$0 as diagnostic rather than a direct optimization target. This distinction matters: OPEFO does not maximize entropy itself, nor does it optimize a learned entropy model. Instead, it uses local first-order entropy-change estimates to decide how much existing reward-driven gradients should be amplified or attenuated.

The reported computational overhead is negligible. Computing $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$1 and the required aggregations is $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$2. In the per-batch runtime comparison, approximate on-policy GRPO with 8 updates per rollout takes about $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$3 s, while strict on-policy OPEFO with 1 update per rollout and a larger batch takes about $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$4 s. OPEFO does not require extra networks such as value networks, entropy critics, or auxiliary models beyond the nominal RLVR setup (Xu et al., 12 May 2026).

The method also introduces no new main hyperparameters. $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$5 is computed rather than tuned, and only a small numerical stability constant such as $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$6 is required in the denominator. All ordinary RLVR hyperparameters—learning rate, batch size, rollout length, reward design—remain unchanged (Xu et al., 12 May 2026).

This low-overhead, strict on-policy character distinguishes OPEFO from several nearby methods. “Flow Matching Policy Gradients” constructs PPO-style ratios from flow-matching or diffusion denoising losses, using an ELBO surrogate rather than exact likelihoods (McAllister et al., 28 Jul 2025). “Relative Entropy Pathwise Policy Optimization” controls entropy and KL through a pathwise natural-gradient-style update in continuous control (Voelcker et al., 15 Jul 2025). These methods are technically different, but they underscore that entropy-aware policy optimization often becomes expensive once it requires auxiliary critics, replay buffers, divergence estimators, or nontrivial surrogate objectives. OPEFO’s claim is that token-level entropy balancing can be achieved with much lighter machinery in RLVR (Xu et al., 12 May 2026).

5. Empirical performance, exploration, and ablations

The reported empirical evaluation covers six mathematical reasoning benchmarks: AIME24, AIME25, AMC23, MATH500, Minerva Math, and OlympiadBench. The base models are Qwen2.5-Math-7B and Qwen3-4B-Base, trained on DAPO-17K, an RLVR-style reasoning dataset with verifiable rewards. Baselines include approximate on-policy GRPO, strict on-policy GRPO, entropy regularization, Clip-higher, Clip-Cov, and KL-Cov (Xu et al., 12 May 2026).

On Qwen2.5-Math-7B, the best baseline average is reported as Clip-Cov at $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$7 or KL-Cov at $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$8, while OPEFO reaches $$J(\theta)=\mathbb{E}_{x\sim\mathcal{D},\,y\sim\pi_\theta(\cdot|x)}[r(x,y)].$$9, a gain of $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$0 over the best baseline and $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$1 over strict on-policy GRPO at $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$2. On Qwen3-4B-Base, the best baseline is strict on-policy GRPO at $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$3 or KL-Cov at $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$4, while OPEFO reaches $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$5, a gain of $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$6 over strict GRPO (Xu et al., 12 May 2026).

Training dynamics are central to the method’s empirical identity. On Qwen2.5-Math-7B, GRPO and strict GRPO both show sharp entropy collapse toward zero. Entropy regularization eventually produces uncontrolled entropy growth after roughly step $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$7, accompanied by very long responses. Clip-higher partially mitigates collapse but exhibits oscillations. OPEFO is reported to maintain a smooth, stable entropy trajectory while achieving competitive, steadily rising reward and longer but controlled responses. The interpretation given in the paper is that OPEFO avoids both premature exploitation and late-stage entropy explosion (Xu et al., 12 May 2026).

The exploration effect is particularly visible in Pass@$$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$8. On Qwen2.5-Math-7B, OPEFO is best on Pass@32 across AIME24, AIME25, AMC23, and MATH500. On AIME24, the scaling table reports:

Method	Pass@8	Pass@16	Pass@32	Pass@64
GRPO	45.1	54.3	59.8	65.6
Strict GRPO	50.9	56.2	61.3	66.5
Clip-Cov	51.3	53.2	59.0	64.7
KL-Cov	50.5	54.1	60.1	65.5
OPEFO	52.5	56.5	62.4	68.4

The paper interprets the increasing margin with larger $$\nabla_\theta J(\theta)=\mathbb{E}_{x,y\sim\pi_\theta}\Bigg[\sum_{t=1}^{|y|}\nabla_\theta \log T_\theta(y_t\mid x,y_{<t})\,A_t\Bigg],$$9 as evidence of broader exploration and more diverse correct reasoning trajectories (Xu et al., 12 May 2026).

Ablations on $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$0 further isolate the mechanism. On Qwen2.5-Math-7B, strict on-policy GRPO with $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$1 obtains Avg $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$2; static scaling with fixed $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$3 yields $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$4; one-sided scaling on $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$5 only gives $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$6; one-sided scaling on $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$7 only gives $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$8; and full OPEFO with adaptive $$A_t=\frac{r(y)-\mathrm{mean}(r(y^{1:K}))}{\mathrm{std}(r(y^{1:K}))}.$$9 on both sets yields $K$0. The paper’s conclusion is that any nonzero balancing helps, adaptive scaling is better than static scaling, and balancing both entropy-increasing and entropy-decreasing tokens is best (Xu et al., 12 May 2026).

The observed training-time behavior of $K$1 is also consistent with the entropy-flow hypothesis. It stays positive throughout training, varies non-monotonically, decreases early, then increases later, and exhibits bounded oscillations. The paper interprets this as adaptation to non-stationary entropy flow and as confirmation that entropy-decreasing updates systematically dominate unless counterbalanced (Xu et al., 12 May 2026).

6. Theoretical status, assumptions, and related research

OPEFO’s formal core is diagnostic rather than fully convergent. The entropy-change formula in Theorem 1 relies on a tabular-softmax assumption with independent logits per state-action pair and a first-order Taylor expansion. The zero batch-level entropy-flow condition is presented as a sufficient stabilizing heuristic rather than a globally optimal entropy schedule. The paper explicitly notes that higher-order interactions in large transformers may not be captured, and no formal convergence theorem is given (Xu et al., 12 May 2026).

These caveats place OPEFO within a larger literature on entropy-regularized and entropy-structured policy optimization. The soft policy gradient theorem provides a principled on-policy maximum-entropy foundation in which the policy gradient contains $K$2 and the critic backup includes entropy reward (Liu et al., 2019). A complementary continuous-time perspective is given by a Fisher-Rao policy gradient flow for entropy-regularized MDPs, where the mirror-descent-like flow is globally well posed and converges exponentially under the regularized objective (Kerimkulov et al., 2023). OPEFO does not use these formulations directly, but they support the broader claim that entropy can be treated as part of the geometry of policy optimization rather than merely as an additive exploration bonus.

Several later papers develop adjacent notions of entropy flow without defining the same algorithm. “E-GRPO: High Entropy Steps Drive Effective Reinforcement Learning for Flow Models” argues that high-entropy denoising steps in flow-model RL carry more informative reward signals and therefore deserve special treatment in on-policy optimization (Zhang et al., 1 Jan 2026). “Entropy Importance Sampling Policy Optimization” uses predictive entropy to partition token sequences and allocate adaptive clipping ranges in LLM RL (Sheng et al., 29 Nov 2025). “Agentic Entropy-Balanced Policy Optimization” balances entropy in both rollout branching and clipping behavior for web agents (Dong et al., 16 Oct 2025). These methods are not OPEFO in the narrow RLVR sense of (Xu et al., 12 May 2026), but they suggest a broader family of entropy-routing strategies in which uncertainty determines where sampling effort and gradient budget should flow.

In that broader sense, OPEFO can be read as one concrete instance of an emerging research pattern: identify localized entropy dynamics, diagnose where optimization is systematically collapsing uncertainty, and alter the update so that entropy-preserving behavior is not overwhelmed by entropy-reducing gradients. The specific contribution of OPEFO is to do this in token-level RLVR with strict on-policy updates, no reference policy, no importance ratios, negligible computational overhead, and empirically improved stability and final reasoning performance on mathematical benchmarks (Xu et al., 12 May 2026).