Beyond Uniform Credit Assignment: Selective Eligibility Traces for RLVR

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has become a key approach for improving the reasoning abilities of LLMs. However, widely used critic-free algorithms such as Group Relative Policy Optimization (GRPO) necessitate a ``uniform credit assignment'' assumption that indiscriminately broadcast trajectory-level advantages, hindering learning efficiency by failing to distinguish critical reasoning steps. To address this limitation, we propose Selective Eligibility Traces (S-trace). Grounded in the intuition of partial trust region preservation, we initially introduce P-trace as a sample-efficient, critic-free eligibility traces method, upon which we build S-trace, implementing a sparse eligibility traces mechanism to further mitigate variance and achieve fine-grained credit assignment by selectively masking low-entropy tokens. Theoretically, we contextualize the recent Group Sequence Policy Optimization (GSPO) method within the critic-free eligibility traces framework, identifying it as a special instance of the eligibility traces method operating under uniform credit assignment. Experiments demonstrate that S-trace not only outperforms GRPO, showing gains of 0.49\% on Qwen3-1.7B and 3.16\% on Qwen3-4B, and maintaining a robust 2.98\% improvement when scaled further to Qwen3-8B in average pass@16, but notably achieves this with simultaneously higher sample and token efficiency.

• The paper introduces P-trace, a critic-free eligibility-trace method that generalizes GRPO by replacing uniform token credit with recency-weighted importance signals.
• The paper proposes S-trace, which masks low-entropy trace contributions while preserving each token’s direct gradient, improving stability and outperforming random masking.
• The paper reports S-trace gains of up to 2.98 percentage points in average pass@16 over GRPO on Qwen3-8B, alongside faster training and shorter responses, though broader transfer remains untested.

Motivation and problem statement

Reinforcement Learning with Verifiable Rewards (RLVR) has become the dominant post-training paradigm for improving LLM reasoning, with critic-free algorithms such as Group Relative Policy Optimization (GRPO) serving as the de facto standard. The central limitation the paper identifies is GRPO's "uniform credit assignment" assumption: a trajectory-level advantage, standardized across a group of sampled responses, is broadcast identically to every token. For mathematical reasoning traces spanning thousands of tokens, this indiscriminately reinforces routine tokens alongside critical reasoning steps, injecting substantial gradient noise and leaving the dense temporal structure of reasoning chains unexploited.

The authors address this by reviving the actor-only eligibility trace paradigm of Kimura and Kobayashi (1998), which has largely been confined to value-function estimation in classical RL. Their proposal proceeds in two stages: P-trace, a sample-efficient critic-free eligibility traces method derived from a reformulation of the PPO policy gradient, and S-trace, which sparsifies the trace computation via entropy-based masking to control variance and improve generalization (2605.05965).

From PPO gradients to eligibility traces

The theoretical foundation is Proposition 1, which shows that—absent clipping and the min operator—the PPO policy gradient equals an expectation over products of TD errors $\delta_t$ and actor eligibility traces $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$. This establishes PPO itself as an off-policy actor-only eligibility traces method, extending the on-policy result in GRPO($\lambda$) (Parthasarathi et al., 30 Sep 2025). Replacing the TD error with the group-level advantage $\hat{A}_i = \delta_1$ (an approximation bounded by Lemma 1 of GRPO($\lambda$)) yields a critic-free estimator.

The key algorithmic step in P-trace is approximating historical importance weights $r_{<t}(\theta)$ inside the trace term by the current weight $r_t(\theta)$, justified by a "proximal assumption" that off-policy deviation is roughly uniform across tokens. This induces what the authors call partial trust region preservation: when an ancestral token's ratio falls outside the clip range but a causally dependent downstream token remains unclipped, the learning signal for the ancestor survives through the trace coefficient. This contrasts with standard clipped objectives, where an out-of-bound ratio nullifies the token's gradient entirely.

The mechanism is implemented through an eligible importance weight (EIW):

$$\tilde{r}_{i,t}(\theta)=\text{sg}\!\left[\frac{r_{i,t}(\theta)}{r^\lambda_{i,t}(\theta)}\right]\cdot r^\lambda_{i,t}(\theta)$$

where $\text{sg}[\cdot]$ is the stop-gradient operator and $r^\lambda_{i,t}$ is a recency-weighted product of per-token likelihood ratios with exponents $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$0. Numerically EIW matches the standard importance ratio, but its differentiation produces exactly the eligibility-trace gradient structure. Setting $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$1 recovers GRPO, making P-trace a strict generalization.

A notable theoretical contribution contextualizes GSPO (Zheng et al., 24 Jul 2025) within this framework: the terminal-token objective of trace-style GRPO($$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$2 differs from GSPO's sequence-level objective only in that uniform exponents $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$3 replace the geometric decay $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$4. GSPO is thus characterized as an eligibility traces method operating under uniform credit assignment—a claim supported empirically by comparisons showing recency-based methods (GRPO($$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$5), S-trace) achieve higher average pass@16 (52.18 and 52.17 vs. 49.72 for GSPO on Qwen3-4B) with shorter mean response lengths.

Selective sparsification

Dense traces risk propagating credit to noisy tokens; the paper's own experiments confirm that GRPO($$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$6)-0.9 fails to beat GRPO on any of five benchmarks at 1.7B scale, and larger $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$7 degrades performance further. S-trace addresses this by introducing a Bernoulli mask $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$8 that zeroes trace contributions from tokens whose entropy falls in the bottom $$e_t = r_t(\theta)\nabla_\theta\log\pi_\theta(o_t|x,o_{<t}) + \gamma\lambda e_{t-1}$$9 fraction of each rollout, with $\lambda$0 following the empirical 80/20 rule on high-entropy "forking tokens" [2505.xxxx, Wang et al. 2025].

Proposition 2 formalizes the variance benefit: under assumptions of uncorrelated temporal gradients and zero-mean, uniformly bounded historical terms,

$\lambda$1

strictly smaller than the P-trace variance for $\lambda$2. S-trace therefore interpolates between low-variance GRPO ($\lambda$3) and high-variance P-trace ($\lambda$4). An important implementation detail is the leave-own-out (LOWO) variant, which exempts the current token from its own mask so that every token retains its intrinsic gradient component. This isolates the effect of sparse traces from sparse policy gradients—the regime studied in prior work—and guarantees degeneration to GRPO as $\lambda$5. The ablation is decisive: replacing the entropy mask with a random mask drops average pass@16 on Qwen3-4B from 52.17 to 47.04, confirming that gains stem from informed selection rather than stochastic regularization alone.

Empirical results

Training uses DAPO-Math-14k within veRL, evaluating pass@16 on MATH500, AIME24/25, AMC23, Minerva, and (at 8B) BeyondAIME. Key results:

Model	Method	Avg. pass@16
Qwen3-1.7B	GRPO	39.92
Qwen3-1.7B	S-trace-0.9	40.41
Qwen3-4B	GRPO	49.01
Qwen3-4B	GRPO($\lambda$6)-0.9	52.18
Qwen3-4B	S-trace-0.9	52.17
Qwen3-8B	GRPO	54.64
Qwen3-8B	S-trace-0.9	57.62

On Qwen3-8B, S-trace achieves a 2.98% improvement over GRPO, with particularly large gains on AIME24 (54.82 vs. 44.99) and BeyondAIME (27.84 vs. 17.68). These accuracy gains are accompanied by higher sample efficiency—the 4B models reach GRPO's plateau reward in roughly half the training iterations—and consistently shorter response lengths, indicating better token efficiency. At 8B scale, GRPO($\lambda$7)'s reward curve overlaps GRPO entirely, while S-trace preserves its efficiency advantage, supporting scalability claims.

Analysis of clip fractions provides mechanistic insight: GRPO($\lambda$8) exhibits persistently elevated clipping (up to 24× higher than P-trace at step 200 when $\lambda$9), functioning as an inadvertent stochastic-dropout regularizer that stabilizes training at the cost of signal retention. P/S-trace maintain low clip fractions, preserving richer gradients—an alignment with CISPO's principle of retaining truncated importance weights rather than silencing gradients.

Limitations and open questions

The paper is candid about several constraints. First, P-trace suffers severe training instability: extreme gradient-norm spikes preclude reporting its results on Qwen3-1.7B even at $\hat{A}_i = \delta_1$0, and it destabilizes at $\hat{A}_i = \delta_1$1 on Qwen3-4B. S-trace mitigates but does not eliminate this; the authors acknowledge that partial trust region preservation "inevitably compromises strict trust region boundaries." Second, the variance analysis rests on strong assumptions—uncorrelated temporal gradients and zero-mean historical terms—that are approximations rather than guarantees. Third, the selective rate is fixed at $\hat{A}_i = \delta_1$2 throughout, with sensitivity to this hyperparameter left unexplored. Fourth, evaluation is confined to mathematical reasoning with Qwen3 backbones; transferability to other domains or model families is asserted but not demonstrated. Finally, the authors explicitly leave open why recency-based credit assignment yields strictly superior token efficiency to uniform assignment, and how sparse traces might be constructed on more principled theoretical grounds.

Conclusion

This paper reframes critic-free RLVR algorithms as instances of actor-only eligibility traces, derives a practical recency-weighted variant (P-trace), stabilizes it through entropy-based selective sparsification (S-trace), and subsumes GSPO as the uniform-decay special case. The empirical record—consistent pass@16 gains across three model scales coupled with reduced training samples and response lengths—supports the claim that non-uniform, temporally structured credit assignment improves both optimization efficiency and generalization relative to uniform broadcasting, provided the trace is selectively masked.