Value-Gradient Hypothesis of RL for LLMs

Abstract: Reinforcement learning substantially improves pretrained LLMs, but it remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains. We develop a value-gradient perspective of critic-free RL for LLM post-training. First, under a differentiable rollout and additive-noise parameterization, we show that the actor update is value-gradient-like in expectation: the backward pass propagates costates whose conditional expectation equals the value gradient. Second, for discrete transformer policies, we show that autodifferentiation through attention produces empirical costates that approximate this value signal, with an error controlled by the sampling gap and policy entropy. These results motivate a decomposition of RL impact into value gradient signal and reachable reward headroom, yielding a criterion for when RL should be most effective along a pretraining trajectory.

• The paper introduces a theoretical framework showing that critic-free RL methods, via continuous relaxation and additive noise, propagate a value-gradient-like signal.
• The paper empirically validates that transformer attention mechanisms can effectively transport credit across token trajectories, with costate approximation error scaling with policy entropy.
• The paper bridges score-function and pathwise-derivative estimators, yielding a practical checkpoint selection criterion for predicting RL impact in LLM post-training.

Value-Gradient Hypothesis in Critic-Free RL for LLMs

Formulation and Theoretical Analysis

The work presents a rigorous perspective on critic-free RL algorithms, notably PPO/GRPO, in the context of LLM post-training. The central hypothesis posits that critic-free methods are not devoid of value-signal: the actor-update in these RL protocols, when analyzed via a continuous relaxation and additive-noise parameterization, propagates a value-gradient-like signal. Specifically, in a differentiable rollout setting, backpropagation through time (BPTT) transports costates whose expected value matches the state-value gradient. For discrete transformer policies, empirical evidence demonstrates that autodifferentiation through the attention mechanism yields costates approximating the true BPTT value-gradient signal, with an approximation error governed by the entropy of the policy and the discrete-sampling gap.

A crucial theoretical development is the equivalence between score-function (SF) and pathwise-derivative (PD) estimators in expectation under a shift/additive-noise policy. This bridges canonical RL gradient estimators and continuous-backpropagation approaches. The analysis reveals that, despite the absence of an explicit critic, the backward pass in critic-free RL effectively transports credit across the token-level trajectory, making them value-gradient-like in expectation.

Discrete Transformers and Attention-Based Credit Transport

In practical transformer-based LLMs, credit transport is facilitated by attention mechanisms, which provide differentiable pathways across token positions. The discrete nature of token sampling creates non-differentiable boundaries, leading to an empirical costate mismatch with the exact BPTT costate. The paper formalizes the structure and recursion of empirical costates in this setting, quantifying the error due to sampling gaps. The key result demonstrates that this error scales with policy entropy, behaving as $O(H_t \log |V|)$ where $H_t$ is local token entropy and $|V|$ the vocabulary size.

Attention is shown to compensate for the discrete bottleneck: even as sampling loses some temporal credit flow, the multi-layer, multi-head attention network enables hidden-state credit assignment across the entire context. Proposition 3 provides a rank and magnitude bound for the attention-pathway Jacobian, indicating that distributed attention leads to richer temporal credit signals. When the model operates in a low-entropy regime typical of well-trained LLMs, the approximation gap becomes negligible, ensuring the empirical costate remains a powerful estimator of the state-value gradient.

RL Impact Decomposition and Predictive RL Readiness Criterion

The value-gradient hypothesis motivates a two-factor decomposition of RL impact for LLM checkpoint selection:

• Usable Value-Gradient Signal: Quantified by the similarity between the empirical costate and the true value-gradient, measuring how much credit-assignment capacity is present in the actor update.
• Reachable Reward Headroom: Defined as the difference between the expected maximal reward under trajectory reweighting and the current reward, reflecting the latent reward improvement potential in the checkpoint's trajectory distribution.

This framework yields a checkpoint selection criterion: RL is most effective when a model has both strong value-gradient signal and substantial reward headroom. The main predictive statement formalizes RL gain as proportional to the product of these two metrics. Explicitly, the optimal checkpoint occurs where their product is maximal, providing a practical method for RL readiness prediction during pretraining.

Empirical Validation and Numerical Results

Experiments employ OLMo-2 checkpoints and synthetic differentiable reward tasks to precisely measure costate approximation and RL impact predictions. The entropy-bound for the costate approximation error is empirically tight, confirming the theoretical scaling of the discrete-sampling gap with policy entropy.

For RL-impact prediction, costate-based signal and headroom terms are computed and used to predict checkpoint-wise RL gain. The combined impact score exhibits strong rank-order correlation with observed RL improvement (Spearman $\approx 0.70$), outperforming predictors that rely solely on current reward or costate signal. These results robustly support the hypothesis: RL gain is checkpoint-specific and determined by both trajectory competence and value-gradient quality.

Implications for RL and LLM Post-Training

The analysis reframes the success of critic-free RL in LLMs as a consequence of hidden-state computation in continuous space and transformer attention, rather than an exception to classical credit-assignment theory. It challenges the notion that scalar episodic baselines or group-normalized statistics alone explain RL efficiency in LLMs, showing that the backward pass's value-gradient properties are fundamental.

This perspective informs both the theory and practice of RL for LLMs. Practically, it yields a predictive mechanism for checkpoint selection, promising more efficient post-training by targeting the phase of maximal RL impact. Theoretically, it encourages further investigation into the interplay between model entropy, attention structure, and reward landscape for optimizing RL signals.

Future directions will likely include refining RL algorithms to explicitly leverage attention-mediated credit transport and low-entropy regimes, developing actor-free or value-gradient-driven policies, and extending the costate framework to more complex reward structures and environments. Additionally, further integration with value-gradient flow-based learning methods may enable policy architectures that exploit value information more directly.

Conclusion

The value-gradient hypothesis rigorously explains the mechanism underlying the effectiveness of critic-free RL methods for LLM post-training. By identifying the costate as a Monte Carlo estimator of the value gradient propagated through continuous and attention-based transformer computation, it establishes the conditions under which RL yields maximal gains. Empirical results substantiate these claims, revealing correlations between costate-based RL readiness scores and realized RL improvement. The theoretical framework paves the way for checkpoint-dependent, signal-driven RL strategies, enhancing the efficiency and efficacy of LLM post-training protocols (2605.21654).