Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Abstract: Reinforcement learning algorithms for LLMs are largely distinguished by their variance reduction strategy. Group-relative methods like GRPO reduce gradient variance by sampling multiple rollouts per prompt, but provide only sequence-level credit. Training is also blocked by straggler rollouts, reducing throughput and increasing off-policyness. Learned value functions theoretically address both problems, providing token-level advantages without requiring large groups. However, additional infrastructure engineering challenges combined with the practical success of critic-free methods have made it difficult to justify their inclusion in RL pipelines. We propose two complementary strategies to improve the performance of value function RL: 1) Privileged Value Functions (PVF) which provide an elegant mechanism to inject additional task-relevant token-level signal without biasing the policy objective; 2) TETHER, a baseline that adaptively interpolates between group-relative and value baselines depending on the value function accuracy. Across several reasoning tasks, both strategies consistently improve over the standard value function baseline, and are competitive with or outperform mean-baseline GRPO.

• The paper introduces PVFTeal, which conditions value functions on admissible training-time information and outperforms ordinary value and group baselines across Reasoning Gym, CodeIO, and Sudoku.
• The paper presents TetherPurple, which adaptively blends leave-one-out group baselines with token-level values, improving over ordinary critics across four evaluated settings and matching or exceeding group baselines on several tasks.
• The paper shows that privileged critics can improve variance reduction and long-horizon credit assignment, but added compute, limited model scale, and non-compute-matched experiments leave efficiency and generalization open questions.

The paper addresses a central design trade-off in reinforcement learning for LLMs: group-relative methods such as GRPO provide a robust, critic-free variance-reduction mechanism but assign only sequence-level credit, whereas learned value functions provide token-level advantages at the cost of additional model infrastructure and possible critic miscalibration. “Le Critique: Privileged Value Functions for LLM Reinforcement Learning” (2608.16739) proposes two methods intended to make value-function-based RL more competitive: Privileged Value Functions (PVFTeal), which condition critics on information unavailable to the policy, and TetherPurple, which adaptively combines learned token-level values with leave-one-out group baselines.

Problem setting and motivation

For a rollout trajectory with terminal return $R_i$, policy-gradient methods update token probabilities using an advantage estimate. Group-relative algorithms estimate this advantage by comparing multiple responses sampled for the same prompt. The standard group-mean baseline repeats a sequence-level quantity across all tokens, while the leave-one-out (LOO) estimator excludes the current trajectory from the baseline and therefore avoids the direct dependence of the baseline on its own realized return.

This strategy is attractive because it avoids training and serving a separate critic. Its limitations are equally important. First, every token in a response receives essentially the same credit signal, even when the response contains long stretches of reasoning with heterogeneous causal relevance. Second, group-based training requires multiple rollouts per prompt and must often wait for the slowest trajectory. In asynchronous systems, this synchronization can increase rollout staleness and off-policy error. These costs become more consequential as response lengths and environment horizons increase.

A learned value function instead estimates expected return from each partial prefix. In the terminal-reward setting studied here, the critic predicts the expected final reward conditioned on the prompt and generated prefix. The resulting token-level baseline can reduce gradient variance and distinguish early, intermediate, and late states. However, an inaccurate critic can be worse than a group baseline, particularly during early training. The paper therefore treats the problem not as a choice between “critic” and “no critic,” but as a problem of constructing a baseline that exploits whichever information source is currently more reliable.

Privileged Value Functions

The first contribution is the Privileged Value Function, or PVFTeal. The policy remains conditioned only on its ordinary observable history $h_{i,t}$, whereas the critic additionally receives training-time context $z_{i,t}$. The critic estimates

$$V^\pi(h_{i,t},z_{i,t}) = \mathbb{E}[R_i \mid h_{i,t},z_{i,t}],$$

and the policy uses $R_i - V_\phi(h_{i,t},z_{i,t})$ as its advantage.

The essential distinction from self-distillation is that privileged information enters only through a baseline. It does not define a new policy target or alter the policy objective. Under the paper’s admissibility condition, the privileged variable must not depend on the current token conditional on its history. Thus, a fixed reference solution, an oracle answer, or independently sampled sibling trajectories may be used; future tokens from the current trajectory, its realized reward, and subsequent verifier feedback may not. With Monte Carlo advantages, this preserves the expected policy gradient while potentially reducing its variance.

The theoretical basis is conditional-variance reduction. An optimal predictor conditioned on $(h,z)$ cannot have greater mean-squared prediction error than an optimal predictor conditioned on $h$ alone. This guarantee applies to the population-optimal critic, not automatically to a finite neural critic: irrelevant or overly complex context can make optimization harder, and privileged information helps only when the model can exploit it. The paper is appropriately explicit about this distinction.

The proposed mechanism generalizes several kinds of information. Reference answers can convert value prediction from solving an entire task into assessing whether a partial trajectory is consistent with a known target. In tasks without references, the critic can condition on the other responses in the rollout group and their returns. This makes the LOO group baseline itself interpretable as a particularly restricted, training-free privileged critic: it uses the sibling returns but discards the current partial trajectory and all structure in the sibling responses.

The conceptual architecture is summarized below.

Figure 1: A privileged critic conditions on the policy history and admissible training-time context, such as a reference solution or leave-one-out sibling trajectories.

The approach is closely related to asymmetric actor-critic methods in control, but the paper’s contribution is its application to LLM policy-gradient training, where the privileged signal can be a reference answer, verifier specification, or group-level trajectory information. The distinction from self-distillation is substantive. A privileged teacher can use retrospective feedback to define a new token distribution, but that changes the optimization objective and may expose information the student cannot reproduce. PVFTeal uses the same information only to improve a control variate. It therefore cannot use current-trajectory hindsight feedback, but it retains the original RL objective under the stated conditions.

Empirical evaluation of privileged critics

The PVFTeal experiments compare three baselines: the group-mean baseline, an ordinary token-level value function, and a privileged value function with the same training configuration as the ordinary critic. The policy is based on Qwen3-4B-Instruct-2507. All value-based methods use Monte Carlo targets and $\lambda=1$, deliberately isolating the effect of variance reduction from bias introduced by truncated GAE.

The evaluation spans Reasoning Gym, CodeIO, and Sudoku. Reasoning Gym is tested both without groups ($K=1$) and with groups of eight. CodeIO uses groups of four, with the critic receiving the other three responses and their rewards. Sudoku uses groups of four, and the critic receives the complete solved grid. The training horizons are nontrivial: up to 8,192 tokens for Reasoning Gym, 12,288 total input-plus-completion tokens for CodeIO, and 32,768 tokens for Sudoku.

PVFTeal is reported as the best-performing method in all four settings.

Figure 2: PVFTeal improves seed-averaged training reward relative to ordinary value and group-mean baselines across Reasoning Gym, CodeIO, and Sudoku.

The result is notable in CodeIO because the privileged context is not task-specific oracle information. The critic instead receives the other group responses and their returns. Ordinary VF slightly underperforms the group-mean baseline, whereas the leave-one-out-conditioned PVFTeal surpasses both, with the advantage increasing over training. This supports the paper’s claim that group information can be used more effectively by a learned token-level critic than by a scalar group mean.

The largest qualitative gain occurs in Sudoku. The paper attributes this to the long multi-turn horizon and the structure of the task: evaluating an intermediate grid requires determining whether it remains compatible with a globally consistent solution. An ordinary critic must implicitly solve much of that inference problem. Supplying the solved grid makes the value-estimation problem substantially easier. The implication is that privileged conditioning is especially useful when the policy’s observable history is highly aliased with respect to eventual success.

Reasoning Gym provides a more qualified result. With $K=1$, both ordinary VF and PVFTeal improve at similar rates initially, but the ordinary critic plateaus earlier. With $h_{i,t}$0, the improvement from privileged conditioning is smaller, and both value baselines outperform the group mean. This suggests that group sampling and privileged value prediction are not mutually exclusive sources of variance reduction: a value method can still benefit from groups, while the incremental value of privileged information depends on the task and rollout configuration.

The paper also measures explained variance, defined from residual reward variance after subtracting value predictions. PVFTeal explains more return variance than ordinary VF in every environment. The ordering of explained-variance improvements corresponds qualitatively to the policy-reward differences: the smallest critic gap occurs in Reasoning Gym with $h_{i,t}$1, while the largest occurs in Sudoku. Because $h_{i,t}$2, residual value error is directly the variance of the Monte Carlo advantage up to the policy-gradient weighting, making explained variance a relevant diagnostic rather than merely an auxiliary regression metric.

The authors discuss group size carefully. Increasing $h_{i,t}$3 reduces the sampling error of the prompt-level baseline, but it also increases the number of trajectories available for averaging policy-gradient contributions. The latter benefit is not unique to group-mean methods. Moreover, improved prompt-level estimation may have limited effect after the first token because it cannot represent within-response changes in expected return. The paper does not experimentally establish behavior at substantially larger $h_{i,t}$4; its argument remains analytical and the tested groups are only $h_{i,t}$5.

TetherPurple: adaptive interpolation

The second contribution, TetherPurple, addresses the failure mode of a poorly fitted critic. It combines the LOO baseline $h_{i,t}$6 with the ordinary token-level value $h_{i,t}$7:

$h_{i,t}$8

At $h_{i,t}$9, the method recovers the LOO group baseline; at $z_{i,t}$0, it recovers the ordinary value baseline. Intermediate values use the group estimate to stabilize critic errors while retaining token-level variation.

The mixture coefficient is fitted by least squares to predict observed Monte Carlo returns. Crucially, the coefficient used for a batch is estimated from previous data, not from that batch’s own returns. After advantages for batch $z_{i,t}$1 are computed, the current batch is used to estimate a new coefficient, which is exponentially smoothed and applied only to $z_{i,t}$2. This temporal separation is necessary: fitting the baseline on the same returns used for the policy update would make the baseline depend on the current trajectory outcomes and could bias the policy gradient.

Figure 3: TetherPurple interpolates between the LOO group baseline and token-level value predictions using an adaptively fitted mixture coefficient.

At the population level, the method minimizes squared return-prediction error along the line segment connecting the two baselines. Consequently, an oracle mixture cannot have higher mean-squared prediction error than the better endpoint. This dominance is a statement about return prediction, not directly about policy-gradient covariance. The paper acknowledges that the gradient-optimal baseline weights residuals by the squared score-function norm, whereas TetherPurple does not. Thus, its regression objective is a practical surrogate rather than the exact minimum-variance policy-gradient solution.

The experimental comparison includes Reasoning Gym, CodeIO, Sudoku, and MiniF2F. MiniF2F uses Qwen3.5-4B because Qwen3-4B-Instruct-2507 produced no useful training signal. TetherPurple uses group sizes of four or eight and an EMA decay of $z_{i,t}$3.

TetherPurple improves over the ordinary value baseline in all four settings.

Figure 4: TetherPurple consistently improves training reward over the ordinary value baseline and approaches or exceeds the group-mean baseline depending on the task.

The comparison with the group mean is task-dependent. TetherPurple outperforms Mean on Reasoning Gym and MiniF2F, matches it on CodeIO, and narrows—but does not eliminate—the gap on Sudoku. This result is more restrained than the PVFTeal result: adaptive interpolation makes value-based RL more reliable, but it does not universally dominate a strong group baseline.

Figure 5: Final-window reward aggregates show the cross-task comparison between privileged critics, ordinary value functions, group baselines, and TetherPurple.

The dynamics of $z_{i,t}$4 provide an additional diagnostic. All runs begin at $z_{i,t}$5, so early policy updates use the LOO baseline while the critic is poorly calibrated. The coefficient then moves away from zero as the critic predicts return variation beyond the group baseline. Its convergence is strongly task-dependent. Sudoku is particularly informative: it converges to the largest value-function weight despite the weakest performance of the standalone VF baseline. The paper hypothesizes that early reliance on the poorly fitted critic causes compounding policy-learning failures; TetherPurple avoids this initial failure mode while eventually incorporating more critic information. This interpretation is plausible, but the experiments do not isolate early-training causal effects from other factors such as critic staleness or policy distribution shift.

TetherPurple can also be understood as a restricted PVFTeal. The LOO return is privileged information, compressed into a scalar baseline, and combined with the ordinary value prediction. The more expressive CodeIO PVFTeal instead conditions directly on sibling responses and returns, allowing the critic to learn which aspects of the group are relevant to the current prefix. This establishes a continuum between hand-designed group statistics and learned privileged critics.

Asynchronous value-function infrastructure

The paper treats systems design as part of the contribution because value functions introduce an additional inference and training path. The implementation separates the value evaluator, which serves predictions for advantage computation, from the value trainer, which updates critic parameters from replayed trajectories. The evaluator periodically adopts the latest published trainer weights while continuing to serve requests.

Figure 6: Value evaluation and value training proceed asynchronously, with evaluated trajectories entering a bounded FIFO replay buffer.

The infrastructure uses a FIFO buffer of 256 trajectories and limits each trajectory to two replay uses. Trajectories enter replay only after evaluation and advantage computation. This ordering prevents the value trainer from altering the critic used to score the same batch retrospectively. Dedicated evaluator and trainer replicas permit value inference to overlap optimization; colocated deployment saves hardware but introduces blocking between inference and training.

The system exploits an asymmetry between policy and value learning. Policy updates are sensitive to stale trajectories and require importance correction, whereas value learning can generally tolerate a wider replay window. This permits bounded reuse of value data without imposing a synchronization barrier on policy training. Nevertheless, the setup adds substantial computational cost. In the reported configurations, value-backed runs generally use two additional nodes relative to group-mean runs: one value trainer and one dedicated value evaluator. The experiments match inference trajectories rather than total compute, so their reward comparisons do not constitute compute-normalized scaling results.

The value model is initialized from the base policy with a randomly initialized value head and receives only 20 warmup updates before policy training. For binary rewards, the implementation uses binary cross-entropy while retaining continuous predicted expectations in $z_{i,t}$6. The authors report a small early advantage over MSE in preliminary experiments but do not provide a systematic loss-function ablation.

Limitations and open questions

The empirical scope is limited to approximately 4B-parameter models and reasoning environments with verifiable rewards. Although Sudoku reaches 32,000-token trajectories, the study does not evaluate long-horizon agentic tasks with richer action spaces, tool use, or persistent state. The number of random seeds is also small: two for Reasoning Gym and three for most other experiments. Consequently, the consistent direction of the reported effects is informative, but precise task-level effect sizes and robustness across model families remain uncertain.

The comparisons are not compute-matched between group-mean and value-based methods. Value-backed runs require additional accelerator allocations and a more complex asynchronous pipeline. The paper therefore demonstrates optimization-quality differences under matched policy settings, not superior reward per unit of total hardware or wall-clock compute. Whether privileged critics remain advantageous after accounting for evaluator inference, trainer updates, replay utilization, and rollout throughput is an open empirical question.

The main experiments fix $z_{i,t}$7. An ablation on Reasoning Gym reports that reducing $z_{i,t}$8 to approximately $z_{i,t}$9—chosen so that a first-token advantage retains roughly 40% of the terminal signal over an 8,192-token response—substantially improves both VF and PVFTeal. This is a consequential result because the numerical change in $$V^\pi(h_{i,t},z_{i,t}) = \mathbb{E}[R_i \mid h_{i,t},z_{i,t}],$$0 is small but its effect compounds over long sequences. It also means that the headline comparisons do not identify the best value-function configuration. A systematic study must separate target bootstrapping from advantage estimation and account for sequence length.

The admissibility condition for PVFTeal is another boundary condition rather than a universal guarantee. Reference answers and independent sibling trajectories are valid baseline context under the paper’s formulation, but feedback generated by the current trajectory is not. In practical environments, distinguishing independent privileged context from information causally downstream of the current action may be nontrivial. The paper leaves open whether useful hindsight signals can be incorporated while preserving an unbiased policy-gradient interpretation, or whether they should instead be treated explicitly as a changed objective such as self-distillation.

Finally, TetherPurple estimates a single coefficient across all token positions. The paper proposes token-bucketed coefficients as a natural extension, motivated by the differing difficulty of value prediction at early and late prefixes, but does not evaluate it. Nor does TetherPurple optimize the exact gradient-variance objective because its regression criterion omits score-function weighting. These are specific unresolved questions about whether better baseline prediction translates reliably into better optimization under realistic token-level gradient statistics.

Conclusion

The paper presents a coherent argument for reconsidering learned critics in LLM reinforcement learning. PVFTeal uses admissible privileged context to improve token-level return prediction without changing the policy objective, and it outperforms ordinary value and group baselines across the reported Reasoning Gym, CodeIO, and Sudoku experiments. TetherPurple addresses critic unreliability by adaptively combining LOO and value baselines, improving over ordinary VF in all four of its evaluated settings and matching or exceeding the group mean on several tasks.

The contributions are strongest as methods for integrating value functions into existing group-relative pipelines without requiring an abrupt replacement of their more reliable baseline. Their significance is conditioned by compute overhead, small-scale evaluation, limited seed counts, and the unresolved choice of long-horizon advantage parameters. The paper’s central empirical claim is therefore specific: under the tested reasoning-task configurations, privileged conditioning and adaptive group–value interpolation make value-backed LLM RL more effective than an ordinary critic and, in several cases, competitive with strong critic-free baselines (2608.16739).