Best Practice Critic Optimization

Abstract: Group-based reinforcement learning methods such as GRPO for LLMs avoid training a critic by sampling multiple responses for each prompt. A reliable critic could instead estimate token-level advantages from one response, but standard critic-based training recipes are often unstable. We study this instability and develop Best Practice Critic Optimization (BPCO), a recipe that combines DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training, BPCO can also condition it on reward-defining information, such as a reference answer or grading rubric, that is hidden from the policy. Controlled experiments isolate the effect of each design choice. Across mathematical reasoning tasks with models ranging from 1.5B parameters to 30B-A3B mixtures of experts, BPCO improves a strong critic-based baseline consistently, and matches or exceeds a group-based baseline while sampling one response per prompt. The same recipe also improves learning with rubric-based rewards. These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation. Code is available at https://github.com/QPHutu/golden_critic.

• The paper introduces Best Practice Critic Optimization (BPCO), a methodology that addresses instability in reinforcement learning with large language models (LLMs) by aligning policy objectives, critic parameterization, value targets, advantage scaling, and response-length dependencies.
• BPCO demonstrates improved stability and efficiency through components such as reward-range bounded value predictions, unbiased Monte-Carlo critic targets, and length-adaptive GAE, outperforming both critic and group-based optimization methods with a single response per prompt.
• The study shows that privileged information, providing reward-defining insights to the critic during training, enhances performance but requires careful handling to avoid overfitting and maintain generalization.

The paper presents Best Practice Critic Optimization (BPCO), a single-rollout actor–critic recipe for reinforcement learning with LLMs. Its central claim is that the instability commonly attributed to critic-based LLM RL is not intrinsic to value estimation; rather, it results from mismatches among the policy objective, critic parameterization, value targets, advantage scaling, and response-length dependence. BPCO addresses these mismatches by combining Divergence Proximal Policy Optimization (DPPO), reward-range-bounded value predictions, unbiased Monte Carlo critic targets, unnormalized advantages, and length-adaptive GAE. It further introduces privileged critic inputs, allowing the training-only critic to access reward-defining information unavailable to the policy. Across mathematical reasoning and rubric-based reward settings, the method improves a strong critic baseline and matches or exceeds group-based optimization while sampling only one response per prompt (2608.23566).

Problem formulation and motivation

Outcome-based RL for autoregressive LLMs assigns a scalar reward to a complete response and must convert that response-level signal into token-level policy updates. Group-based methods such as GRPO avoid learning a value function by sampling multiple responses for each prompt and comparing their rewards. Their principal advantage is implementation simplicity and robustness to critic error. Their cost is substantial: several rollouts are required per prompt, and the resulting advantage is typically constant across all tokens in a response.

A critic-based method instead estimates the expected return conditioned on each response prefix. This permits token-level credit assignment from a single rollout, but it introduces several failure modes. Standard PPO applies a common probability-ratio clipping threshold to all sampled tokens. In a large vocabulary, that creates unequal absolute probability changes: a fixed ratio change can correspond to a very small probability movement for a low-probability token and a much larger movement for a high-probability token. DPPO replaces this ratio-based constraint with an approximately uniform absolute-probability constraint on the sampled token, addressing the policy-side asymmetry identified in prior work (Qi et al., 4 Feb 2026).

The paper identifies three additional sources of instability. First, an unconstrained linear value head can produce predictions outside the known reward interval, even though an expected bounded return must lie within that interval. Second, bootstrapped value targets can partially reproduce the old critic rather than accurately fit observed outcomes. Third, batch-wise advantage normalization can amplify residual estimation noise as the policy approaches an optimum, preventing the policy update from naturally vanishing. Finally, a fixed GAE parameter gives terminal rewards exponentially decreasing influence on early tokens as response length increases.

BPCO is therefore framed as an alignment of four objects: the critic’s output range, the critic’s training target, the critic’s available inputs, and the advantage signal induced for the policy. This framing is more specific than simply advocating “better critics”; it attributes instability to identifiable interactions between estimator design and LLM-specific sequence lengths.

Components of BPCO

DPPO and probability-space clipping

BPCO uses DPPO rather than standard PPO. For a sampled token with behavior-policy probability $\mu(y_t \mid s_t)$, the clipping boundary is scaled inversely with that probability. Equivalently, the update constrains the sampled token’s absolute probability change to a common threshold. This distinction matters because ratio clipping does not impose a uniform constraint in probability space.

The controlled sanity test provides a direct diagnostic. On 1,460 mathematical problems that the initial 1.5B-parameter model could already solve, standard PPO with $\lambda = 1$ experienced a collapse in training reward after an initial increase. Replacing PPO with DPPO stabilized optimization and allowed the model to fit the deliberately solvable dataset. However, setting $\lambda = 0.99$ made DPPO unstable again. This result isolates policy-objective clipping as necessary but insufficient: DPPO prevents one instability, while critic bootstrapping introduces another.

Bounded value predictions

BPCO maps the critic output into the known reward range using a scaled arctangent parameterization. For binary rewards, every prediction lies in the open interval $(0,1)$ rather than extending arbitrarily beyond $[0,1]$. This parameterization imposes a structural constraint consistent with the semantics of the value function.

In the sanity test, the unbounded linear head generated values outside the binary reward range and was associated with unstable training reward and deteriorating AIME 2025 average accuracy over 32 samples. Bounding the value prediction removed these extreme outputs and allowed training reward to approach one. The implication is not merely numerical regularization: when the critic is used to construct advantages, out-of-range values can generate implausible residuals and thereby distort the policy gradient. Enforcing the reward range constrains the critic-induced policy signal at its source.

The larger-scale ablation confirms that this effect persists beyond the small solvable dataset. On approximately 40.3K DeepScaleR mathematical problems, removing value bounding slowed training-reward improvement and reduced AIME 2025 average accuracy. Thus, value-range consistency remains beneficial even when the dataset is sufficiently large to reduce some small-data pathologies.

Monte Carlo targets for critic training

The paper separates the GAE parameter used for the policy advantage from the parameter used to train the critic. It retains $\lambda_\pi = 0.99$ for the policy but sets $\lambda_V = 1$ for critic training. With outcome-only rewards and $\gamma = 1$, the critic target then telescopes to the observed terminal reward. Each response outcome is consequently an unbiased Monte Carlo sample of the expected return under the rollout policy.

This distinction exposes a limitation of explained variance as commonly reported. When the critic is trained against a bootstrapped target containing the old critic’s predictions, it can achieve explained variance close to one while remaining inaccurate relative to the actual observed reward. The sanity experiment exhibits precisely this behavior: explained variance against the bootstrapped target rapidly approaches one even while policy training remains unstable. After replacing the target with the observed outcome, explained variance becomes a more meaningful diagnostic and training becomes more stable and faster.

The result supports a specific methodological conclusion: critic fit should be evaluated against an external outcome target, not only against a self-referential bootstrapped target. The use of Monte Carlo targets increases target variance, but in the reported setting it removes systematic critic bias that was more damaging to policy optimization than the additional variance.

Unnormalized advantages

BPCO does not normalize advantages across each batch. The paper argues that normalization is especially inappropriate near policy convergence. If the true advantages and their variance both shrink, dividing by the batch standard deviation maintains an artificially large update magnitude. Subtracting the batch mean can also reverse the sign of examples whose positive advantage is smaller than the batch average.

The sanity experiment shows that removing normalization keeps the advantage range small and stable while achieving comparable training reward. It also mitigates overfitting on AIME 2025 average accuracy. With normalization, the magnitude of normalized advantages grows as training progresses, consistent with the claim that normalization prevents the policy update from diminishing naturally. The practical implication is that raw advantages preserve an implicit trust in the absolute scale of the residual policy signal, rather than forcing every batch to produce an update of comparable magnitude.

On DeepScaleR, reintroducing normalization produces growing advantage magnitudes, although the performance difference is modest because training had not fully converged. The authors therefore distinguish between the immediate performance effect and the proposed general-purpose default: the latter is justified primarily by convergence behavior and stability, not by a uniformly large early-training gain.

Length-adaptive GAE

For a response of length $L$, BPCO sets the policy GAE parameter to

$\lambda_\pi(L) = 1 - \frac{1}{\alpha L}.$

This makes the cumulative weight assigned to the terminal residual approximately invariant to response length. With a fixed $\lambda = 1$0, the terminal reward receives a factor proportional to $\lambda = 1$1, so early-token advantages in long responses depend predominantly on bootstrapped critic residuals. Length-adaptive GAE reduces this disparity while retaining some variance reduction relative to $\lambda = 1$2.

In the sanity test, fixed $\lambda = 1$3 produced rapid training-reward improvement but a pronounced decline in held-out AIME 2025 accuracy. Setting $\lambda = 1$4 avoided this decline but slowed optimization. Length-adaptive GAE with $\lambda = 1$5 provided the reported compromise: better training efficiency than full Monte Carlo policy advantages while mitigating the validation degradation associated with fixed low-$\lambda = 1$6 GAE.

This result is particularly relevant for long-chain-of-thought training. It indicates that a single global GAE parameter is poorly matched to variable-length responses, and that the bias–variance trade-off should be calibrated to the number of tokens over which the terminal reward must propagate.

Privileged information for the critic

Because the critic is discarded after training, BPCO allows it to receive information that the deployment policy cannot access. In mathematical reasoning, this information may be a reference answer or official solution; in rubric-based evaluation, it may be the prompt-specific rubric. The policy continues to condition only on the original prompt and its generated prefix.

The construction is formally justified by treating the privileged information as fixed by the prompt. It does not alter the ideal value associated with the prompt, but it can reduce the approximation burden on a finite critic. In the small-data sanity test, supplying the reference answer accelerates training-reward improvement and increases explained variance. However, validation accuracy peaks earlier and then declines, indicating that the more informative critic can overfit the limited training distribution. The paper therefore makes a deliberately conditional claim: privileged information improves critic learning, but improved critic fit does not guarantee improved policy generalization.

On DeepScaleR, where reference answers are available for the full dataset and official solutions for approximately 7.3K of the 40.3K problems, reference-answer conditioning yields faster training, higher explained variance, and better AIME 2025 performance. Official-solution conditioning provides a more modest improvement despite incomplete coverage. These results suggest that privileged inputs are most useful when they provide consistent reward-relevant information and when the training regime has not yet entered an overfitting-dominated phase.

The rubric experiment further qualifies the result. Supplying the rubric to the critic increases explained variance but does not improve final performance relative to BPCO without privileged information, plausibly because the task is comparatively simple. Thus, critic predictability and policy benefit are empirically separable quantities.

Empirical evaluation

The evaluation spans a 1.5B-parameter dense model, two Qwen3-30B-A3B mixture-of-experts models, mathematical verification rewards, and rubric-based judge rewards. The baselines include a critic recipe using decoupled GAE, unbiased critic targets, and length-adaptive GAE, as well as a Dr. GRPO group baseline using 16 responses per prompt. To equalize total trajectories per iteration, the group baseline uses fewer distinct prompts. Critic-based methods use one response per prompt.

On the approximately 40.3K-problem DeepScaleR dataset, BPCO variants consistently outperform both the group-based and critic-based baselines in training and AIME 2025 validation performance. BPCO also maintains higher explained variance against the Monte Carlo target, supporting the claim that its critic is more accurate under the selected diagnostic. The strongest results are obtained with privileged reference-answer or solution inputs where applicable, although the paper does not provide a consolidated numerical score table in the supplied text; the evidence is presented primarily through learning curves.

The 30B-A3B experiments establish that the recipe is not confined to the 1.5B setting. On DAPO-Math-17K, BPCO improves over the critic baseline for both Qwen3-30B-A3B-Base and Qwen3-30B-A3B. For the non-base Qwen3-30B-A3B model, the critic baseline fails to improve AIME 2025 accuracy beyond approximately the first 100 training steps, whereas BPCO continues to improve. BPCO outperforms the group baseline on Qwen3-30B-A3B and performs comparably on Qwen3-30B-A3B-Base. These comparisons support the paper’s strongest efficiency claim: a carefully designed critic can replace 16-way group sampling without sacrificing performance, while requiring only one rollout per prompt.

The rubric-based experiment uses Qwen3-4B-Base as both policy and critic initialization, OpenRubrics prompts, and a frozen Qwen3-4B-Instruct-2507 judge. Both BPCO variants learn faster than the group and critic baselines. The group method eventually reaches comparable performance, while the critic baseline finishes somewhat lower. BPCO without privileged rubric input nevertheless performs strongly, showing that bounded values and unnormalized advantages retain utility even when reward supervision is generated by a rubric-based evaluator rather than exact mathematical verification.

Limitations and open questions

The empirical evidence is limited to mathematical reasoning and rubric-based rewards. It does not establish whether the same stability mechanisms transfer to preference optimization, tool-use trajectories, agentic environments, dense rewards, or highly multimodal tasks. BPCO also assumes that the reward range is known, an assumption that may be difficult to satisfy for learned or dynamically calibrated evaluators.

Privileged critic inputs require access to evaluator information such as reference answers, official solutions, or rubrics. Although these inputs do not enter the deployed policy, they may be unavailable, costly to construct, or distributionally mismatched with deployment data. The small-data experiments show a clear overfitting risk: privileged information can improve critic fit and accelerate reward gains while causing earlier validation decline. The appropriate regularization, early stopping, or data-scaling conditions for avoiding this trade-off remain unresolved.

The trajectory-matched comparisons also do not capture the full computational cost of critic-based training. Critics introduce additional parameters, forward passes, optimizer state, activation memory, and value-target computation. Consequently, sampling one response per prompt does not imply lower total wall-clock or memory cost than group-based methods. The paper establishes a reduction in rollout multiplicity, not a complete systems-level efficiency advantage.

Finally, BPCO combines several interventions, and the controlled study isolates their effects sequentially rather than exhaustively evaluating all interactions across model scales and reward types. The results establish strong empirical coherence, but they leave open which components are indispensable in particular regimes and whether bounded value prediction or raw-advantage updates remain beneficial under nonstationary, unbounded, or heavily noisy rewards.

Conclusion

“Best Practice Critic Optimization” argues that critic-based LLM RL becomes reliable when the critic and policy update are designed coherently. DPPO stabilizes the policy trust region, bounded value heads enforce return-consistent outputs, Monte Carlo targets prevent self-referential critic fitting, raw advantages preserve the natural decay of the policy signal, and length-adaptive GAE limits response-length-dependent bias. Privileged critic inputs can further improve value estimation when reward-defining information is available, although they introduce overfitting risks and do not guarantee policy gains. Across 1.5B and 30B-A3B models, mathematical datasets, and rubric rewards, BPCO improves critic baselines and matches or exceeds group-based methods with one response per prompt. The paper’s principal contribution is therefore an empirically validated training recipe in which critic-based single-rollout optimization is presented as a practical alternative to group-relative advantage estimation (2608.23566).