Best Practice Critic Optimization (BPCO)

• Best Practice Critic Optimization (BPCO) is a paradigm that optimizes critics based on downstream decisions rather than isolated prediction loss, improving actor-critic methods in LLM training.
• BPCO considers target alignment, output constraints, gradient usefulness, distributional reliability, and complexity control to ensure critic outputs align with policy-improvement requirements.
• Applications range from LLM post-training to continuous-control RL, multi-agent systems, and offline optimization, with challenges including reward concentration, model learning costs, and integrating various critic signals.

Best Practice Critic Optimization (BPCO) is a critic-centered optimization paradigm in which a critic is designed, trained, selected, and evaluated according to the downstream decisions it enables rather than according to an isolated prediction loss. The term is formally introduced for large-language-model reinforcement learning in “Best Practice Critic Optimization” (Qi et al., 24 Aug 2026), where it denotes a recipe combining DPPO, reward-range-bounded value predictions, Monte Carlo critic targets, unnormalized policy advantages, and length-adaptive GAE. More broadly, BPCO encompasses methods that align critic outputs with policy-improvement requirements, control critic reliability and complexity, balance heterogeneous critic signals, regulate extrapolation, and allocate expensive computation according to critic-informed evidence.

1. Conceptual foundations and scope

BPCO addresses a recurring mismatch between conventional critic objectives and the information actually consumed by an optimizer. A critic may be trained to minimize value error, TD error, or regression loss, while the actor uses derivatives, token-level advantages, task-specific value components, uncertainty estimates, or critic-mediated search constraints. In such cases, low prediction loss is an incomplete criterion for critic quality.

In deterministic continuous-control actor–critic methods, for example, the actor uses the action derivative

$\nabla_a Q(s,a)\big|_{a=\pi_\theta(s)},$

rather than only the scalar value $Q(s,\pi_\theta(s))$. “How to Learn a Useful Critic? Model-based Action-Gradient-Estimator Policy Optimization” shows that value accuracy does not directly control action-gradient accuracy and proposes MAGE, which explicitly trains a critic to provide policy-improvement-relevant action gradients (D'Oro et al., 2020). MAGE’s central distinction is between value accuracy and policy-improvement usefulness: a critic can predict sampled values plausibly while producing incorrect, flat, noisy, or excessively steep action derivatives.

BPCO therefore treats critic optimization as a multi-dimensional design problem involving:

• Target alignment: whether critic targets correspond to the quantity required by policy improvement.
• Output constraints: whether critic predictions respect known value or reward ranges.
• Gradient usefulness: whether critic derivatives provide reliable update directions.
• Distributional reliability: whether optimization remains within regions where learned models are supported.
• Task and objective balance: whether heterogeneous tasks or reward channels receive appropriate critic influence.
• Temporal and policy nonstationarity: whether critic training remains stable as the evaluated policy changes.
• Complexity control: whether the critic uses an appropriate amount of representational capacity.
• Experimental validity: whether critic comparisons use fair, reproducible, and diagnostic benchmarking.

The term does not denote a single universal architecture. Its formal meaning is supplied by the LLM post-training method in (Qi et al., 24 Aug 2026), while the broader methodological interpretation is supported by research on action-gradient critics, multi-critic policy optimization, offline surrogate regularization, functional critics, trajectory-optimization critics, analog-design search critics, adaptive critic utilization, critic balancing, and critic-complexity control.

2. The formal BPCO recipe for LLM reinforcement learning

The formal BPCO method is designed for outcome-reward LLM reinforcement learning, particularly mathematical reasoning and rubric-based rewards. For a prompt $x$, the generated response is

$y=(y_1,\ldots,y_T),$

with token state

$s_t=(x,y_{<t}).$

The response receives an outcome reward $R(x,y)$, while intermediate rewards are zero. Group-based methods such as GRPO estimate advantages by sampling multiple responses for the same prompt. With group rewards $R_i$, GRPO uses

$$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$

and Dr. GRPO removes standard-deviation normalization:

$\widehat A^{\mathrm{Dr.GRPO}}_{t,i}=R_i-\mu_R.$

BPCO instead uses one response per prompt and trains a critic to estimate token-level values and advantages.

The recipe contains five principal components:

1. DPPO: probability-change-based clipping.
2. Reward-range-bounded value predictions.
3. Monte Carlo value targets.
4. Unnormalized policy advantages.
5. Length-adaptive GAE.

The method may additionally provide the critic with reward-defining information hidden from the policy, such as a reference answer, official solution, or grading rubric.

DPPO

For behavior policy $\mu$ and current policy $Q(s,\pi_\theta(s))$0, the sampled-token probability ratio is

$Q(s,\pi_\theta(s))$1

Standard PPO clips this ratio to a fixed interval. DPPO instead uses a token-dependent interval:

$Q(s,\pi_\theta(s))$2

This approximately constrains the absolute probability change:

$Q(s,\pi_\theta(s))$3

The motivation is that a fixed multiplicative ratio interval produces different absolute probability changes for tokens with different behavior-policy probabilities. In the reported sanity experiment, replacing PPO with DPPO stabilized training when $Q(s,\pi_\theta(s))$4, although DPPO alone became unstable again at $Q(s,\pi_\theta(s))$5.

Bounded value predictions

If all possible returns lie in $Q(s,\pi_\theta(s))$6, then the conditional value must lie in the same interval. BPCO enforces this using

$Q(s,\pi_\theta(s))$7

where $Q(s,\pi_\theta(s))$8 is an unconstrained neural output. The resulting value lies in $Q(s,\pi_\theta(s))$9.

For binary rewards,

$x$0

This prevents impossible predictions such as values outside the reward range. In the reported sanity experiment, unbounded value heads produced extreme predictions associated with unstable reward and validation behavior, whereas bounded predictions stabilized training.

Monte Carlo critic targets

For outcome-only rewards,

$x$1

and the terminal value is zero. BPCO separates the policy and critic GAE parameters:

• $x$2 is used for policy advantages.
• $x$3 is used for critic targets.

With $x$4 and outcome-only rewards, the critic target telescopes to the observed outcome:

$x$5

The critic is therefore trained directly against the Monte Carlo outcome rather than against a target containing old critic predictions. This avoids self-referential bootstrapping error, although the target has higher variance.

BPCO evaluates explained variance against the Monte Carlo target:

$x$6

This is distinguished from explained variance against a bootstrapped target, which can be high even when the critic predicts observed outcomes poorly.

Unnormalized advantages

BPCO uses raw GAE values directly and does not apply batch-wise advantage normalization. The method therefore preserves the natural scale of the policy signal: large residual advantages produce stronger updates, while advantages near zero produce naturally small updates.

Batch normalization can amplify noise when advantage variance becomes small near convergence. It can also reverse the signs of genuinely positive examples through subtraction of the batch mean. In the reported experiments, removing normalization improved validation behavior in the small sanity test and was retained as the general default.

Length-adaptive GAE

For fixed $x$7, the contribution of a terminal reward to an early token decreases as

$x$8

Long responses therefore receive greater bootstrapping attenuation. BPCO uses

$x$9

where $y=(y_1,\ldots,y_T),$0 is response length and $y=(y_1,\ldots,y_T),$1 controls the bias–variance tradeoff. The resulting early-token terminal coefficient is approximately

$y=(y_1,\ldots,y_T),$2

which is approximately independent of response length.

The reported best setting in the sanity study was $y=(y_1,\ldots,y_T),$3. This produced a compromise between the speed of $y=(y_1,\ldots,y_T),$4 and the stability of $y=(y_1,\ldots,y_T),$5.

3. Critic usefulness beyond value fitting

BPCO’s central principle is that the correct critic objective depends on the downstream use of the critic.

Action-gradient usefulness

MAGE formulates critic training around the action gradient used by deterministic policy gradients. The actor update is

$y=(y_1,\ldots,y_T),$6

The relevant critic error is therefore derivative error rather than value error alone. MAGE differentiates a model-based TD target through a learned differentiable dynamics model and trains the critic using an action-gradient term together with an ordinary TD-error penalty. Its loss is

$y=(y_1,\ldots,y_T),$7

The TD penalty is necessary because action-gradient-only optimization can produce a degenerate flat critic with

$y=(y_1,\ldots,y_T),$8

MAGE consequently exemplifies BPCO as policy-improvement-aware critic optimization, but its objective remains a proxy for action-gradient consistency rather than a direct optimization of true-return improvement.

Token-level credit assignment

BPCO for LLMs uses the critic to estimate values for individual prefixes, allowing policy updates to assign different signals to planning tokens, formatting tokens, and trailing tokens. Group-based methods assign the same response-level advantage to every token in a sampled response. The critic-based alternative can therefore reduce rollout multiplicity while increasing temporal resolution of credit assignment.

The reported comparisons used 16 responses per prompt for the group baseline and one response per prompt for critic-based methods. BPCO matched or exceeded the group baseline on the reported mathematical and rubric-based tasks while using one response per prompt.

Critic-guided search and design decisions

In analog design, the Actor–Critic Optimization Framework (ACOF) uses an actor to propose a design subregion and a critic to audit, repair, or redirect it before Bayesian optimization and Ngspice simulation. ACOF is not a standard RL actor–critic method: it does not use TD updates, policy gradients, replay buffers, or an actor–critic loss. Its relevance to BPCO lies in the separation between proposal and judgment.

The ACOF cycle is

$y=(y_1,\ldots,y_T),$9

The critic enforces PDK legality, detects malformed or physically implausible regions, identifies unproductive search directions, and incorporates evidence from prior rounds. This establishes a broader BPCO pattern in which the critic evaluates search actions rather than merely estimating a scalar state or action value.

Reliability under surrogate exploitation

In offline model-based optimization, GABO with adaptive Source Critic Regularization (aSCR) uses a Wasserstein-style source critic to distinguish offline reference data from generated candidates. The source critic does not estimate calibrated posterior uncertainty or pointwise prediction error. Instead, it estimates distributional discrepancy.

The constrained objective is

$s_t=(x,y_{<t}).$0

subject to

$s_t=(x,y_{<t}).$1

The corresponding Lagrangian is

$s_t=(x,y_{<t}).$2

The adaptive mechanism selects a regularization strength through a dual search over $s_t=(x,y_{<t}).$3, with

$s_t=(x,y_{<t}).$4

This prevents the optimizer from maximizing a learned surrogate in regions where the source critic indicates strong distributional departure. The method supports a BPCO interpretation in which the critic functions as a reliability or conservatism constraint. It does not establish that distributional similarity guarantees low surrogate error.

4. Multi-critic, functional, and uncertainty-aware optimization

BPCO can use multiple critics when a single scalar estimate obscures heterogeneous objectives, policies, or sources of uncertainty.

Reward-specific critics

Multi-Critic Policy Optimization (MCPO), also called Polymorph Policy Gradient Optimization, uses one shared stochastic actor and multiple value estimators. For $s_t=(x,y_{<t}).$5 UAV agents, the global state is concatenated and the actor produces a joint action:

$s_t=(x,y_{<t}).$6

$s_t=(x,y_{<t}).$7

Each critic estimates a separate reward component:

$s_t=(x,y_{<t}).$8

The critic-specific advantage is

$s_t=(x,y_{<t}).$9

MCPO retains a vector of advantages rather than immediately reducing all reward signals to a scalar. Its motivation is to prevent large or dense rewards from dominating smaller but important coordination and safety signals.

MCPO is related to BPCO through critic plurality, reward-specific feedback, dynamic critic management, and coordinated actor updates. However, it does not define an explicit best-critic selector, learned critic ranking, or formal weighting rule. Its “best” performance refers to the multi-critic architecture as a whole.

Policy-conditioned functional critics

“Functional Critic Modeling for Provably Convergent Off-Policy Actor-Critic” conditions the critic on the evaluated policy:

$R(x,y)$0

The critic is intended to represent values for a family of policies rather than only the current policy. This addresses two problems:

• off-policy critic instability under the deadly triad;
• the moving-target problem caused by continual policy updates.

Differentiating the policy-conditioned critic supplies the exact off-policy objective gradient:

$R(x,y)$1

The second term is obtained by backpropagating through the policy input of the critic, avoiding a separately learned emphasis critic. The theoretical analysis establishes critic tracking and approximate actor stationarity in a linear-functional setting under boundedness, ergodicity, covariance, Lipschitz, and step-size assumptions. The neural experiments on DeepMind Control tasks are preliminary and do not establish neural convergence.

Uncertainty-guided data allocation

CACTO-BIC uses a value critic trained from trajectory-optimization cost-to-go values and gradients, together with a second network called the std-critic. The value critic supports policy improvement through

$R(x,y)$2

The std-critic predicts the standard deviation of the critic’s value estimate. After the first iteration, $R(x,y)$3 candidate initial states are sampled and the $R(x,y)$4 states with largest predicted uncertainty are selected for trajectory optimization.

This implements a critic-aware data-allocation mechanism:

• the value critic identifies what the actor should do;
• the std-critic identifies where expensive trajectory optimization should be performed.

The paper reports comparable performance with approximately $R(x,y)$5 of the trajectory-optimization episodes and GPU speedups under the tested configurations. The method does not establish that std-critic uncertainty is calibrated or that high uncertainty always identifies valuable basin boundaries.

Adaptive critic utilization

EVPO addresses whether a learned critic should be trusted for a particular training batch. It compares a state-dependent critic baseline with a batch-mean baseline using explained variance:

$R(x,y)$6

The gate is

$R(x,y)$7

Positive explained variance indicates that the critic reduces residual variance relative to the return variance. Zero or negative explained variance indicates that the critic is neutral or harmful relative to the batch mean.

EVPO interprets PPO and GRPO as endpoints of a baseline spectrum. Under its variance model,

$R(x,y)$8

so the adaptive estimator has no greater advantage variance than the better of the critic and batch-mean baselines at each step under the population assumptions. The guarantee concerns advantage-estimator variance, not final return or universal optimization stability.

5. Balancing, normalization, and complexity control

BPCO also concerns the internal geometry and complexity of critic representations.

Multi-task critic balancing

“TOPPO: Rethinking PPO for Multi-Task Reinforcement Learning with Critic Balancing” identifies shared-critic gradient ill-conditioning as a primary cause of tail-task failure (Li et al., 12 May 2026). For task-specific critic gradients

$R(x,y)$9

vanilla PPO uses

$R_i$0

The critic-gradient Gram matrix is

$R_i$1

TOPPO separates three problems:

• Scale spread: task gradients have very different magnitudes.
• Co-linear collapse: task gradients concentrate in similar shared-feature directions.
• Unfair aggregation: uniform averaging gives disproportionate influence to dominant tasks.

Its modules are:

• PopArt for value-target scale;
• LN-c for critic representation geometry;
• FairGrad-c for gradient aggregation;
• PCGrad-a for residual actor-gradient conflict.

PopArt applies task-specific affine normalization to value predictions. LN-c applies LayerNorm only to critic hidden layers. FairGrad-c combines task gradients using Gram-aware weights $R_i$2:

$R_i$3

For $R_i$4, the weights satisfy

$R_i$5

which yields the fixed-energy identity

$R_i$6

FairGrad-c is invariant to arbitrary positive per-task gradient rescaling, making it complementary to target normalization. PCGrad-a is applied to actor gradients to remove directional conflict after critic-side scale and geometry problems have been addressed.

The reported MT50 V2 results show that TOPPO improves both mean and worst-10 task success relative to vanilla multi-task PPO. The main improvement is tail recovery rather than mean performance alone. However, critic balancing cannot solve tasks for which useful reward information is never obtained, and it does not replace exploration or reward shaping.

Spectral critic complexity

“Gauging, Measuring, and Controlling Critic Complexity in Actor-Critic Reinforcement Learning” introduces spectral effective-rank entropy as a critic-complexity diagnostic (Garbers, 1 Jul 2026). For weight matrix $R_i$7 with singular values $R_i$8, normalized spectral masses are

$R_i$9

and spectral entropy is

$$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$0

The critic-level complexity is the average entropy across critic layers. Effective rank is related by

$$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$1

A spectral-entropy penalty augments the critic objective:

$$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$2

The paper shows that spectral complexity can be measured throughout training and directly altered by regularization. Its relationship with return, Monte Carlo bias, and volatility is heterogeneous across algorithms and tasks. Moderate regularization improved TD3 on HalfCheetah-v4, but stronger entropy reduction did not produce the best return. In other tasks, complexity control had little effect or worsened performance.

Consequently, BPCO treats complexity as a diagnostic and controllable variable rather than as a universal objective. Complexity should be evaluated jointly with TD error, value loss, Monte Carlo bias, bias volatility, return, return volatility, and downstream policy improvement.

6. Benchmarking methodology and reproducibility

Because critic quality is downstream-dependent, BPCO requires evaluation protocols that measure more than average objective values or final returns. “Best practices for comparing optimization algorithms” recommends treating benchmarking as a deliberately designed scientific experiment (Beiranvand et al., 2017).

A BPCO evaluation should first specify its purpose. Possible purposes include selecting an algorithm for an application, demonstrating the value of a new critic mechanism, comparing software versions, or studying algorithmic options and parameter settings. The purpose determines the test problems, metrics, initialization, budgets, and statistical analysis.

A reliable benchmark should include:

• at least approximately 20 problems, preferably more;
• easy and hard problem groups;
• standard library, application-specific, and generated instances where appropriate;
• known optimal values or minimizers whenever possible;
• identical stored starting points and documented random seeds;
• repeated runs for stochastic methods;
• fixed-target and fixed-cost experiments;
• explicit treatment of infeasibility, failures, timeouts, missing values, and crashes.

The study should collect at least one metric from each of three categories:

• Efficiency: objective, gradient, Hessian, constraint, or critic evaluations; CPU time; wall-clock time; memory; and other method-specific costs.
• Reliability: success rate, feasibility, constraint violation, robustness across starts, and stochastic variability.
• Solution quality: fixed-target cost, fixed-cost accuracy, objective error, distance error, or application-specific quality.

For BPCO, critic-specific costs must be counted explicitly. If a critic uses additional model evaluations, second-order derivatives, source-critic updates, std-critic evaluations, or Gram-matrix solves, these should not be hidden in iteration counts.

Performance profiles are suitable for variable-cost convergence experiments. For performance measure $$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$3, the performance ratio compares each solver with the best solver on problem $$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$4, and the profile reports the fraction of problems solved within a factor $$\widehat A^{\mathrm{GRPO}}_{t,i} = \frac{R_i-\mu_R}{\sigma_R},$$5 of the best. Accuracy profiles are more appropriate for fixed-cost comparisons, while data profiles describe the fraction of problems solved within a normalized evaluation budget.

Performance profiles have limitations. They depend strongly on the convergence test, are unsuitable for fixed-cost data, and primarily compare each method with the best method. Switching effects can make the ordering of second- and third-best curves unreliable. Complementary reporting should include complete result tables, box plots, histograms, convergence plots, trajectory plots, runtime-versus-dimension plots, accuracy profiles, performance profiles, and data profiles.

For stochastic BPCO experiments, the supplied benchmarking paper recommends repeated runs, means, and standard deviations, but does not itself provide a complete methodology for confidence intervals, significance tests, effect sizes, or multiplicity corrections. These are methodological extensions. Paired comparisons, confidence intervals, Wilcoxon or permutation tests, Friedman tests, effect sizes, and corrected multiple comparisons are appropriate additions when evaluating stochastic critics and baselines.

A reproducible report should include:

• complete algorithm descriptions and pseudocode;
• implementations, versions, repositories, commits, and dependencies;
• hardware and software environments;
• datasets, dimensions, constraints, transformations, and ordering;
• starting points, seeds, random-number generators, and repetition counts;
• parameters, hyperparameters, tuning procedures, and budgets;
• evaluation counts and runtime measurements;
• stopping, safety, feasibility, and failure-handling rules;
• raw run-level data;
• statistical procedures and confidence levels;
• scripts for tables and figures.

The minimum reproducibility standard is that the authors can rerun the experiments and recreate the reported results. Full bitwise replication may be impossible as hardware and software evolve.

7. Applications, limitations, and open problems

BPCO has been instantiated or conceptually supported across several domains.

In LLM post-training, BPCO provides a critic-based alternative to group-relative estimation. On mathematical reasoning and rubric-based rewards, the reported method improved strong critic-based baselines across models ranging from 1.5B parameters to 30B-A3B mixtures of experts. It matched or exceeded the group-based baseline while using one response per prompt. Its principal limitations are outcome-reward concentration, the requirement for a known reward range, potential overfitting from privileged critic inputs, additional critic computation, and limited evidence outside mathematical and rubric-based tasks.

In continuous-control reinforcement learning, MAGE demonstrates action-gradient-aware critic optimization, while functional critic modeling addresses policy-conditioned off-policy evaluation. Their theoretical claims rely on assumptions that are stronger than typical neural deep-RL settings. MAGE incurs additional model-learning and second-order derivative costs, while functional critics require policy-space representations that generalize across changing policies.

In multi-agent reinforcement learning, MCPO preserves reward-specific advantages and improves reported UAV coordination relative to single-critic and hybrid alternatives. Its vector advantage reduction is under-specified, and it does not define an explicit best-critic weighting or selection mechanism. Its centralized global-state actor may scale poorly to decentralized or variable-size deployments.

In offline model-based optimization, GABO/aSCR shows that a source critic can constrain surrogate optimization against distributional extrapolation. The method does not provide a formal guarantee that critic scores correspond to oracle error. It may suppress genuinely superior out-of-distribution solutions, inherit bias from the offline dataset, or become overly conservative.

In trajectory optimization, CACTO-BIC uses critic values, value gradients, and uncertainty-guided initial-state selection. Its reported data-efficiency improvements depend on differentiable trajectory-optimization models, useful local solutions, and uncertainty that correlates with valuable exploration. The method does not prove convergence of biased sampling, std-critic calibration, global optimality, or robustness under model mismatch.

In analog design, ACOF separates proposal, critic audit, Bayesian optimization, simulation, and reflection. It provides a region-level search and judgment framework, but its LLM-based critic is not a trained value function. The numerical BO backend, prompts, critic decision accuracy, and generalization across circuits and PDKs remain incompletely specified.

In multi-task PPO, TOPPO demonstrates that target normalization, critic representation conditioning, Gram-aware aggregation, and actor-gradient surgery can improve tail-task performance. Its additional computational complexity includes per-task gradients, Gram-matrix construction, and nonlinear weight solving. The correct intervention depends on whether heterogeneity arises from reward scale, representation geometry, gradient conflict, or insufficient exploration.

Finally, spectral critic complexity provides a measurable but non-universal diagnostic. Lower spectral entropy can coincide with improved stability and return in some settings, but it can also have no effect or degrade performance. Complexity reduction should therefore be controlled jointly with critic accuracy, bias, volatility, and policy outcomes.

Open problems for BPCO include:

• defining critic quality as a multidimensional, downstream-dependent quantity;
• combining value accuracy, gradient accuracy, uncertainty, fairness, safety, and complexity in a single auditable protocol;
• developing principled critic weighting and selection mechanisms;
• establishing finite-sample guarantees for adaptive critic gates;
• calibrating uncertainty and distributional-reliability critics;
• handling dense, noisy, nonstationary, and unbounded rewards;
• controlling policy and critic overfitting under privileged information;
• extending theoretical guarantees from linear functional critics to neural networks;
• measuring critic usefulness under distribution shift and changing replay distributions;
• determining when complexity regularization improves rather than harms policy learning;
• standardizing computational accounting for critic training, model learning, second-order derivatives, and adaptive data selection.

The unifying BPCO doctrine is that critic optimization should be judged by the quality, reliability, fairness, stability, and efficiency of the decisions that the critic enables. A critic is not necessarily best because it minimizes value error, has the lowest spectral complexity, predicts the highest reward, or produces the largest gradient. Its quality is conditional on the optimization context: the actor statistic it supplies, the reward structure it represents, the distribution on which it operates, the constraints it enforces, and the empirical consequences of trusting it.