Critic-Free Reinforcement Learning

• Critic-free reinforcement learning is a method that omits explicit value-function estimation, using Monte Carlo trajectory returns and groupwise normalization instead.
• It employs group-relative baselines like GRPO to generate centered or standardized advantage estimates, reducing bias and simplifying policy gradient updates.
• Empirical studies show this approach performs well in short-horizon and combinatorial tasks by mitigating instability and variance common to traditional critics.

Critic-free reinforcement learning (RL) encompasses policy-gradient algorithms that forgo explicit value-function (critic) estimation during training, instead constructing all policy updates and advantage estimates directly from Monte Carlo trajectory returns and groupwise normalization. Recent advances have established Group Relative Policy Optimization (GRPO) as a canonical critic-free method, eliminating the value network and its associated sources of bias, hyperparameter sensitivity, and computational overhead. These approaches have engendered new theoretical insights and design choices in policy optimization, particularly in domains where value-function critics are unstable, biased, or difficult to train.

1. Fundamentals of Critic-Free RL and Group-Relative Advantage

In standard actor–critic algorithms (e.g., PPO), the critic $V_\phi(s)$ serves dual variance-reduction and bias–variance tradeoff roles: it provides a time-step baseline, $b(s_t) = V_\phi(s_t)$, and enables bootstrapping via temporal-difference estimators such as GAE. Critic-free policy-gradient methods remove this component, relying strictly on returns and batch-level normalization.

GRPO is a principal instantiation of this paradigm. For a group $G = \{\tau_1, \dots, \tau_{|G|}\}$ of complete trajectories, the group-relative baseline is defined as

$$\mu_G = \frac{1}{|G|}\sum_{j=1}^{|G|} R(\tau_j), \qquad \sigma_G^2 = \frac{1}{|G|} \sum_{j=1}^{|G|} (R(\tau_j) - \mu_G)^2,$$

yielding two common advantage estimators:

• Centered: $\hat{A}_t = R(\tau_i) - \mu_G$,
• Centered-and-scaled: $$\hat{A}_t = \frac{R(\tau_i) - \mu_G}{\sigma_G + \epsilon}$$.

This baseline, being purely a function of the group of returns, maintains unbiasedness as a zero-mean control variate but cannot deliver fine-grained state-dependent credit assignment beyond trajectory-level information (Oliveira et al., 5 Nov 2025).

2. Policy Gradient Objectives and Optimization

The core update in GRPO is the clipped surrogate policy gradient objective, structurally similar to PPO but omitting the value loss:

$$L_{\mathrm{clip}}(\theta) = \mathbb{E}\Bigl[\min\left(r_t(\theta)\,\hat{A}_t,\; \mathrm{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\,\hat{A}_t\right)\Bigr]$$

where $$r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_\text{old}}(a_t|s_t)}$$, and $\hat{A}_t$ is the group-relative (standardized or centered) advantage.

Pseudocode implementations maintain standard RL infrastructure, performing policy-only updates via minibatch stochastic gradient descent, and incorporating entropy regularization and temperature scheduling for stabilization (Oliveira et al., 5 Nov 2025, Sepúlveda et al., 30 Mar 2026, Ye et al., 17 Mar 2026).

3. Empirical Properties and Domain-Specific Behavior

Empirical analyses show that critic-free methods, specifically GRPO, display distinctive behavior across domains:

• For short-horizon environments (e.g., CartPole), episodic Monte Carlo returns suffice for effective policy updates, and critic-free methods can rival or surpass critic-based baselines (Oliveira et al., 5 Nov 2025).
• In long-horizon or sparse-reward settings (e.g., HalfCheetah, combinatorial routing, or sequential feature generation), group-relative normalization mitigates instability inherent to value-function approximation and reduces gradient variance in early-stage training (Sepúlveda et al., 30 Mar 2026, Ye et al., 17 Mar 2026).
• The effectiveness of GRPO is sensitive to group size; smaller groups yield better performance, likely due to the preservation of within-group trajectory relevance (Oliveira et al., 5 Nov 2025).
• Discount factors interact nontrivially with group-relative returns: high discount ($\gamma = 0.99$) is generally optimal, but task-specific dynamics may favor lower discount rates in environments lacking early termination (Oliveira et al., 5 Nov 2025).

4. Algorithmic Instantiations and Cross-Domain Applications

Classical Control

In classical RL benchmarks, GRPO operates by sampling small groups of trajectories from parallel environments, computing group-relative returns and updating the policy exclusively via these normalized advantages. While simplifying architecture and removing value-network overhead, the absence of state-dependent baselines limits long-horizon credit assignment unless trajectories are short and returns are well-behaved (Oliveira et al., 5 Nov 2025).

Structured and Combinatorial Decision Problems

In neural combinatorial optimization (e.g., coverage path planning on maritime hexagonal grids), GRPO leverages within-instance normalization to achieve high Hamiltonian path completion rates and solution quality, outperforming heuristic and critic-based approaches in challenging graph topologies. Transformer-based pointer policies parameterize the action distribution; group-relative advantages stabilize optimization across diverse spatial graphs—empirically achieving 99.0% Hamiltonian success, with reduced path length and turn count compared to baseline heuristics (Sepúlveda et al., 30 Mar 2026).

Sequential Feature Generation

For multi-user human activity recognition, GRPO is embedded within autoregressive Transformer-based feature generators. Here, per-batch episodic rewards, comprising class discrimination, user invariance, and temporal fidelity components, replace sparse or delayed external rewards. Groupwise normalization suppresses user and batch-level distribution bias, yielding robust cross-user generalization and accelerated convergence—achieving, for instance, 88.53% accuracy on DSADS and 75.22% on PAMAP2, outperforming standard ERM and critically matching or surpassing PPO-variant counterparts (Ye et al., 17 Mar 2026).

5. Theoretical Rationale and Statistical Properties

The group-relative baseline is a zero-mean, affine-invariant control variate, ensuring unbiasedness in policy gradients under the group-average distribution. Unlike traditional critics, whose value approximation error introduces bias and variance—especially in nonstationary or multimodal reward landscapes—GRPO's batch-level normalization yields advantage estimates invariant to reward scaling and translation. This property underpins stable training in domains with heterogeneous data, long horizons, and structurally biased critics (Ye et al., 17 Mar 2026).

However, a principal limitation is the loss of fine-grained, state-dependent baselining: group normalization cannot replace bootstrapping mechanisms required for effective long-horizon credit assignment in general MDPs (Oliveira et al., 5 Nov 2025).

6. Implementation Considerations and Best Practices

Key hyperparameters in GRPO frameworks include group size $b(s_t) = V_\phi(s_t)$0, discount factor $b(s_t) = V_\phi(s_t)$1, clipping threshold $b(s_t) = V_\phi(s_t)$2, and normalization strategy (centered or standardized). Empirical findings recommend:

• Smaller group sizes (e.g., $b(s_t) = V_\phi(s_t)$3) to maintain trajectory relevance and normalization fidelity (Oliveira et al., 5 Nov 2025, Sepúlveda et al., 30 Mar 2026).
• High discount factors ($b(s_t) = V_\phi(s_t)$4), with adjustments in environments lacking early termination (Oliveira et al., 5 Nov 2025).
• Careful normalization (standardization over group returns) stabilizes advantage scales, especially when per-episode reward statistics are diverse (Sepúlveda et al., 30 Mar 2026, Ye et al., 17 Mar 2026).

The overall architecture is simpler: no value network, no value loss, no associated hyperparameters, and reward computation is batched at the trajectory-level.

7. Limitations and Outlook

Critic-free RL, epitomized by GRPO, is effective where the value-function critic is unreliable or unnecessary: short-horizon environments, combinatorial routing, sequential generative modeling. In canonical control tasks with substantial temporal depth, learned critics remain indispensable for efficient credit assignment and optimization (Oliveira et al., 5 Nov 2025).

In high-dimensional, nonstationary, or batch-heterogeneous settings, critic-free normalization provides robust, drift-free gradient updates. A plausible implication is that hybrid approaches—combining group-relative normalization with partial state-dependent baselining—may overcome the bias–variance tradeoff, but such schemes require further empirical validation. Current evidence circumscribes critic-free methods’ viability to a subclass of RL problems, but demonstrates clear utility in sequence modeling and neural combinatorial optimization (Oliveira et al., 5 Nov 2025, Sepúlveda et al., 30 Mar 2026, Ye et al., 17 Mar 2026).