---
title: On-Policy Optimization in RL
url: https://www.emergentmind.com/topics/on-policy-optimization
type: topic
---

# On-Policy Optimization in RL

On-policy optimization denotes a family of reinforcement learning (RL) algorithms wherein all policy updates are based strictly on data generated by the current policy or a small mixture of recent policies. This paradigm underpins key trust-region and policy-gradient approaches in sequential decision-making, favoring robust monotonic improvement at each step, often at the expense of sample efficiency relative to off-policy methods. On-policy optimization encompasses explicit policy-iteration, policy-gradient, and their modern variants equipped with trust-region and clipping mechanisms. Theoretical guarantees for monotonic improvement, the precise structure of surrogate objectives, trade-offs between step-size and policy divergence, and recent extensions for accelerated or robust learning form the foundations of the field.

## 1. Core Principles and Theoretical Foundations

On-policy optimization is fundamentally derived from policy improvement theory in MDPs, which quantifies the expected return difference between a new candidate policy π and the current policy πₖ via a surrogate lower bound. For discounted problems,

\[
J(\pi) - J(\pi_k) \geq \frac{1}{1-\gamma} \mathbb{E}_{(s,a)\sim d^{\pi_k}}\left[ \frac{\pi(a|s)}{\pi_k(a|s)} A^{\pi_k}(s,a) \right] - \frac{2\gamma C^{\pi,\pi_k}}{(1-\gamma)^2} \mathbb{E}_{s\sim d^{\pi_k}}[\mathrm{TV}(\pi,\pi_k)(s)]
\]

where $A^{\pi_k}(s,a)$ is the advantage, $C^{\pi,\pi_k}$ is a bound on advantage variation, and $\mathrm{TV}$ is the total-variation distance over actions. This guarantee ensures that, provided the policy shift is constrained (i.e., TV or KL divergence is small), every update results in non-negative improvement [2111.00072].

To maintain monotonic improvement, modern on-policy algorithms constrain policy updates via explicit trust-region penalties or surrogate clipping:

- **Surrogate Optimization**: Replace $J(\pi)$ with a sample-based lower bound or a local linear/quadratic surrogate, maximizing tractable objectives that match gradients at the current policy.
- **Trust-Region Constraint**: Enforce a KL-divergence or total-variation constraint on $\pi(\cdot|s)$ to limit distributional shift per update [1502.05477].
- **Clipping**: Instead of explicit constraints, clip the likelihood ratio between the candidate and behavior policy within a set range, thereby approximating a trust region [1707.06347].

Extensions to mixtures of recent policies or off-policy data require additional penalties for the increased statistical divergence, but the same lower-bounding and monotonicity logic applies [2111.00072, 1710.03442].

## 2. Algorithmic Structures: Surrogate Objectives and Update Rules

The operational workflow of on-policy optimization typically involves the following phases:

1. **On-Policy Data Collection**: Interact with the environment under policy πₖ for $n$ steps, generating a batch for update.
2. **Advantage Estimation**: Compute advantage estimates, e.g., via Generalized Advantage Estimation (GAE) or group-relative/empirical means [1707.06347, 2505.22257].
3. **Surrogate Objective Construction**: Formulate the core optimization objective, such as:
   \[
   L^{\text{PPO}}_k(\pi) = \mathbb{E}_{(s,a)\sim d^{\pi_k}} \left[ \min \left( r_k(s,a)A^{\pi_k}(s,a), \text{clip}(r_k(s,a), 1-\epsilon, 1+\epsilon)A^{\pi_k}(s,a) \right) \right]
   \]
   where $r_k(s,a)=\pi(a|s)/\pi_k(a|s)$ [1707.06347, 2111.00072].
4. **Policy Update with Regularization**: Optimize the surrogate via stochastic gradient methods, subject to TV/KL or clip-based trust region enforcement.

Proximal algorithms (TRPO, PPO, GePPO) and their derivatives perform multiple epochs of minibatch updates per data batch, computing gradients with respect to the surrogate, and regulating the policy shift via hyperparameters such as KL bound $\delta$ or clip range $\epsilon$ [1502.05477, 2111.00072].

Recent variants employ additional mechanisms:
- **Reflective Policy Optimization (RPO)** introduces future-trajectory introspection via additional surrogate terms to contract the feasible policy update set, thus accelerating convergence while maintaining monotonicity [2406.03678].
- **Generalized PPO (GePPO)** interpolates between pure on-policy and partial off-policy update regimes by incorporating mixture batches of previous policies, with adaptive learning rates to control cumulative policy divergence [2111.00072].
- **V-MPO** and related EM-style algorithms alternate between building a nonparametric “target” policy by reweighting high-advantage actions, and fitting the parametric policy via constrained maximum likelihood under a KL bound [1909.12238].

## 3. Stability–Efficiency Trade-offs and Sample Reuse

Balancing stability (monotonic improvement, low-variance updates) and sample efficiency is central in on-policy optimization:

- **Stability**: Strict trust-region enforcement or conservative clipping ensures every update is safe, even for high-dimensional, nonconvex policies [1502.05477, 1707.06347].
- **Sample Efficiency**: Traditional on-policy methods discard previous data per update; however, empirical and theoretical advances such as GePPO show that principled reuse of up to $M$ recent policy batches (with appropriate mixture weights) can boost effective batch size or update frequency without violating stability [2111.00072].
- **Adaptive Schedules**: Methods like P3O adaptively toggle between on-policy and off-policy gradient steps based on effective sample size (ESS), automatically regularizing the update as overlap with old policies diminishes [1905.01756].

The table below illustrates three key approaches:

| Algorithm | Policy Shift Constraint         | Sample Reuse                |
|-----------|-------------------------------|-----------------------------|
| TRPO      | KL-divergence hard bound      | No (pure on-policy)         |
| PPO       | Clipped likelihood ratio      | Bounded epoch reuse         |
| GePPO     | TV/clip + mixture penalties   | Recent batch mixture (M)    |

By varying the depth of sample reuse and the strength of trust-region penalties, modern approaches can traverse the stability–efficiency Pareto frontier [2111.00072, 1905.01756].

## 4. Unified Perspectives: Gradient Forms and Surrogate Parameterizations

A unified analytical perspective classifies on-policy algorithms by their “form” (direction of the ascent, e.g., vanilla PG vs. advantage-centered) and “scale” (how the advantage/estimator modulates the update, including reweighting and clipping):

- **Form**: PG (likelihood ratio gradient), PGPB (advantage-centered, control-variated), or entropy-augmented variants.
- **Scale**: Choices range from identity (REINFORCE), first-order corrections (TRPO), maximum-likelihood-inspired scaling, to clipped forms (PPO) [2206.08499].

All classical and recent on-policy methods (REINFORCE, TRPO, PPO, V-MPO, SIL, GePPO) fit within this two-dimensional space, with semantics for bias–variance trade-off, step-size selection, and trust-region management [2206.08499, 1707.06347, 1909.12238].

## 5. Structural Variants and Extensions

On-policy optimization exhibits wide methodological diversity:

- **Discrete Action Parameterization**: Discretizing continuous action spaces into factorized categorical distributions, optionally with ordinal stick-breaking, results in richer and more expressive policies that outperform Gaussians on high-dimensional control tasks and remain compatible with on-policy objectives [1901.10500].
- **Wasserstein Trust Regions**: Viewing policy updates as Wasserstein gradient flows over probability measures yields a geometric justification for step-size and trust-region choices, generalizing natural gradients and suggesting new proximal surrogates [1808.03030].
- **Entropy Regularization**: Adding entropy bonuses smooths the policy landscape, aiding optimization by connecting local minima, expanding the range of stable learning rates, and improving exploration [1811.11214].
- **Robust MDPs and Optimistic Updates**: On-policy approaches incorporating robustness to MDP kernel uncertainty make use of Fenchel-conjugate duality, maintaining monotonic regret under adversarial transition dynamics [2209.13841].

Recent research geometrizes policy learning (Wasserstein flows), systematizes parameter update schedule adaptation (meta-gradient optimism), and blends on- and off-policy regimes to push the sample-efficiency envelope while maintaining performance guarantees [2306.10587, 2111.00072, 1905.01756].

## 6. Empirical Outcomes and Practice

On-policy optimization algorithms, when equipped with advanced surrogates and carefully tuned trust-region or clipping parameters, consistently deliver strong performance across high-dimensional simulated benchmarks:

- **PPO and TRPO** attain robust, monotonic improvements with minimal hyperparameter tuning, as shown in simulated locomotion (MuJoCo), complex games (Atari-2600), and real-world-inspired control tasks [1707.06347, 1502.05477].
- **GePPO** achieves 8–65% higher average returns than PPO with 15–77% fewer samples on diverse MuJoCo tasks, through safe sample reuse and adaptive learning-rate control [2111.00072].
- **Reflective Policy Optimization** accelerates convergence and improves sample efficiency by contracting the solution space, outperforming prior on-policy baselines across a wide suite of continuous and discrete environments [2406.03678].
- **Group Relative Policy Optimization** avoids reliance on value critics by using on-policy group-normalized reward advantages, matching or exceeding off-policy alternatives in sequence modeling tasks [2505.22257].
- **Architectural integration** with interpretable models (e.g., ANFIS) is feasible; PPO reliably trains neuro-fuzzy controllers with faster convergence and reduced variance versus off-policy (DQN) frameworks [2507.01039].

Empirical ablations consistently support the view that tight trust-region enforcement, judicious surrogate construction, and, when possible, controlled sample reuse and regularization are critical to stability, efficiency, and ultimate performance.

## 7. Limitations, Open Problems, and Future Directions

Despite strong theoretical and empirical grounding, on-policy optimization methods face several well-established limitations:

- **Sample Efficiency**: Pure on-policy algorithms remain substantially less sample efficient than off-policy variants unless augmented with safe sample reuse (e.g., GePPO, P3O).
- **Hyperparameter Tuning**: Performance can be sensitive to trust-region or clipping thresholds, learning rates, and advantage estimation parameters; adaptive schedules remain an important area of research [2111.00072, 1707.06347].
- **Variance/Bias Trade-off**: The choice of surrogate, advantage estimator, and scaling affects the balance between variance reduction and estimator bias; principled meta-learning of update rules is an active area [2306.10587].
- **Architectural Restrictions**: Some EM-based and surrogate-tightening methods presuppose log-concave policies or may struggle with arbitrary deep-net architectures [1612.08967].
- **Robustness**: Standard on-policy methods are not robust to model misspecification, prompting extensions for adversarial MDP uncertainty [2209.13841].
- **Generalization to Off-Policy Regimes**: While analytical update rules now permit direct application of on-policy surrogates in off-policy data settings, controlling divergence penalties and estimation error remains nontrivial [2112.02045].

Future research targets include deeper meta-adaptation of optimization schedules (optimism/adaptivity), robustified policy surrogates, principled blending of on- and off-policy data, interpretability, and unifying geometric perspectives on policy space optimization. The interplay between surrogate construction, trust-region management, and algorithmic acceleration continues to drive innovation in on-policy RL.

---

**References**:  
[1502.05477], [1707.06347], [2111.00072], [2406.03678], [2206.08499], [1901.10500], [1808.03030], [2505.22257], [1909.12238], [2507.01039], [1905.01756], [2209.13841], [1811.11214], [2306.10587], [1612.08967], [2112.02045], [2302.09408], [1710.03442]

Source: https://www.emergentmind.com/topics/on-policy-optimization