Gradient-Based Policy Optimization

• Gradient-Based Policy Optimization is a reinforcement learning method that updates policy parameters in the direction of the estimated gradient to maximize expected rewards.
• The framework leverages various gradient estimators, variance reduction techniques, and mirror descent to handle on-policy, off-policy, risk-sensitive, and multi-objective settings.
• Practical implementations integrate trust-region methods, experience replay, and deep function approximation to improve sample efficiency and stability in complex environments.

A gradient-based policy optimization algorithm is a class of methods in reinforcement learning (RL) and stochastic control that iteratively updates the parameters of a parameterized policy in the direction of the (estimated) gradient of a performance objective, typically to maximize expected return or a related criterion. These algorithms leverage the differentiability of the policy parameterization, accessing or estimating the gradient information through on-policy or off-policy sampling, score-function identities, or alternative functional-analytic principles. The framework generalizes across single-objective, multi-objective, risk-aware, and nonconvex RL formulations, and provides a foundation for most modern deep RL methods.

1. Mathematical Structure and Variants

Consider a Markov decision process (MDP) $(\mathcal S, \mathcal A, P, r, \gamma, \rho_0)$ and a stochastic policy $\pi_\theta(a|s)$ parameterized by $\theta \in \mathbb{R}^d$. The canonical objective is the expected discounted return:

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{t = 0}^\infty \gamma^t r(s_t, a_t) \right].$$

Gradient-based policy optimization aims to perform steps of the form:

$$\theta_{k+1} = \theta_k + \eta_k \widehat{\nabla}_\theta J(\theta_k)$$

where $\widehat{\nabla}_\theta J(\theta_k)$ is an unbiased or controlled-bias estimator of the gradient.

Extensions alter the inner objective $J$ to handle multiple objectives $f(J_1(\theta),...,J_M(\theta))$ (Bai et al., 2021), risk criteria (smooth or coherent) (Vijayan et al., 2022, Wang et al., 19 Sep 2025), quantile criteria (Jiang et al., 2022), or entropy-regularized/surrogate losses (Son et al., 2023, Bolland et al., 2023).

Formulations span:

• Vanilla policy gradient: On-policy, REINFORCE-style update (Kämmerer, 2019).
• Actor–critic: Combine Monte Carlo or TD critic with policy gradient update (Kämmerer, 2019).
• Natural gradient/mirror descent: Preconditioned updates using the Fisher (or Bregman) geometry (Huang et al., 2021, Liu et al., 2023).
• Trust-region/proximal: PPO, TRPO use KL-regularization or clipping to enforce smooth policy changes (Son et al., 2023).
• Variance reduction: Momentum, STORM, and experience-replay schemes lower gradient variance (Zheng et al., 2022, Huang et al., 2021).
• Off-policy correction: Incorporate state/action/trajectory importance ratios for data reuse (Zheng et al., 2022, Son et al., 2023).
• Policy-based black-box optimization: Single-step, score-function gradient of expectations, for function optimization and stateless control (Viquerat et al., 2021).

2. Core Algorithmic Principles

Policy Gradient Theorem

For any differentiable parameterization, the policy gradient admits the likelihood-ratio identity:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}\left[ \sum_{t=0}^\infty \nabla_\theta \log \pi_\theta(a_t|s_t) R_t(\tau) \right],$$

where $R_t(\tau)$ is the reward-to-go from $t$ onward (Kämmerer, 2019). This formula enables model-free estimation of the objective's gradient via unbiased or variance-controlled estimators from sampled trajectories (Kämmerer, 2019, Bai et al., 2021).

Mirror Descent and Natural Gradient

Mirror descent generalizes standard stochastic gradient by performing updates in a dual geometry defined by a strongly convex mirror map $\psi$. The Bregman-gradient update is

$$\theta_{k+1} = \arg\min_\theta \left\langle \hat{g}_k, \theta \right\rangle + \frac{1}{\eta_k} D_\psi(\theta, \theta_k),$$

where $D_\psi$ is the Bregman divergence (Huang et al., 2021). With $\psi(\theta) = \theta^\top F \theta$, this recovers natural gradient steps (Liu et al., 2023).

Variance Reduction and Experience Replay

Gradient-based PG estimators—especially in high-variance or limited-data settings—benefit from statistical techniques such as:

• Momentum/stochastic averaging (Huang et al., 2021).
• Multi-sample and replay buffer mixture importance ratios (Zheng et al., 2022).
• Optimistic evaluations/UCB bonuses for efficient exploration in the online RL regime (Liu et al., 2023).

3. Specialized Formulations and Theoretical Guarantees

Algorithmic Class	Sample Complexity (stationarity/optimality)	Notable Features
Vanilla PG/REINFORCE	$O(1/\epsilon^4)$ episodes (Huang et al., 2021)	On-policy, high variance, unbiased estimator
Bregman/VR-BGPO	$O(1/\epsilon^4)$/ $O(1/\epsilon^3)$ (Huang et al., 2021)	Mirror-descent and variance-reduced (STORM) updates
Risk-sensitive smooth PG (SF)	$O(1/\sqrt{N})$ stationarity (Vijayan et al., 2022)	Zeroth-order smoothed functional, mean-variance and distortion risk
Multi-objective PG	$O(M^4\sigma^2/\left[(1-\gamma)^8\epsilon^4\right])$ (Bai et al., 2021)	Independent batches for gradient/bias control, general concave scalarization
Quantile-based PG (QPO/QPPO)	Converges globally under two-time-scale (Jiang et al., 2022)	Likelihood-ratio of indicator/smooth kernel gradient of quantile
Optimistic NPG (linear MDPs)	$\tilde O(d^2/\epsilon^3)$ to $\epsilon$-opt. (Liu et al., 2023)	On-policy, Fisher-mirror step, UCB bonus for optimistic evaluation
Policy optimization as WGF	Convexity; global convergence in $W_2$ (Zhang et al., 2018)	Particle/measure-based, trust-region in Wasserstein space

In all cases, proper step-size, batch size, and surrogate/smoothing parameter tuning are essential for achieving stated rates (Kämmerer, 2019, Bai et al., 2021, Huang et al., 2021).

4. Non-Euclidean and Functional-Analytic Perspectives

Recent advances interpret gradient-based policy optimization as gradient flows in non-Euclidean spaces:

• Wasserstein gradient flows: Policies are probability measures, and optimization is a gradient flow with respect to the $W_2$ geometry, extending particle-based, trust-region, and entropy-regularized methods (Zhang et al., 2018).
• Bregman/mirror descent: The choice of Bregman divergence recovers natural PG, trust-region, or new updates (Huang et al., 2021).
• Policy-parameter versus policy-distribution optimization: Theoretical connections clarify how Gaussian policies plus entropy regularization induce implicit surrogate smoothing and continuation dynamics (Bolland et al., 2023).

These perspectives unify and generalize classical policy gradient, providing structure for efficiently shaping the optimization landscape and for extending to risk, multi-objective, and constrained formulations.

5. Modern Extensions: Multi-Objective, Risk, and Distributional Criteria

Gradient-based algorithms extend naturally to structured objectives:

• Multi-objective RL: Replace the scalar return with $J(\theta) = f(J_1(\theta),...,J_M(\theta))$ for a concave $f$. The policy gradient is given by a chain rule over $f$, with sample complexity $O(M^4\sigma^2/[(1-\gamma)^8\epsilon^4])$ (Bai et al., 2021).
• Risk-sensitive and robust RL: Incorporate a smooth or coherent risk measure (e.g., distortion, CVaR, mean-variance), requiring specialized gradient estimators and zeroth-order stochastic approximation when the measure is non-additive (Vijayan et al., 2022, Wang et al., 19 Sep 2025).
• Quantile/value-at-risk optimization: Estimate the gradient of the quantile using likelihood-ratio identities and two-timescale algorithms for stable convergence (Jiang et al., 2022).

For Bayesian-parameterized MDPs and general convex loss, risk-adjusted policy gradient estimators are derived from dual representations with rigorous finite-time analysis (Wang et al., 19 Sep 2025).

6. Implementation and Sample Efficiency Considerations

Key recommendations for implementation across algorithms include:

• Variance-reduced gradient estimators via momentum, importance mixing, or experience replay (Huang et al., 2021, Zheng et al., 2022).
• Adaptive step-size and batch-size selection following sample complexity scaling laws (Bai et al., 2021).
• Exploitation of function approximation and deep neural policies, with careful regularization, entropy bonuses, or trust constraints (Viquerat et al., 2021, Son et al., 2023).
• Proper problem-specific selection of update form and scaling/IW functions (e.g., parametric f in PPO/MLA family) for improved speed and stability (Gummadi et al., 2022).

Modern approaches also integrate analytical (“reparameterization trick”) gradients into LR-based PG frameworks, with adaptive interpolation to control variance and bias (Son et al., 2023).

7. Practical Impact and Limitations

Gradient-based policy optimization methods underpin most high-performing RL algorithms, including REINFORCE, policy-gradient actor–critic, TRPO, PPO, Soft Actor–Critic, and numerous recent risk-sensitive, distributional, or multi-objective variants. They have robust theoretical underpinnings, including global convergence for LQR/LQG and Markov jump LQ via gradient dominance and smoothness under stabilizability (Jansch-Porto et al., 2020, Han et al., 2023).

Limitations include high on-policy sample complexity for vanilla PG, sensitivity to step-size and surrogate choice, and non-global convergence in nonconvex landscapes absent specific regularity (local minima, reward sparsity, function approximation bias). Practical algorithms often incorporate trust-region, surrogate objectives, or hybrid approaches to address these challenges (Kämmerer, 2019, Zhang et al., 2018, Huang et al., 2021, Son et al., 2023).

Gradient-based policy optimization remains a central and rapidly developing family of algorithms, connecting stochastic optimization, information geometry, and reinforcement learning at both theoretical and practical levels.