---
title: Zeroth-Order Policy Gradient (ZPG)
url: https://www.emergentmind.com/topics/zeroth-order-policy-gradient-zpg
type: topic
---

# Zeroth-Order Policy Gradient (ZPG)

Zeroth-Order Policy Gradient (ZPG) methods are a class of reinforcement learning (RL) and control optimization algorithms that estimate policy gradients exclusively via function (or cost) evaluations, leveraging randomized finite-difference techniques without access to explicit gradients. ZPG algorithms are particularly suited to settings where policy or value functions are non-differentiable, black-box, or too complex to admit efficient analytical or automatic differentiation. Recent advances have rigorously established the theoretical and practical viability of ZPG—unifying it with policy optimization concepts, deriving convergence guarantees in both standard control (LQR, output feedback) and RL settings (actor-critic, meta-learning, RL from human feedback), and benchmarking its performance against contemporary RL algorithms [2006.07314, 2409.01477, 2503.00385, 2409.01477, 2601.19284, 2506.14460, 2602.18933, 2405.17370].

## 1. Fundamental Principles and Motivations

ZPG is defined by its reliance on derivative-free gradient estimates based on random perturbations of policy parameters or actions, using only cost/reward function evaluations. The approach is motivated by the following observations:

- Many real-world systems involve non-differentiable or black-box policies (e.g., rule-based controllers, simulators).
- Differentiability assumptions in standard policy gradient algorithms are sometimes violated, leading to biased updates or the necessity of unreliable function approximation.
- Finite-difference smoothing and perturbation approaches, central in black-box optimization, can be interpreted as stochastic policy gradient methods under a locally averaged objective [2506.14460].
- ZPG enables provable policy improvement even when explicit gradients of the cost, reward, or $Q$-function are unavailable or inaccurate.

The key theoretical insight is that the expectation of the finite-difference gradient estimator, under suitable randomization, equals the gradient of a smoothed objective. This parallels the REINFORCE/score-function gradient and admits well-controlled bias-variance tradeoffs, tunable by smoothing radius and sampling parameters [2506.14460, 2405.17370, 2006.07314].

## 2. Mathematical Foundations and Estimator Construction

The generic ZPG update arises from random smoothing identities. For a parameterized policy $\pi_\theta$, with objective $J(\theta) = \mathbb{E}_{s_0}[Q^{\pi_\theta}(s_0, \pi_\theta(s_0))]$, and finite-difference smoothing radius $\mu$, consider

\[
\widehat{g} = \frac{1}{m} \sum_{j=1}^m \frac{f(\theta+\mu u_j) - f(\theta)}{\mu}\, u_j, \quad u_j \sim \mathrm{Unif}(\mathbb{S}^{d-1})
\]

where $f(\cdot)$ denotes the cost/reward functional, and $u_j$ are random directions. This estimator is unbiased for the gradient of the mollified (smoothed) objective $J_\mu(\theta) = \mathbb{E}_u[f(\theta + \mu u)]$, with bias scaling as $O(\mu d)$ under $L$-smoothness [2506.14460, 2409.17401, 2405.17370].

A widely used alternative is the two-point estimator:

\[
\widehat{g} = \frac{1}{m} \sum_{j=1}^m \frac{f(\theta+\mu u_j) - f(\theta - \mu u_j)}{2\mu} u_j
\]

which reduces estimator bias for symmetric objective functions [2006.07314, 2405.17370]. In deterministic policy optimization, ZPG is often applied in action-space, yielding compatible updates for actor-critic architectures [2409.01477, 2006.07314].

In meta-learning and multi-task control, ZPG naturally extends to hierarchically smoothed objectives, using nested perturbations and Monte-Carlo outer/inner loops to provide unbiased estimators for meta-policy gradients [2503.00385, 2405.17370].

## 3. Algorithmic Instantiations and Pseudocode

Prototypical ZPG algorithms follow a stochastic gradient descent (SGD) paradigm, employing sample-based finite-difference policy updates. Generic pseudocode for the canonical case is as follows [2405.17370, 2503.00385, 2601.19284, 2602.18933, 2006.07314]:

1. **Sample Perturbations**: Draw $m$ random unit directions $\{u_j\}_{j=1}^m$ (in parameter or action space).
2. **Evaluate Costs**: For each $u_j$, evaluate the cost/reward at $\theta + \mu u_j$ (and optionally, $\theta - \mu u_j$).
3. **Estimate Gradient**: Form the averaged gradient estimate as above.
4. **Parameter Update**: Perform a stochastic descent/ascent step: $\theta \leftarrow \theta - \alpha \widehat{g}$.
5. **Repeat** until convergence criteria (e.g., norm of gradient estimator) is satisfied.

Specific applications build on this prototype:

- **LQR and Output Feedback Stabilization**: Perturb linear feedback gains, simulate rollouts under perturbed policies, estimate the cost, and compute the gradient estimator [2601.19284, 2602.18933].
- **Meta-Learning (MAML style)**: For each task, sample perturbations, take inner ZPG step(s), aggregate outer-loop meta-gradient estimator over tasks [2503.00385, 2405.17370].
- **Actor-Critic with ZPG (oCPG)**: Apply two-point ZPG to the action argument of the $Q$-function, integrate within delayed policy updates and replay buffer optimization [2409.01477].
- **RL from Human Feedback without Reward Model**: Estimate value differences via collected human preferences on pairs of trajectories, invert a known preference-link function, and use zeroth-order policy update [2409.17401].

## 4. Theoretical Properties and Convergence Guarantees

Rigorous upper bounds are established for estimator bias, variance, and sample complexity under regularity conditions (smoothness, boundedness, stability):

- **Bias**: For $L$-smooth objectives, $\|\mathbb{E}\,\widehat{g} - \nabla J(\theta)\| = O(\mu d)$; symmetric estimators can reduce this.
- **Variance**: Scales as $O(\frac{\operatorname{Var}[f]}{m\mu^2} d)$, controlled by perturbation batch size $m$ and smoothness.
- **Convergence Rates**: Under Polyak–Łojasiewicz (PL) or gradient domination, ZPG methods attain stationary points in $O(1/\epsilon^2)$ iterations for $L$-smooth objectives with bounded variance [2405.17370, 2503.00385, 2602.18933], and exhibit corresponding sample complexity scaling for LQR, meta-learning, and RLHF tasks [2409.17401].

In the RL context, ZPG can circumvent the incompatibility of gradient estimation under function approximation, providing provable convergence to stationary policies in nonconvex, black-box MDPs [2409.01477, 2006.07314].

For RL from human feedback, ZPG achieves the first polynomial query/sample complexity guarantees for stochastic MDPs without reward inference, despite high constants and slow scaling in parameter dimension and horizon [2409.17401].

## 5. Practical Considerations and Variance Reduction

Key algorithmic choices impact ZPG’s efficiency, robustness, and applicability:

- **Smoothing Radius $\mu$**: Trades off bias (decreases with $\mu$) and variance (increases with $\mu$). Adaptive or schedule-based tuning is common.
- **Perturbation Batch Size $m$**: Larger $m$ reduces variance; batch sizes in high action- or parameter-dimensions are commonly up to $O(1/\epsilon^2)$ for gradient-norm tolerance $\epsilon$.
- **Two-Point vs. One-Point Estimators**: Two-point estimators generally exhibit lower bias; one-point estimators are more query-efficient in certain settings [2506.14460, 2006.07314].
- **Variance Reduction**: Theoretical analysis reveals that symmetric baselines (central function value subtraction) act as optimal variance-reducing baselines in the policy gradient interpretation. Algorithms such as ZoAR further improve variance via averaged baselines and query reuse (experience replay), with provable gains in convergence and empirical performance [2506.14460].
- **Stability and Projection**: For control tasks (LQR), explicit construction of projections or initialization in stabilizing sets ensure that all iterates remain stabilizing, a property absent in pure first-order approaches [2503.00385, 2601.19284].

## 6. Applications and Benchmarks

ZPG approaches are applied in areas where gradient access is restricted or unreliable:

- **Model-free Control**: Stabilization of unknown linear and partially observed dynamical systems without requiring system identification [2601.19284, 2602.18933].
- **Meta-Policy Optimization**: Model-agnostic meta-policy learning across ensembles of ergodic LQRs with stability and sample complexity guarantees [2503.00385, 2405.17370].
- **Deterministic Policy Optimization in RL**: ZPG integrated into actor-critic frameworks improves compatibility, robustness, and outperforms or matches TD3, SAC on MuJoCo benchmarks under standard settings [2409.01477].
- **RL from Human Feedback**: Direct policy optimization using preference-based ZPG sidesteps reward modeling, supporting more general MDPs and preference-link functions with quantifiable sample/query efficiency [2409.17401].
- **Black-Box Optimization/Adversarial Attacks**: Finite-difference-based ZPG yields state-of-the-art query complexity when paired with variance-reduced schemes [2506.14460].

## 7. Limitations and Future Directions

Despite broad applicability, ZPG has inherent limitations:

- **Sample Complexity**: Higher than first-order methods, especially for small smoothing radii and high-dimensional parameter spaces.
- **Variance Scaling**: Increases with ambient dimension; advanced variance-reduction and block-coordinate techniques are proposed to mitigate this [2409.17401, 2506.14460].
- **Bias-Versus-Variance Tradeoff**: Requires problem-specific tuning of smoothing and batch size parameters.
- **Empirical vs. Theoretical Scaling**: Theoretical guarantees may be pessimistic compared to observed practical performance, particularly in the RLHF and high-dimensional continuous control regimes.
- **Convergence Rates**: Lower bounds for ZPG are typically $\mathcal O(1/\epsilon^2)$ in stationary point finding, compared to faster rates in convex or first-order smooth problems.
- **Hybrid Methods**: Research is ongoing into hybrid actor-critic/zeroth-order methods, off-policy variance reduction, and adaptive smoothing [2006.07314, 2409.01477].

Future research directions include integrating variance reduction via control variates or antithetic sampling, combining ZPG with off-policy data reuse, and extending convergence analysis to more general nonconvex RL objective landscapes [2409.01477, 2506.14460, 2409.17401].

Source: https://www.emergentmind.com/topics/zeroth-order-policy-gradient-zpg