--- title: Variance Reduced Policy Gradient url: https://www.emergentmind.com/topics/variance-reduced-policy-gradient type: topic --- # Variance Reduced Policy Gradient Variance reduced policy gradient (VRPG) methods comprise a family of algorithms in reinforcement learning (RL) that address the statistical inefficiency of policy gradient estimators by systematically controlling or minimizing the variance of the gradient estimate without sacrificing unbiasedness or convergence to optimality. High variance in standard stochastic policy gradients propagates as slow learning, suboptimal sample efficiency, and instability. The last decade has seen intensive development of variance-reduction techniques tailored for RL, spanning analytically structured control variates, off-policy experience replay, stochastic optimization adaptations (SVRG, SARAH, STORM), action-dependent and trajectory-level baselines, and advanced replay-buffer selection and weighting frameworks. These innovations have delivered quantifiable reduction in sample complexity and improved stability, leading VRPG to become a central paradigm underlying modern scalable policy optimization. ## 1. High Variance in Policy Gradients: Core Problem Statement Standard policy gradient methods, such as REINFORCE and actor-critic, estimate the gradient of the expected return $$ J(\theta) = \mathbb{E}\Big[\sum_{t=0}^{T-1}\gamma^t R(S_t, A_t)\Big], $$ via the Monte Carlo estimator $$ \tilde\nabla J(X) = \sum_t \gamma^t G_t(X) \nabla_\theta\log\pi_\theta(A_t|S_t), $$ where $G_t$ is the empirical return-to-go [2206.06827]. This estimator exhibits high variance because both $G_t$ and the score function $\nabla_\theta\log \pi$ are random. The excess variance compels the use of exceptionally small step sizes or large gradient-batch sizes, directly impeding learning efficiency. Variance-reduction techniques aim to retain unbiasedness while introducing structure (e.g., control variates, replay methods, recursive estimators) to systematically lower this variance. ## 2. Control Variates, Baselines, and the Limits of Classic Techniques The use of control variates in RL traces to the baseline trick: subtracting a value $b(S_t)$ from returns before multiplying by the policy score ensures unbiasedness but can meaningfully lower variance [2206.06827, 2107.04987]. In practice, baselines may be: - **State-dependent** (classic critic): fit by least-squares (A2C-style), - **Action-dependent**: constructed via Stein's identity or using the structure of factorized policies, leading to stronger variance reductions, especially for high-dimensional or continuous-action policies [1710.11198, 1803.07246]. Coordinate-wise and layer-wise vector baselines further reduce variance by optimizing the baseline at each parameter coordinate or layer [2107.04987]. These vector-structured baselines empirically and theoretically dominate standard scalar ones in variance reduction, particularly for deep/large policies. Recent work also exploits **trajectory-wise control variates**: by recursively expanding the law of total variance along the full trajectory, these methods eliminate not only the immediate (state-action) variance but also the variance arising from future-trajectory realizations. Theoretically, trajectory-wise CVs provably achieve the minimum residual variance among all admissible CVs that depend on current policy noise, and empirically speed up long-horizon RL [1908.03263]. ## 3. Off-Policy Sample Reuse: Experience Replay with Variance Reduction Off-policy methods seek to further improve sample efficiency by reusing experience not just from the current policy but also from past (behavior) policies. **Variance Reduction Experience Replay (VRER)** frameworks—both for full trajectories and per-step (partial trajectory) reuse—introduce sophisticated weighting, selection, and mixture strategies [2602.05379, 2205.02976, 2110.08902]. The key elements are: - **Importance sampling correction:** Each reused sample is weighted by a likelihood ratio $w_{i,k}(s,a) = \pi_{\theta_k}(a|s)/\pi_{\theta_i}(a|s)$ to ensure unbiasedness. - **Mixture likelihood ratio and selective reuse:** Instead of uniform replay, samples are weighted/admitted based on their estimated relevance (via variance, KL-divergence, or a variance upper-bound selection rule). This guarantees that only samples sufficiently similar to the current policy are reused, avoiding catastrophic variance inflation [2110.08902, 2205.02976]. VRER achieves $O(1/|U_k|)$ reductions in variance compared to on-policy estimators, modulo a selection constant $c$, while careful buffer and selection management bounds the bias from policy drift and Markov mixing [2602.05379]. The finite-time convergence theory explicitly quantifies buffer size, sample “age,” selection threshold, and their bias-variance trade-off. ## 4. Variance Reduction via Stochastic Optimization: SVRG, SARAH, STORM, and Loopless Protocols A distinct line of work adapts stochastic variance-reduction schemes from optimization—namely SVRG, SARAH, and STORM—to the RL context [1710.06034, 2003.04302, 2202.00308, 1905.12615, 2102.08607]. The essence is to construct a recursively updated auxiliary gradient estimator that blends a high-accuracy snapshot (“anchor”) with fast stochastic increments: - **SVRG-style:** Large-batch gradient computed at a “snapshot” parameter $\tilde \theta$; in the inner loop, small-batch updates correct the current estimate by adding the difference between gradients at $\theta$ and $\tilde\theta$, with importance sampling used for off-policy correction [1710.06034, 1905.12615]. - **SARAH/STORM:** Recursive momentum-style updates, where the gradient estimator at each iteration is an exponential moving average of the previous estimator and the current stochastic estimate (potentially with Hessian-vector corrections), obviating the need for periodic “restarts” [2003.04302, 2205.08253]. Loopless methods such as PAGE-PG randomize between large-batch and small-batch updates via a probabilistic switch, maintaining unbiasedness and obtaining sharp $O(\epsilon^{-3})$ or better sample complexity [2202.00308]. Truncation-based protocols (TSIVR-PG) further address the critical bottleneck of uncontrolled importance-weight variance by incorporating trust-region-style parameter updates. This provides rigorous control over IS variances and enables global sample-complexity results without unverifiable assumptions [2102.08607]. ## 5. Theoretical Guarantees and Sample Complexity A central metric for VRPG techniques is sample complexity: the number of environment interactions to achieve $\epsilon$-stationarity (i.e., $\mathbb{E}\|\nabla J(\theta)\|^2 \leq \epsilon^2$). The following table summarizes the leading results, where $K$ is the number of updates, $\epsilon$ the target accuracy, and $N, B$ denote large and small batch sizes: | Algorithm | Batch/epoch structure | Sample Complexity | Notable Properties | |------------------|------------------------|-------------------|--------------------------------------| | REINFORCE/GPOMDP | On-policy, SGD | $O(\epsilon^{-4})$| Baseline only, no variance reduction | | SVRPG | SVRG-epoch | $O(\epsilon^{-2})$| Improved by tighter IS analysis [1905.12615] | | TSIVR-PG | SVRG/truncation | $O(\epsilon^{-3})$| IS variance controlled, global rates, nonlinear objectives [2102.08607] | | SRVRPG/PAGE-PG | Loopless, recurs. | $O(\epsilon^{-3})$| No epoch tuning; minimal storage [2202.00308] | | STORM-PG | SARAH-momentum | $O(\epsilon^{-3})$| Exponential averaging; single-iteration loop [2003.04302] | | SHARP | Hessian-aided, mom. | $O(\epsilon^{-3})$| Checkpoint-free, IS-free [2205.08253] | | VRER (experience replay) | Buffer-based, offline | $O(\epsilon^{-3})$ asymp. | Selection rule bounds bias-variance [2602.05379] | | Multi-objective, nonlin. (MO-TSIVR-PG) | SVRG, nonlinear $f$ | $O(M^2 \epsilon^{-2})$ | $M$ objectives, improves $M$-dependence [2508.10608] | Variance-reduced approaches can dominate standard policy gradients by up to one or two orders of magnitude in sample complexity, with $O(\epsilon^{-2})$ rates feasible under stringent conditions such as global concavity and overparameterization [2102.08607]. ## 6. Empirical Analysis and Practical Recommendations Empirically, VRPG techniques consistently outperform baselines (vanilla policy gradient, A2C, on-policy PPO/TRPO) across benchmarks. Key findings include: - **Experience replay with variance-based selection** accelerates convergence and reduces policy and gradient-variance in both simple and high-dimensional tasks: PPO-VRER converges up to 50% faster and achieves higher asymptotic rewards across environments (e.g., CartPole, Hopper, Inverted Pendulum) [2602.05379, 2110.08902]. - **Empirical variance minimization of the control variate** (EV-based approaches) can achieve up to $10^2$–$10^3\times$ reduction in gradient-variance relative to normal A2C, yielding stable rewards and compressed learning curves [2206.06827]. - **Stochastic recursive, loopless, or Hessian-aided VRPG** (e.g., STORM-PG, SHARP) demonstrate superior sample efficiency and stability, eliminating the need for large checkpoint batches or IS, with stability improvements verified over many random seeds [2003.04302, 2205.08253, 2202.00308]. - **Action-dependent and vector-structured baselines** substantially reduce variance and increase sample efficiency, especially in continuous control and high-dimensional action spaces [1710.11198, 2107.04987, 1803.07246]. - **Buffer size and selection thresholds are critical:** Small buffers limit reuse; large ones introduce bias unless controlled via variance/KL-based criteria. Empirically, selection constants $c\approx1.02$–$1.06$ and buffer sizes $\approx$300–500 optimize the bias-variance trade-off [2602.05379]. - **Adaptive shrinkage baselines** (e.g., James–Stein) further improve training stability in large-scale RL from human feedback (RLHF) and large language model fine-tuning, with measurable reductions in gradient variance and improved final task performance [2511.03710]. ## 7. Advanced Directions and Open Problems Recent advances extend VRPG frameworks to: - **Average-reward infinite-horizon MDPs:** Implicit Gradient Transport and Hessian-based algorithms now achieve order-optimal regret bounds, $\tilde O(\sqrt{T})$, closing the gap to theoretical lower bounds for model-free RL in this setting [2404.02108]. - **Multi-objective RL (MORL):** Variance-reduction with control variates admits sample-complexity that scales only quadratically in $M$, the number of objectives, independent of state/action space dimension [2508.10608]. - **Generic policy parameterizations and global optimality:** Combining VRPG with natural policy gradients and function-approximation theory yields global convergence guarantees (modulo function-approximation error) and pushes practical algorithms closer to minimax-optimal sample efficiency [2211.07937]. Notable open questions include tightening the dependence on horizon $H$ and discount factor $1/(1-\gamma)$, generalizing without strong IS-variance assumptions, and extending provably optimal VRPG schemes to partially observed or multi-agent domains. --- **References** - [2602.05379] Variance Reduction Based Experience Replay for Policy Optimization - [2206.06827] Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization - [2110.08902] Variance Reduction based Experience Replay for Policy Optimization - [1710.06034] Stochastic Variance Reduction for Policy Gradient Estimation - [2205.08253] Momentum-Based Policy Gradient with Second-Order Information - [2003.04302] Stochastic Recursive Momentum for Policy Gradient Methods - [2202.00308] PAGE-PG: A Simple and Loopless Variance-Reduced Policy Gradient Method with Probabilistic Gradient Estimation - [2102.08607] On the Convergence and Sample Efficiency of Variance-Reduced Policy Gradient Method - [2404.02108] Order-Optimal Regret with Novel Policy Gradient Approaches in Infinite-Horizon Average Reward MDPs - [2508.10608] Variance Reduced Policy Gradient Method for Multi-Objective Reinforcement Learning - [2511.03710] Shrinking the Variance: Shrinkage Baselines for Reinforcement Learning with Verifiable Rewards - [2107.04987] Coordinate-wise Control Variates for Deep Policy Gradients - [1710.11198] Action-dependent Control Variates for Policy Optimization via Stein's Identity - [1803.07246] Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines - [2211.07937] An Improved Analysis of (Variance-Reduced) Policy Gradient and Natural Policy Gradient Methods - [1908.03263] Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods Source: https://www.emergentmind.com/topics/variance-reduced-policy-gradient