Policy-Gradient Reinforcement Learning

• Policy-Gradient Reinforcement Learning is a direct policy optimization method that computes gradients to maximize the expected cumulative reward.
• It leverages stochastic approximations, neural networks, and actor-critic architectures to effectively navigate high-dimensional or continuous spaces.
• This approach underpins diverse algorithms—including REINFORCE, PPO, and risk-sensitive models—enabling safe exploration and scalable performance.

Policy-Gradient Reinforcement Learning (PGRL) is a set of direct policy-optimization methods fundamental to both classical and contemporary reinforcement learning. These algorithms ascend the expected cumulative reward or surrogate objectives using the gradient of a parameterized policy, often leveraging stochastic approximation principles and neural function approximation. Unlike value-based methods, PGRL directly parameterizes and improves a stochastic policy through sample-based estimates of the policy gradient. This approach supports rich model parameterizations, facilitates learning in high-dimensional or continuous spaces, and underpins a range of robust, scalable RL methodologies.

1. Policy-Gradient Formulations and Theorems

Classical policy-gradient methods represent the RL objective as the expected discounted return under a parameterized stochastic policy $\pi_\theta(a|s)$, frequently modeled by neural networks (Phon-Amnuaisuk, 2018). The standard policy-gradient theorem expresses the gradient of the objective as:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\bigg[\sum_{t=0}^\infty \gamma^t Q^{\pi_\theta}(s_t, a_t)\nabla_\theta \log \pi_\theta(a_t|s_t)\bigg]$$

where $Q^{\pi_\theta}(s,a)$ is the state-action value function. Variants include actor-only methods (REINFORCE), actor-critic architectures (A2C, PPO), off-policy extensions, and risk-sensitive or constraint-aware gradients.

Recent work has extended policy-gradient theory to general utilities as non-linear functionals of the occupancy measure $\mu^\pi(s, a)$ (Kumar et al., 2022), distortion risk objectives (Vijayan et al., 2021), performative MDPs (Basu et al., 23 Dec 2025), and formulations in continuous time and space (Jia et al., 2021). Weak-derivative methods replace the score-function estimator, reducing variance and facilitating unbiased, almost-sure convergence (Bhatt et al., 2020).

2. Algorithms, Architectures, and Training Strategies

PGRL algorithms encompass:

• REINFORCE and Variants: On-policy gradient ascent using Monte Carlo return estimates (Phon-Amnuaisuk, 2018).
• Actor-Critic: Simultaneous learning of a policy (actor) and value function (critic), with the critic providing a learned baseline for variance reduction (Phon-Amnuaisuk, 2018).
• Hybrid Estimators: Convex combinations of unbiased and biased estimators (REINFORCE/SARAH-style), yielding improved sample complexity $O(\varepsilon^{-3})$ for composite policy optimization (Pham et al., 2020).
• Distributional Policy Gradients: Critic architectures ingesting full quantile information via Implicit Quantile Networks (IQN) or comparable methods for superior sample efficiency and expressivity (Jeon et al., 2024).
• DAG-based Meta-Learning: Direct encoding of PGRL algorithms (VPG, PPO, DDPG, TD3, SAC) as directed acyclic graphs for automated meta-optimization and architecture search (Luis, 2020).

Neural architectures span feed-forward networks, CNNs, RNNs, and Transformer-based designs with specialized regularization (e.g., consistent dropout for stability in large models (Hausknecht et al., 2022)).

3. Exploration, Robustness, and Safe Learning

Exploration is a central issue in PGRL—vanilla approaches exhibit restricted coverage and local convergence. Algorithms such as PC-PG employ an ensemble "policy cover" updated with each episode, granting robust ensemble-based exploration using feature-space bonuses and off-policy updates (Agarwal et al., 2020). Smoothing policies and adaptive meta-parameter scheduling ensure monotonic improvement with high probability, supporting safe deployment on physical systems by constraining policy update magnitude and batch size (Papini et al., 2019).

Safe learning with probabilistic constraints has been formalized; explicit gradient expressions for maintaining state trajectories within designated safety sets at predetermined probability levels enable direct integration into actor-only and actor-critic policy-gradient loops (Chen et al., 2022).

Policy adaptation using automatic step-size selection via Polyak principles is now viable, mitigating manual learning-rate tuning and improving convergence stability (Li et al., 2024).

4. Generalized Objectives: Risk, Utility, and Performative Adaptation

PGRL has been generalized to maximize concave utilities, risk measures, or accommodate changes induced by the deployed policy:

• Distortion Risk Measures (DRM): Policy-gradient algorithms estimating gradients for coherent risk objectives via the Choquet integral, either on-policy or with off-policy trajectory reuse and likelihood-ratio corrections. This supports tail-risk and CVaR optimization in RL (Vijayan et al., 2021).
• Nonlinear Utilities: Policy-gradient theorems and sample-based algorithms for arbitrary differentiable functions of the state-action occupancy measure, applicable to pure exploration, information gain, imitation, and constrained RL (Kumar et al., 2022, Zhang et al., 2020).
• Performative Policy Gradient: In performative settings, the environment is responsive to the policy itself, shifting transitions and rewards as a function of the agent’s deployment. The PePG algorithm augments the score-function gradient with additional terms capturing these shifts, converging (provably) to performatively optimal policies (Basu et al., 23 Dec 2025).

5. Dimensionality Reduction, Surrogate Models, and Inverse Problems

Advanced applications of PGRL include infinite-dimensional SBOED for PDE-constrained inverse problems (Shen et al., 9 Jan 2026):

• The experimental design process is cast as a finite-horizon MDP.
• The policy is a neural network mapping experiment history to design actions.
• Dual dimension reduction: active subspace projection for parameter space, principal component analysis for state space.
• Highly scalable surrogate models (LANO) inform reward evaluation and gradient propagation.
• Efficient Laplace-based D-optimality rewards (and alternatives) drive high information gain, and the full pipeline is amortized—once trained, policies may be evaluated online without repeated optimization.

6. Empirical Results and Applications

PGRL methods have demonstrated practical success across diverse domains:

• Atari and MuJoCo: Distributional critics in policy-gradient pipelines (PG-Rainbow) outperform baseline PPO on a majority of Atari games, achieving superior mean scores and enhanced sample efficiency (Jeon et al., 2024). Consistent dropout regularization enables stable online training of architectures with inherent dropout such as GPT, matching or exceeding vanilla baselines even for high dropout probabilities (Hausknecht et al., 2022).
• Wireless Optimization: PGRL yields robust, always-on model-free association in wireless networks, achieving monotonic cost reduction and implementation scalability alongside guaranteed local convergence, outperforming Q-learning in robustness (Combes et al., 2013).
• Experimental Design: In sequential sensor placement for PDE inverse problems, PGRL + LANO achieves approximately 100× speedup in utility evaluation over finite element baselines, wins in >97% of test cases against random placements, and produces interpretable policies (such as "upstream" tracking) (Shen et al., 9 Jan 2026).
• Safety and Constraints: Demonstrated safety improves with principled constraint penalties, balancing collision avoidance and trajectory optimality in continuous navigation tasks (Chen et al., 2022).

7. Theoretical Foundations and Sample Complexity

Modern work provides rigorous convergence and sample complexity results for PGRL extensions:

• Weak-derivative gradients admit lower-variance estimators, almost-sure convergence to stationary points, and $O(1/\sqrt{k})$ sample complexity; theoretical variance improvements are significant for Gaussian policies (Bhatt et al., 2020).
• Hybrid estimators achieve $O(\varepsilon^{-3})$ trajectory complexity for composite objectives, outperforming prior REINFORCE and SVRPG methods (Pham et al., 2020).
• Risk-sensitive, generalized-utility, and performative PG theorems preserve implementability—bias/variance proofs, stationarity rates, and robust improvement guarantees are available (Kumar et al., 2022, Vijayan et al., 2021, Basu et al., 23 Dec 2025, Zhang et al., 2020, Papini et al., 2019).

• “Sequential Bayesian Optimal Experimental Design in Infinite Dimensions via Policy Gradient Reinforcement Learning” (Shen et al., 9 Jan 2026)
• “Learning to Play Pong using Policy Gradient Learning” (Phon-Amnuaisuk, 2018)
• “Policy Gradient using Weak Derivatives for Reinforcement Learning” (Bhatt et al., 2020)
• “Policy Gradient Methods for Distortion Risk Measures” (Vijayan et al., 2021)
• “Consistent Dropout for Policy Gradient Reinforcement Learning” (Hausknecht et al., 2022)
• “The association problem in wireless networks: a Policy Gradient Reinforcement Learning approach” (Combes et al., 2013)
• “PG-Rainbow: Using Distributional Reinforcement Learning in Policy Gradient Methods” (Jeon et al., 2024)
• “Cold-Start Reinforcement Learning with Softmax Policy Gradient” (Ding et al., 2017)
• “A Hybrid Stochastic Policy Gradient Algorithm for Reinforcement Learning” (Pham et al., 2020)
• “Performative Policy Gradient: Optimality in Performative Reinforcement Learning” (Basu et al., 23 Dec 2025)
• “Policy Gradient for Reinforcement Learning with General Utilities” (Kumar et al., 2022)
• “Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms” (Jia et al., 2021)
• “Policy Gradient Algorithms with Monte Carlo Tree Learning for Non-Markov Decision Processes” (Morimura et al., 2022)
• “Combining policy gradient and Q-learning” (O'Donoghue et al., 2016)
• “Policy Gradients for Probabilistic Constrained Reinforcement Learning” (Chen et al., 2022)
• “Policy Gradient RL Algorithms as Directed Acyclic Graphs” (Luis, 2020)
• “Enhancing Policy Gradient with the Polyak Step-Size Adaption” (Li et al., 2024)
• “Smoothing Policies and Safe Policy Gradients” (Papini et al., 2019)
• “PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning” (Agarwal et al., 2020)
• “Variational Policy Gradient Method for Reinforcement Learning with General Utilities” (Zhang et al., 2020)