Infinite-Horizon Policy-Gradient Estimation

Abstract: Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a {\em biased} estimate of the gradient of the {\em average reward} in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter $\beta\in [0,1)$ (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter $\beta$ is related to the {\em mixing time} of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward

• The paper presents the MCG algorithm that approximates the average reward gradient with storage efficiency and a single discount parameter.
• It leverages simulation-based estimation in POMDPs without requiring explicit state knowledge, ensuring practical scalability.
• Convergence proofs and bias-variance bounds validate its robustness for policy improvement in diverse reinforcement learning settings.

Infinite-Horizon Policy-Gradient Estimation

The paper "Infinite-Horizon Policy-Gradient Estimation" by Jonathan Baxter and Peter L. Bartlett addresses the problem of gradient-based policy search for reinforcement learning (RL) in partially observable environments. The methodology proposed leverages simulation-based approaches to compute a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs), which are controlled by parameterized stochastic policies.

Main Contributions

The paper introduces the Markov Chain Gradient (MCG) algorithm for approximating the gradient of the average reward in a parameterized Markov chain. Key features of the algorithm include:

• Storage Efficiency: It requires storing only twice the number of policy parameters, thus maintaining computational efficiency.
• Single Free Parameter: The algorithm uses a discount factor $\beta \in [0,1)$ which provides a natural bias-variance trade-off mechanism.
• Independence from State Knowledge: It does not require knowledge of the underlying states of the Markov chain.

Theoretical Foundations

Gradient Approximation

The paper establishes that for any $\beta \in [0,1)$, the gradient $\nabla \eta$ of the average reward $\eta(\theta)$ can be expressed as:

$$\nabla\eta = (1-\beta)\nabla\pi'J_\beta + \beta\pi' \nabla P J_\beta$$

where $J_\beta(\theta)$ is the expected discounted reward. This factorization isolates a term that becomes negligible as $\beta$ approaches 1, providing a pathway to approximate the true gradient.

Convergence Proofs

The paper rigorously demonstrates that the MCG algorithm converges to a biased estimate of the gradient. Specifically, for a sufficiently small $\beta$ (yet close to 1), the biased estimate $\eta(\theta)$ generated by MCG converges to the true gradient $\nabla\eta(\theta)$. The convergence analysis applies to both discrete and continuous state, observation, and control spaces.

Numerical Results

Theoretical results are underpinned by bounds on the bias and variance of the estimates. The mixing time of the Markov chain, determined by its eigenvalues, influences these bounds. Remarkably, the approximation becomes accurate when $1-\beta$ is small relative to the inverse of the spectral gap of the transition probability matrix.

Extensions

The MCG algorithm is flexible and extendable to various RL settings:

1. Multiple Agents: It can handle multiple agents acting independently yet optimizing a common reward signal, leveraging a decentralized approach.
2. Policies with Internal States: Adaptations for policies with internal states, such as belief states in partially observable environments, are straightforward.
3. Higher-Order Derivatives: Extensions for computing higher-order derivatives (e.g., Hessians) enable second-order optimization algorithms.
4. Markov Chains with Non-Distinct Eigenvalues: Variance and bias bounds extend to chains with non-distinct eigenvalues, though these bounds involve the number of states in the chain.

Practical and Theoretical Implications

The proposed biased gradient estimation has significant implications in RL:

• Practical Optimization: MCG can be utilized in temporally extended decision-making problems, where state information is often incomplete or unknown.
• Robustness and Scalability: The reliance on parameterized stochastic policies and controlled bias-variance trade-off renders it practical and scalable for large-scale systems.
• Policy Improvement Assurance: Contrast to traditional approximate value function methods that may degrade on updating, MCG ensures that gradient-based updates directly aim to improve policy performance.

Future Developments

The research opens several avenues for future exploration:

• Continuous-Time Extensions: Formulating continuous-time analogs of the MCG algorithm.
• Alternative Filtering: Developing optimal filtering techniques for policy-gradient estimators to potentially enhance convergence rates and accuracy.
• Empirical Validations: Extensive empirical tests in various high-dimensional and partially observable environments are necessary to validate and refine the algorithm.

This paper makes a strong contribution to reinforcement learning by providing a robust, scalable method for policy improvement in partially observable settings. The MCG algorithm stands out for its simplicity, efficiency, and theoretical robustness, offering valuable tools for both theoretical research and practical RL applications.