Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

Abstract: In this paper we propose two algorithms in the tabular setting and an algorithm for the function approximation setting for the Stochastic Shortest Path (SSP) problem. SSP problems form an important class of problems in Reinforcement Learning (RL), as other types of cost-criteria in RL can be formulated in the setting of SSP. We show asymptotic almost-sure convergence for all our algorithms. We observe superior performance of our tabular algorithms compared to other well-known convergent RL algorithms. We further observe reliable performance of our function approximation algorithm compared to other algorithms in the function approximation setting.

• The paper presents novel RL algorithms (AC, CA, and AC-FA) that achieve almost-sure convergence for the stochastic shortest path problem.
• The paper demonstrates that the proposed methods outperform traditional approaches like Q-learning and SARSA in both random MDPs and benchmark environments.
• The paper provides thorough convergence analysis backed by extensive numerical experiments, validating stability and efficiency in various settings.

Convergent Reinforcement Learning Algorithms for Stochastic Shortest Path Problem

Introduction

The paper introduces three reinforcement learning algorithms designed for tackling the Stochastic Shortest Path (SSP) problem, which is a significant problem in reinforcement learning due to its applicability in various cost-criteria scenarios. The proposed methods include two novel algorithms for the tabular setting and one for the function approximation setting, all of which exhibit asymptotic almost-sure convergence.

Algorithm Design and Analysis

Tabular Setting Algorithms

In the tabular setting, the paper presents two algorithms: the Actor-Critic (AC) algorithm and the Critic-Actor (CA) algorithm. Both algorithms are established on two-timescale stochastic approximation, a crucial characteristic ensuring convergence to optimal solutions.

• Actor-Critic (AC) Algorithm: This algorithm updates value function parameters on a faster timescale compared to policy parameters, effectively mimicking policy iteration methods. The AC algorithm benefits from stability due to its on-policy nature, converging to optimal values effectively even when dealing with stochastic environments.
• Critic-Actor (CA) Algorithm: Unlike the AC, this algorithm leverages value iteration philosophies by reversing the timescales, updating policy parameters faster than value function parameters. Despite these differences, both algorithms provide stable learning and show minimal variances in convergence behavior.

Both algorithms deliver impressive performance in tabular environments, surpassing well-known algorithms like Q-learning and SARSA, particularly in scenarios demanding theoretically robust exploration techniques.

Function Approximation Algorithm

For environments requiring function approximation, the paper outlines an Actor-Critic algorithm tailored specifically for SSP problems.

• AC with Function Approximation (AC-FA): This algorithm is structured to accommodate linear function approximation, incorporating soft-max policy parameterizations for policies and value functions. The convergence of this algorithm is shown to be reliable, particularly vital due to the known instability of off-policy Q-learning algorithms in function approximation scenarios.

Convergence Analysis

The convergence analysis for both tabular and function approximation settings is comprehensive, leveraging existing theoretical frameworks for stochastic approximation. The tabular algorithms demonstrate convergence in environments satisfying the Basic Assumption, with potential extensions to more complex scenarios under the Standard Assumption by employing projection techniques.

Numerical Experiments

Extensive experiments are conducted to validate the algorithms’ performance. The novel algorithms outperform traditional methods in various test beds including random MDPs and benchmark environments like FrozenLake, evidencing both faster convergence and higher stability.

• Figure 1: Illustrates the performance on a random MDP, highlighting the effectiveness of the tabular algorithms in converging to optimal value functions.

  Figure 1: Random MDP with State space size |S|=20, Action space size |A|=4.

• Figure 2: Demonstrates superior performances of tabular AC and CA algorithms in FrozenLake environments, emphasizing stability under exploration and exploitation strategies.

  Figure 2: FrozenLake gym Environment with size 4x4(left) and 8x8(right).

• Figure 3: Highlights the convergence behavior of function approximation algorithms, with AC-FA showing reliable performance against Q-LFA and SARSA-LFA.

  Figure 3: Plots for comparison between AC-FA, Q-LFA and SARSA-LFA.

Conclusion

The study successfully extends the capabilities of reinforcement learning algorithms to address the SSP problem effectively, particularly under settings demanding both tabular and function approximation strategies. The proposed methods not only excel in convergence guarantees but also display robustness across both simple and complex environments.

Further research could explore the application of constrained optimization algorithms for SSP problems and develop enhanced natural actor-critic mechanisms to further improve convergence rates and stability. The exploration of varying assumptions beyond the Basic Assumption presents an exciting area for extending these algorithms’ applicability.