Soft Actor-Critic for Discrete Action Settings (1910.07207v2)

Abstract: Soft Actor-Critic is a state-of-the-art reinforcement learning algorithm for continuous action settings that is not applicable to discrete action settings. Many important settings involve discrete actions, however, and so here we derive an alternative version of the Soft Actor-Critic algorithm that is applicable to discrete action settings. We then show that, even without any hyperparameter tuning, it is competitive with the tuned model-free state-of-the-art on a selection of games from the Atari suite.

• The paper adapts the Soft Actor-Critic framework to discrete domains, simplifying policy objectives while retaining its maximum entropy foundation.
• It modifies the soft Q-function and employs a softmax policy to directly compute discrete action probabilities for enhanced computational efficiency.
• Experimental results on Atari games show that SAC-Discrete outperforms Rainbow in half the cases and achieves competitive sample efficiency.

Soft Actor-Critic for Discrete Action Settings

This paper addresses a notable gap in reinforcement learning (RL) by extending the Soft Actor-Critic (SAC) algorithm, traditionally successful in continuous action settings, to accommodate discrete action domains. The focus is on adapting SAC to environments characterized by discrete actions, a common scenario in practical applications.

Background and Motivation

Reinforcement learning has seen significant advancements and successful implementations in various domains, such as video games and robotics. However, its adoption in real-world applications has faced obstacles due to inherent issues like sample inefficiency. The SAC algorithm, known for its state-of-the-art sample efficiency in continuous domains, presents a promising solution. The extension of SAC to discrete action settings aims to bridge its applicability and effectiveness to a broader range of environments, particularly those involving discrete actions like in the Atari suite.

Derivation of SAC-Discrete

The paper begins by revisiting the fundamentals of the SAC algorithm, highlighting its focus on optimizing a maximum entropy objective. The theoretical framework is maintained while the transition from continuous to discrete actions necessitates several algorithmic modifications:

1. Soft Q-Function Modification: The Q-function is redefined to output Q-values for each possible action, enhancing computational efficiency given the finite action space.
2. Policy Output Change: The policy is adjusted to directly output action probabilities through a softmax layer, replacing the need for mean and covariance outputs suitable for continuous actions.
3. Direct Expectation Calculations: For both the soft state-value function and temperature loss, the ability to directly compute expectations over discrete actions reduces variance in these estimates, negating the need for the Monte Carlo methods previously used in continuous settings.
4. Simplified Policy Objective: The reparameterization trick, necessary for gradient calculation in continuous actions, is no longer required. The policy objective is computed directly.

The proposed SAC-Discrete algorithm reflects these adjustments, effectively extending the SAC methodology while maintaining the core objective functions.

Experimental Evaluation

The performance of SAC-Discrete was assessed using a subset of 20 games from the Atari suite. When evaluated against the Rainbow algorithm— a leading model-free RL algorithm— SAC-Discrete demonstrated competitive results. Despite the absence of hyperparameter tuning for SAC-Discrete, it outperformed Rainbow in half of the games and was found to be competitive in terms of sample efficiency. These outcomes suggest that SAC-Discrete is a viable alternative for RL tasks involving discrete action settings.

Implications and Future Work

The derivation and testing of SAC-Discrete have several implications:

• Practical Applications: By extending SAC to discrete domains, the algorithm’s applicability broadens significantly, providing a new tool for tasks that were previously unsuitable for SAC.
• Sample Efficiency: As demonstrated by the results, SAC-Discrete holds potential for enhanced sample efficiency in discrete action settings, making it an attractive choice for real-time and resource-constrained environments.
• Further Research: Future work may explore hyperparameter tuning for SAC-Discrete to potentially improve its performance further. Moreover, extending this approach to hybrid action spaces, which include both discrete and continuous elements, could provide a comprehensive SAC framework applicable to all RL settings.

In summary, this paper successfully adapts SAC for discrete action environments, opening new pathways for its application in reinforcement learning. Through careful derivation and preliminary testing, the SAC-Discrete variant proves to be a substantial contribution to the field, offering a well-rounded approach to tackling discrete action challenges.