Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games

Abstract: This work analyses the disparity in performance between Decision Transformer (DT) and Decision Mamba (DM) in sequence modelling reinforcement learning tasks for different Atari games. The study first observed that DM generally outperformed DT in the games Breakout and Qbert, while DT performed better in more complicated games, such as Hero and Kung Fu Master. To understand these differences, we expanded the number of games to 12 and performed a comprehensive analysis of game characteristics, including action space complexity, visual complexity, average trajectory length, and average steps to the first non-zero reward. In order to further analyse the key factors that impact the disparity in performance between DT and DM, we employ various approaches, including quantifying visual complexity, random forest regression, correlation analysis, and action space simplification strategies. The results indicate that the performance gap between DT and DM is affected by the complex interaction of multiple factors, with the complexity of the action space and visual complexity (particularly evaluated by compression ratio) being the primary determining factors. DM performs well in environments with simple action and visual elements, while DT shows an advantage in games with higher action and visual complexity. Our findings contribute to a deeper understanding of how the game characteristics affect the performance difference in sequential modelling reinforcement learning, potentially guiding the development of future model design and applications for diverse and complex environments.

• The paper presents a detailed comparison between Decision Transformer and Decision Mamba across 12 Atari games using quantifiable complexity metrics.
• It employs regression and correlation analyses to assess how action space and visual features, measured by image entropy and compression ratio, influence performance.
• Results indicate that DT’s transformer design excels in complex environments through effective long-term dependency handling, while DM performs better in simpler settings.

Analyzing Sequential Decision Making: A Comparative Study of Decision Transformer and Decision Mamba in Atari Games

The paper entitled "Decision Transformer vs. Decision Mamba: Analysing the Complexity of Sequential Decision Making in Atari Games" by Ke Yan provides a comparative analysis of two prominent sequence modeling architectures—Decision Transformer (DT) and Decision Mamba (DM)—in the context of solving reinforcement learning (RL) tasks using Atari games as a benchmark. The study articulates a detailed framework for understanding the performance dynamics and limitations of DT and DM, focusing on their architectural differences and the complex interactions between game characteristics.

The crux of the paper revolves around the disparity in performance exhibited by DT and DM across various Atari games. Initial observations identified a trend where DM tends to perform better in simpler environments like Breakout, while DT excels in more complex environments such as Hero. This study was expanded to a pool of 12 games for a comprehensive understanding.

Methodology and Key Findings

The methodology employed involves quantifying game characteristics, employing regression and correlation analysis, and the application of action space simplification strategies to determine the factors influencing model performance. Game features encompassed action space complexity and visual complexity, with metrics such as image entropy, compression ratio, and feature count providing insights into visual attribute contributions.

Strong findings indicated that action space complexity plays a pivotal role in model performance. DM shows superior results in games with relatively simple visual and action elements, whereas DT's architectural design, specifically its transformer-based sequence modeling, grants it an advantage in environments with higher complexity in both action space and visual elements. The DT's self-attention mechanism effectively handles long-term dependencies, explaining its superior performance in more sophisticated game setups.

The research employed Random Forest regression analysis to ascertain the relative importance of various game characteristics, revealing that while action complexity is significant, visual complexity—particularly measured by compression ratio—is influential. Moreover, the introduction of action fusion techniques, like Simple Action Fusion and Frequency-based Action Fusion, provided insights into handling complex decision-making processes through simplification without losing vital game dynamics.

Implications and Future Directions

This study provides pivotal insights into the challenges and potential of using sequence modeling for RL tasks, particularly emphasizing the interaction between environmental complexity and architectural capabilities of DT and DM. The findings suggest that architectural features in decision-making models need to account for both action and visual complexity to optimize performance.

The implications of this research extend beyond Atari games, potentially guiding the development of more sophisticated models capable of handling a wider array of environments. As the field progresses, there is room for evolving these architectures, possibly through hybrid models that harness the strengths of both DT and DM. Future research could explore the theoretical underpinnings of attention mechanisms and state-space models to enhance the adaptability and robustness of these models in diverse RL tasks.

Overall, this paper makes a significant contribution to understanding the efficacy and limitations of sequence modeling architectures in RL, offering practical guidance for developing future models adept at tackling the intricate landscape of sequential decision-making tasks.