A Note on Loss Functions and Error Compounding in Model-based Reinforcement Learning

Abstract: This note clarifies some confusions (and perhaps throws out more) around model-based reinforcement learning and their theoretical understanding in the context of deep RL. Main topics of discussion are (1) how to reconcile model-based RL's bad empirical reputation on error compounding with its superior theoretical properties, and (2) the limitations of empirically popular losses. For the latter, concrete counterexamples for the "MuZero loss" are constructed to show that it not only fails in stochastic environments, but also suffers exponential sample complexity in deterministic environments when data provides sufficient coverage.

• The paper demonstrates that standard loss functions, such as the MuZero loss, contribute to error compounding in model-based RL, especially in stochastic environments.
• The paper contrasts theoretical error propagation guarantees with empirical results, pinpointing discrepancies caused by deterministic modeling and the use of L2 loss.
• The paper advocates exploring model-specific loss designs to bridge the gap between theoretical promises and practical performance.

Revisiting Loss Functions in Model-based Reinforcement Learning Through a Critical Lens

Overview of the Discussion

Model-based reinforcement learning (RL) has long been juxtaposed with model-free approaches, stirring debates on their respective merits and downsides. A pressing enigma in model-based RL is its infamous empirical vulnerability to error compounding—a stark contrast to its superior theoretical assurances. This note ventures into reconciling this paradox, providing a critical examination of the popular losses in the field of model-based RL, chiefly focusing on the "MuZero loss" and its limitations within both stochastic and deterministic environments.

Theoretical Underpinnings versus Empirical Observations

Empirically, model-based RL has been criticized for exacerbating error compounding, a claim contradicting its theoretically sound error propagation mechanisms. The Simulation Lemma highlights that the error propagation in model-based RL should, in theory, align with or surpass the performance of model-free counterparts. Yet, this theoretical promise seems to collapse under empirical scrutiny, an issue attributed to the nuanced divergence between theory and practice. Specifically, practice often employs deterministic models and $L_2$ loss, diverging from the stochastic models and MLE loss assumed in theory, thereby not minimizing total-variation error efficiently.

The Quandary of Loss Functions

The critique extends to popular empirical losses, including the bisimulation loss and the multi-step reward prediction loss (MuZero loss). The bisimulation loss, while theoretically appealing for its focus on task-relevant aspects of the environment, faces challenges in stochastic settings due to its inherent "double-sampling" problem. On the other hand, the MuZero loss, despite its initial allure for offering a unified metric across models, is critically examined for failing in stochastic environments. This critique is substantiated through propositions demonstrating scenarios where the true model (representing the real environment) fails to minimize the loss, leading to incorrect evaluations of policies.

Practical Limitations and Speculations on Future Directions

The note emphasizes that the practical application of model-based RL, particularly concerning the choice of loss functions, warrants a more nuanced understanding. It resonates with a call for future research to explore additional assumptions and structures that might mitigate the outlined limitations. Further, it underscores the potential of investigating properties of the learned model, rather than the true environment, for novel algorithmic insights, which might offer paths to overcoming the present challenges.

Concluding Remarks

This critical examination sheds light on the inherent limitations of popular loss functions in model-based RL, particularly in handling stochastic environments and achieving the theoretical promises of error propagation. The discussion candidly illustrates the chasm between theoretical ideals and empirical realities in model-based RL strategies, encouraging a recalibration of our approach to loss functions and their application in this domain. The note closes with an acknowledgment of the necessity for a deeper dive into the properties of the models learned through these methods, hinting at unexplored avenues that could potentially elevate the efficacy of model-based reinforcement learning.