Three Dogmas of Reinforcement Learning (2407.10583v1)

Abstract: Modern reinforcement learning has been conditioned by at least three dogmas. The first is the environment spotlight, which refers to our tendency to focus on modeling environments rather than agents. The second is our treatment of learning as finding the solution to a task, rather than adaptation. The third is the reward hypothesis, which states that all goals and purposes can be well thought of as maximization of a reward signal. These three dogmas shape much of what we think of as the science of reinforcement learning. While each of the dogmas have played an important role in developing the field, it is time we bring them to the surface and reflect on whether they belong as basic ingredients of our scientific paradigm. In order to realize the potential of reinforcement learning as a canonical frame for researching intelligent agents, we suggest that it is time we shed dogmas one and two entirely, and embrace a nuanced approach to the third.

• The paper challenges core dogmas by critiquing environment-centric focus, static learning as fixed solutions, and reward-based goal formulation.
• The paper advocates for agent-centric modeling and continuous adaptation to promote robust, lifelong reinforcement learning.
• The paper calls for exploring alternative goal formulations beyond scalar rewards to better capture complex, real-world preferences.

An In-depth Analysis of "Three Dogmas of Reinforcement Learning"

The paper "Three Dogmas of Reinforcement Learning" by David Abel, Mark K. Ho, and Anna Harutyunyan puts forward a critical examination of three entrenched assumptions guiding much of contemporary reinforcement learning (RL) research. The authors propose that the field's progress is bound by these implicit dogmas, and suggest subtle shifts to better align with the goal of understanding and developing intelligent agents. Below, we provide a comprehensive and expert overview of the paper's core arguments, supporting evidence, and potential implications for future RL research.

Overview of the Three Dogmas

The authors identify and critique three dogmas pervasive within RL:

1. The Environment Spotlight: The tendency to focus on modeling environments rather than agents.
2. Learning as Finding a Solution: The treatment of learning as a finite process aiming to find a solution to a specific task.
3. The Reward Hypothesis: The assumption that all goals and purposes can be well-encoded as the maximization of a reward signal.

The Environment Spotlight

The first dogma, referred to as the "Environment Spotlight," emphasizes the predominant focus on environment-centric concepts. The authors argue that much of RL research is centered around the formalization of environments, such as Markov Decision Processes (MDPs) and their various extensions (e.g., bandit problems, POMDPs).

The authors critique this focus by highlighting the lack of a canonical mathematical model for agents analogous to the MDP for environments. They contend that without a formal model of an agent, we miss the opportunity to establish general principles of agency and address key questions about agent-centric concepts like resource constraints, agent-environment boundaries, and embedded agency.

Implication: To make RL a complete paradigm for the science of intelligent agents, agents themselves must become the central objects of paper. This requires developing formal models of agents, potentially leading to the discovery of fundamental laws governing their behavior and interaction with environments.

Learning as Finding a Solution

The second dogma revolves around the view of learning as a process of finding a solution to a specific task. The objective is typically framed as the convergence of an agent's policy to an optimal or near-optimal solution, after which learning ceases.

The authors argue for a shift towards viewing learning as an ongoing process of adaptation. This perspective moves away from the idea of a static "solution" and recognizes that learning involves continuous improvement and adaptation to new experiences and tasks.

Implication: By embracing learning as adaptation, future research can focus on designing agents capable of lifelong learning, continual improvement, and adaptation. This perspective aligns with the goals of lifelong and continual RL and may drive the development of new evaluation metrics and learning algorithms not tied to static task solutions.

The Reward Hypothesis

The third dogma is the Reward Hypothesis, which posits that all goals and purposes can be represented as the maximization of a reward signal. While this hypothesis has played a crucial role in advancing RL, the authors point out its limitations, particularly in capturing complex preferences and values which may not be reducible to scalar rewards.

Recent analysis has shown that the Reward Hypothesis imposes stringent conditions on the goals that can be encoded via reward functions. For instance, it assumes that preferences must satisfy the von Neumann-Morgenstern axioms and a "γ-Temporal Indifference" assumption, potentially excluding certain types of goals that involve risk sensitivity or multidimensional value trade-offs.

Implication: Recognizing the limitations of the Reward Hypothesis invites the exploration of alternative formulations of goals and purposes. This might include preferences, logical specifications, or composite objectives, broadening the scope of RL and potentially enhancing its applicability to more complex and nuanced real-world scenarios.

Discussion and Future Directions

The authors encourage the RL community to reflect on these entrenched assumptions and consider subtle yet significant shifts in research focus. They argue that RL is well-positioned to provide a holistic paradigm for the science of intelligent agents, but this requires moving beyond the three dogmas.

1. Agent-Centrism: Developing and formalizing models of agents comparable to MDPs for environments can help establish general principles of agency and foster a better understanding of key concepts in AI.
2. Adaptive Learning: Embracing continuous adaptation rather than static solutions can lead to the development of agents capable of lifelong learning, which is critical for real-world applications.
3. Diversified Goals: Exploring alternatives to the Reward Hypothesis can provide richer and more flexible ways to define and pursue goals, especially in complex domains involving varying and sometimes conflicting values.

Conclusion

The paper "Three Dogmas of Reinforcement Learning" provides a thought-provoking critique of current RL paradigms and suggests vital shifts to better align with the broader goals of AI research. By re-centering on agents, continuous learning, and diverse goal formulations, the RL community can advance towards a more comprehensive understanding and development of intelligent agents. This paper sets the stage for future research that challenges current conventions and explores new frontiers in RL.