The Quest for a Common Model of the Intelligent Decision Maker (2202.13252v3)

Abstract: The premise of the Multi-disciplinary Conference on Reinforcement Learning and Decision Making is that multiple disciplines share an interest in goal-directed decision making over time. The idea of this paper is to sharpen and deepen this premise by proposing a perspective on the decision maker that is substantive and widely held across psychology, artificial intelligence, economics, control theory, and neuroscience, which I call the "common model of the intelligent agent". The common model does not include anything specific to any organism, world, or application domain. The common model does include aspects of the decision maker's interaction with its world (there must be input and output, and a goal) and internal components of the decision maker (for perception, decision-making, internal evaluation, and a world model). I identify these aspects and components, note that they are given different names in different disciplines but refer essentially to the same ideas, and discuss the challenges and benefits of devising a neutral terminology that can be used across disciplines. It is time to recognize and build on the convergence of multiple diverse disciplines on a substantive common model of the intelligent agent.

• The paper presents a unified model for intelligent decision-making featuring core components like perception, reactive policy, value function, and transition model.
• It standardizes diverse terminologies from psychology, AI, economics, control theory, and neuroscience to foster interdisciplinary collaboration.
• The model uses additive rewards for goal formulation, enabling the handling of complex trade-offs and scalable decision strategies.

The Common Model of the Intelligent Decision Maker

In "The Quest for a Common Model of the Intelligent Decision Maker," Richard S. Sutton argues for the establishment of a cross-disciplinary framework that captures the essence of intelligent decision-making. This proposition is grounded in the observation that various fields such as psychology, artificial intelligence, economics, control theory, and neuroscience, while distinct in methodologies and terminologies, share fundamental elements when considering decision-making agents. The aim is to identify these shared components and develop a neutral terminology to facilitate cross-disciplinary collaboration.

Core Perspectives and Definitions

The paper begins by acknowledging the shared interest in goal-directed decision-making across different disciplines and the potential of creating a unified model. Sutton introduces the concept of a "common model" of an intelligent agent, noting its independence from any specific organism, environment, or application domain. The proposed model includes several key elements: a goal-directed framework, interaction dynamics between the agent and its environment, and internal components essential for functioning—namely perception, decision-making, internal evaluation, and a world model.

Terminological Unification

One of the principal challenges highlighted is the establishment of neutral terminology. Terms like "agent" and "world" are preferred over domain-specific alternatives like "organism" or "environment" to support unbiased, cross-disciplinary thinking. Sutton argues for using "observations" instead of "stimuli" or "state" to maintain broad applicability.

Ground Rules for a Common Model

Sutton emphasizes that the common model should comprise concepts that are widely recognized across disciplines, excluding domain-specific details such as vision or language. He argues for a focus on the intersection of fields rather than the union, aiming to build a theoretically and practically viable model applicable across different domains of intelligent decision-making research.

Additive Rewards and Goal Formulation

Central to the model's architecture is the formulation of goals via additive rewards. This approach allows the handling of various complex scenarios, including goals of maintenance and time-uncertainty trade-offs. The scalar nature of these rewards aligns well with longstanding methods in operations research, economics, and AI.

Internal Structure of the Agent

The paper proposes a four-component framework for the internal structure of the agent: perception, reactive policy, value function, and transition model. These components are vital for processing observations, generating actions, evaluating potential decisions, and anticipating future states. Each component is discussed with respect to its prevalence and adaptability across disciplines.

• Perception: Converts experiences into subjective states for further evaluation.
• Reactive Policy: Maps state representations directly to actions.
• Value Function: Provides a measure of the desirability of specific states based on expected rewards.
• Transition Model: Predicts future states as a function of current actions and states, supporting planning and reasoning.

Implications and Limitations

The paper acknowledges potential criticisms and limitations, such as the exclusion of certain elements like exploration or intrinsic motivations due to a lack of universal consensus. However, the proposed model serves as a baseline—a reference point for researchers to delineate how new methodologies augment or modify the standard model.

Conclusion

Richard S. Sutton's paper advances an ambitious interdisciplinary dialogue on intelligent decision-making. By proposing a neutral, broadly applicable model, Sutton provides a foundation for converging diverse academic perspectives. The common model is not an endpoint but a means to stimulate further investigation and innovation across fields. Future developments may refine these ideas, leading to more integrated approaches in AI that unify insights from multiple disciplines.