Do deep reinforcement learning agents model intentions? (1805.06020v2)

Abstract: Inferring other agents' mental states such as their knowledge, beliefs and intentions is thought to be essential for effective interactions with other agents. Recently, multiagent systems trained via deep reinforcement learning have been shown to succeed in solving different tasks, but it remains unclear how each agent modeled or represented other agents in their environment. In this work we test whether deep reinforcement learning agents explicitly represent other agents' intentions (their specific aims or goals) during a task in which the agents had to coordinate the covering of different spots in a 2D environment. In particular, we tracked over time the performance of a linear decoder trained to predict the final goal of all agents from the hidden state of each agent's neural network controller. We observed that the hidden layers of agents represented explicit information about other agents' goals, i.e. the target landmark they ended up covering. We also performed a series of experiments, in which some agents were replaced by others with fixed goals, to test the level of generalization of the trained agents. We noticed that during the training phase the agents developed a differential preference for each goal, which hindered generalization. To alleviate the above problem, we propose simple changes to the MADDPG training algorithm which leads to better generalization against unseen agents. We believe that training protocols promoting more active intention reading mechanisms, e.g. by preventing simple symmetry-breaking solutions, is a promising direction towards achieving a more robust generalization in different cooperative and competitive tasks.

• The paper demonstrates that DRL agents encode other agents' target landmarks in hidden layers using a linear decoder.
• The paper shows that co-adaptation limits generalization when agents face external fixed-behavior counterparts.
• The paper proposes training enhancements, such as shuffling agent orders, to improve generalization in multi-agent environments.

Understanding Intention Modeling in Deep Reinforcement Learning Agents

The paper investigates the intriguing question of whether deep reinforcement learning (DRL) agents can explicitly represent the intentions of other agents during task execution. In a cooperative navigation setting, the authors explore if DRL agents trained using multi-agent reinforcement learning (MARL) paradigms are capable of modeling other agents' goals as part of their learned behavior.

Summary of Methods

The authors employ a cooperative navigation task wherein three agents must coordinate to cover three distinct landmarks. The paper leverages the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm to train the agents. The main focus is on determining if the hidden states of these agents reveal explicit information about the final goals of other agents. A linear decoder was utilized to predict the final landmarks covered by each agent from its hidden state, probing into the encoding of intentions within these network layers.

Additionally, the research examines the generalization capabilities of these agents by introducing external agents with fixed behaviors ("Sheldon" agents) to gauge if the trained agents can adapt to previously unseen policies or if their learned strategies are overfitted to the specific inter-agent dynamics they experienced during training.

Key Findings

1. Intention Representation: The paper demonstrates that DRL agents encode information about other agents' goals in their hidden states, which can be decoded using a linear classifier. Interestingly, the hidden layer representations contain more explicit information regarding intentions compared to the raw observations or the output actions, suggesting a transformation towards goal representation throughout the network's operation.
2. Generalization Challenges: Despite the ability to infer intentions from within the same training environment, the agents show significant co-adaptation to their training partners. This lack of generalizability becomes evident when interacting with agents deploying new strategies, highlighting a critical shortfall in adaptation.
3. Algorithmic Improvements for Better Generalization: The authors propose modifications to the training process, such as randomizing agent orders during episodes, to improve generalization. The MADDPG with shuffling shows an improved ability to generalize better when compared to the standard implementation and other tested modifications.

Implications and Future Directions

The findings underscore the potential for DRL agents to develop rudimentary Theory of Mind (ToM) capabilities within cooperative settings, an essential step towards more sophisticated intention reading in AI. The ability to model another agent's goals, albeit in simplified forms, is pivotal for advancing autonomous systems capable of nuanced interactions in both cooperative and competitive environments.

Practically, this research points towards developing more robust training protocols that can enhance generalization in multi-agent systems. The insights about co-adaptation and generalization are pertinent in fields like autonomous vehicle navigation and strategic game scenarios whereby agents frequently encounter novel opponent strategies.

Future work could explore more complex environments and task settings to further ascertain the scalability and robustness of these intention-reading capabilities. Additionally, investigating architectures such as recurrent networks or memory-augmented networks may provide deeper insights into capturing and utilizing temporal dynamics and historical behaviors for intention modeling, potentially paving the way for more advanced AI systems with enriched interactive capabilities.

Conclusion

This paper provides a foundational understanding of how DRL agents can begin to incorporate intention modeling into their decision-making processes. By exploring the enigma of representing intentions, the research contributes to the burgeoning field of AI research focused on agency and autonomy in multi-agent systems. The insights gained extend not only to the theoretical aspects of AI alignment with human cognitive processes but also offer practical avenues for enhancing the interoperability and adaptability of autonomous agents in diverse application domains.