Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration (2002.09253v4)

Abstract: Developmental machine learning studies how artificial agents can model the way children learn open-ended repertoires of skills. Such agents need to create and represent goals, select which ones to pursue and learn to achieve them. Recent approaches have considered goal spaces that were either fixed and hand-defined or learned using generative models of states. This limited agents to sample goals within the distribution of known effects. We argue that the ability to imagine out-of-distribution goals is key to enable creative discoveries and open-ended learning. Children do so by leveraging the compositionality of language as a tool to imagine descriptions of outcomes they never experienced before, targeting them as goals during play. We introduce IMAGINE, an intrinsically motivated deep reinforcement learning architecture that models this ability. Such imaginative agents, like children, benefit from the guidance of a social peer who provides language descriptions. To take advantage of goal imagination, agents must be able to leverage these descriptions to interpret their imagined out-of-distribution goals. This generalization is made possible by modularity: a decomposition between learned goal-achievement reward function and policy relying on deep sets, gated attention and object-centered representations. We introduce the Playground environment and study how this form of goal imagination improves generalization and exploration over agents lacking this capacity. In addition, we identify the properties of goal imagination that enable these results and study the impacts of modularity and social interactions.

Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration

The paper "Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration" addresses a crucial aspect of developmental machine learning: enabling artificial agents to autonomously generate and pursue imaginative, out-of-distribution goals as a mechanism for creative exploration and open-ended learning. The authors propose "imagine," an intrinsically motivated deep reinforcement learning (DRL) architecture that leverages the compositionality of language as a basis for generating novel goals.

Core Concepts

The paper is anchored on the premise that unlike traditional goal spaces that are often predefined or constrained within known distributions of effects, imaginative agents can benefit from the ability to conceive goals beyond experienced outcomes. This capability parallels the developmental process observed in children, where language serves as a cognitive tool that facilitates the generation of plans for novel and unexperienced outcomes.

Architecture and Methodology

Imagine Architecture: This architecture integrates several key components:

• Language Encoder and Reward Function: It includes an LSTM-based language encoder to map natural language (NL) goals into embeddings and a modular-attention (MA) system for learning a goal-achievement reward function. This function evaluates trajectories against the expected accomplishments as described by NL goals.
• Goal Generator: The system can generate target goals from known and imagined sets, facilitating exploration beyond the immediate experiences. Goal generation relies on leveraging social feedback — NL descriptions by a social partner — to enhance learning.

Playground Environment: A procedurally-generated testbed introduces variable scenes for interaction, allowing the agent to engage with objects across diverse dynamics and categories (e.g., animals, furniture, plants). It serves as a controlled environment to paper systematic generalization across predicates, attributes, objects, and categories.

Results and Implications

The paper reports substantial improvements in generalization and exploration capabilities for agents equipped with goal imagination. This imaginative approach yields higher success rates on unseen test goals and stimulates creative exploration, akin to how children invent and pursue their own problems during play. Agents benefit from this architecture by autonomously adapting their behaviors to novel, imagined goals — a phenomenon termed as behavioral adaptation.

Social Interactions and Modularity: Through the careful design of modular policy and reward functions, which ensure systematic generalization facilitated by object-centered representations and gated attention mechanisms, imagine agents achieve high efficiency in skill transfer. Additionally, results underscore the value of social feedbacks for learning, suggesting possibilities for more realistic human-agent interactions where descriptions may only be part-time available or less exhaustive.

Future Directions

The research sets the foundation for multiple future directions including scaling to more complex LLMs, integrating unsupervised representation learning for perception directly from pixels, and exploring different modalities of interaction with the social partner. There is potential for leveraging systematic generalization in hierarchical settings, and further optimizing goal selection and exploration strategies informed by internal evaluation metrics.

The investigation offers notable insights into how language as a cognitive tool can expand the scope and capability of autonomous learning systems, laying promising groundwork for developments in artificial intelligence that mirror cognitive and developmental processes observed in humans.