Language and Culture Internalisation for Human-Like Autotelic AI (2206.01134v2)

Abstract: Building autonomous agents able to grow open-ended repertoires of skills across their lives is a fundamental goal of AI. A promising developmental approach recommends the design of intrinsically motivated agents that learn new skills by generating and pursuing their own goals - autotelic agents. But despite recent progress, existing algorithms still show serious limitations in terms of goal diversity, exploration, generalisation or skill composition. This perspective calls for the immersion of autotelic agents into rich socio-cultural worlds, an immensely important attribute of our environment that shapes human cognition but is mostly omitted in modern AI. Inspired by the seminal work of Vygotsky, we propose Vygotskian autotelic agents - agents able to internalise their interactions with others and turn them into cognitive tools. We focus on language and show how its structure and informational content may support the development of new cognitive functions in artificial agents as it does in humans. We justify the approach by uncovering several examples of new artificial cognitive functions emerging from interactions between language and embodiment in recent works at the intersection of deep reinforcement learning and natural language processing. Looking forward, we highlight future opportunities and challenges for Vygotskian Autotelic AI research, including the use of LLMs as cultural models supporting artificial cognitive development.

Overview of Vygotskian Autotelic AI

The paper "Language and Culture Internalisation for Human-Like Autotelic AI" presents an approach to enhancing the cognitive abilities of artificial agents by leveraging socio-cultural interactions, particularly through language. The authors propose advancing the reinforcement learning (RL) paradigm by integrating elements inspired by Vygotsky's socio-cultural theory, thus pioneering a new area termed "Vygotskian Autotelic AI." This concept aims at developing artificial agents that, like humans, can learn a wide array of skills throughout their lifetimes by internalizing language and cultural norms, facilitating higher cognitive functions such as abstraction, generalization, and imagination.

Background and Motivation

Reinforcement Learning has traditionally focused on pre-defined reward structures to encourage skill acquisition. However, human-like open-ended skill development necessitates improvisation in goal setting and adaptability to dynamic tasks, inferior in current RL frameworks. Autotelic agents, drawing from Piagetian developmental psychology, embody intrinsic motivation to self-generate goals. Despite some advancements, these agents still exhibit limited goal diversity and exploration abilities.

The paper highlights the pivotal role that immersion in a rich socio-cultural environment, analogous to human cognitive development, could play in overcoming these limitations. Inspired by Vygotsky's work, internalizing linguistic interactions is posited as essential for evolving new cognitive capabilities in autotelic agents. Vygotskian Autotelic AI agents would use language and culture not just as a vehicle of communication but as a cognitive tool, reshaping learning trajectories and mental representations to align more closely with human-like development.

Core Contributions

The paper makes several contributions, crucial among them being the notion of 'internalization' through socio-cultural interactions enabled by language. Key aspects include:

1. Language-Induced Cognitive Development:
   • The mechanism by which language aids in abstract thinking is central to the discourse. Language labels facilitate categorization and the organization of thoughts. By inducing systematic generalization and abstraction, language prompts the development of new skill sets in RL agents that are beyond simple sensory-motor experiences.
2. Role of LLMs:
   • LLMs serve as cultural models encapsulating vast bodies of knowledge, values, and norms. These models can guide agents in understanding and adopting socio-cultural constructs, making them potential avenues for learning abstract concepts and sophisticated planning strategies.
3. Internalization Mechanisms:
   • By learning to generate linguistic cues internally, agents can create self-guided learning processes analogous to human private and inner speech. This involves goal imagination, problem decomposition, and alignment with cultural ethics, augmenting autonomy and enhancing problem-solving capabilities.
4. Challenges and Opportunities:
   • The authors address several challenges, such as the need for designing socio-culturally rich learning environments, more efficient internal language production, and the reliable use of LLMs devoid of biases. These setups could foster agents that carry out long-term, culturally informed goal pursuits.

Implications and Future Directions

The proposed Vygotskian Autotelic AI promises pivotal advancements in AI by fostering agents capable of more nuanced and sophisticated interactions akin to human cognitive growth. Practical implications of this work include the potential for creating assistive agents that integrate smoothly into human social structures, enabling more effective human-machine collaboration.

Theoretical implications call for further exploration of how language and cognitive development interact in both humans and machines, emphasizing how cultural artifacts can be utilized in AI development. Future research could explore refining internal LLMs, ensuring they remain culturally relevant and aligned.

In conclusion, the paper sets a foundational perspective on integrating socio-cultural learning paradigms with AI development, aiming to craft autonomous agents that not only perform tasks but also mirror the cognitive flexibility and adaptability characteristic of human learning. This approach acts as a springboard for further exploration into embedding cultural intelligence within artificial agents, marking a significant shift in developing more socially aware and human-like intelligences.