Autotelic AI: Autonomous Goal-Driven Agents

Updated 4 July 2026

Autotelic AI is a paradigm where agents autonomously generate and pursue self-defined goals guided by intrinsic motivations.
It leverages goal-conditioned reinforcement learning, hindsight experience replay, and structured representations (e.g., language, graphs) to develop diverse skills.
Key challenges include accurate goal representation, competence estimation, and safe exploration, with applications in robotics, text environments, and social AI.

Autotelic AI denotes artificial agents that autonomously represent, generate, select, and pursue self-defined goals, guided by intrinsic motivations rather than externally supplied task rewards or fixed human preferences. In the contemporary literature, the paradigm is usually formalized through goal-conditioned reinforcement learning in reward-free or rewardless settings, where agents build open-ended repertoires of skills by organizing their own curricula over goals, outcomes, and competencies. Across developmental robotics, multi-goal RL, language-grounded agents, and LLM-based online RL, autotelicity is treated not as mere curiosity, but as the coupled problem of goal representation, goal sampling, competence estimation, hindsight reuse of experience, and skill acquisition under sparse feedback (Srivastava et al., 6 Feb 2025, Colas et al., 2020, Teodorescu et al., 2022).

1. Conceptual foundations

Autotelic AI emerged from developmental approaches that contrast with standard extrinsic-reward RL. In standard RL, a policy is optimized against a predefined reward $R(s,a)$ for known tasks. In autotelic RL, the agent instead constructs internal goal representations $g \in G$ , samples goals from its own distributions $P(g)$ , and learns goal-conditioned policies that maximize mastery and diversity over self-generated goals. This formulation is tied to Intrinsically Motivated Goal Exploration Processes (IMGEPs), multi-goal RL, and developmental robotics, and it explicitly distinguishes autotelic learning from unsupervised RL that maximizes surrogate objectives such as state coverage without necessarily organizing behavior around explicit, self-generated goals with competence tracking (Srivastava et al., 6 Feb 2025, Colas et al., 2020).

A central distinction in the literature is between knowledge-based and competence-based intrinsic motivations. Knowledge-based signals drive improvement of world models through novelty, prediction error, or information gain. Competence-based signals drive improvement in the agent’s ability to achieve goals, typically through competence estimates and learning progress. This distinction matters because autotelic systems must both discover what can be controlled and decide what is worth practicing next. The usual developmental interpretation is that open-ended skill acquisition requires both components: novelty and uncertainty help discover candidate goals, while competence progress organizes a curriculum over goals that are neither trivial nor hopeless (Srivastava et al., 6 Feb 2025, Colas et al., 2022).

Autotelic AI is also framed against two developmental traditions. A Piagetian perspective emphasizes learning through physical interaction and intrinsic motivation in the environment. A Vygotskian perspective emphasizes social scaffolding, language, and cultural transmission. Recent work argues that advanced autotelic agents should couple both: autonomous goal pursuit remains central, but social partners, linguistic feedback, and culturally evolved abstractions can reshape the goal space, support decomposition, and overcome exploration limits that arise when the agent is constrained to goals drawn only from prior experience (Akakzia et al., 2022, Colas et al., 2022).

2. Formalization and intrinsic motivation

A standard formalization models autotelic learning in a reward-free MDP $M = (S, A, T, \gamma)$ , augmented with an outcome space $O$ , a goal space $G$ , and a success indicator $I(o,g)$ . Goal-conditioned policies and value functions are written as $\pi_\theta(a \mid s,g)$ , $Q_\theta(s,a,g)$ , and $V_\theta(s,g)$ . A canonical mastery objective is

$g \in G$ 0

often accompanied by diversity objectives such as maximizing the entropy of the outcome distribution. Goal selection can be driven by learning progress through a distribution of the form $g \in G$ 1, where $g \in G$ 2 and $g \in G$ 3 is a competence estimator (Srivastava et al., 6 Feb 2025).

The literature uses several classes of intrinsic signals. Knowledge-based formulations include curiosity via prediction error,

$g \in G$ 4

information gain,

$g \in G$ 5

and novelty measures such as $g \in G$ 6 or count-based rarity. Competence-based formulations include empowerment,

$g \in G$ 7

and goal-wise competence progress, where goals are sampled preferentially from regions in which the agent is currently improving. These mechanisms recur across surveys and implementations, but the field increasingly treats them as components of a larger autotelic loop rather than interchangeable exploration bonuses (Srivastava et al., 6 Feb 2025, Colas et al., 2022).

Experience replay and hindsight relabeling are equally central. In the RL-IMGEP view, the agent stores transitions obtained while pursuing one goal and later relabels them with achieved goals, creating dense training signal in sparse environments. This extends the logic of HER to autotelic settings: trajectories collected under one intention can still improve competence on many other goals. The same principle reappears in language-grounded agents, where achieved goals are extracted from trajectories by a relabeler or a social partner and then used to supervise goal-conditioned learning (Colas et al., 2020, Colas et al., 2023).

A common misconception is that autotelic AI is equivalent to curiosity-only exploration. The comparative analyses explicitly reject this reduction. Curiosity-only systems may chase noise or stochasticity without developing stable goal repertoires, whereas autotelic RL is distinguished by explicit goal/outcome spaces, success criteria, competence estimation, and curricula over goals. This suggests that autotelicity is better understood as a structured control-and-curriculum problem than as undirected novelty seeking (Srivastava et al., 6 Feb 2025).

3. Goal representations, language, and LLM-based autotelic agents

Goal representation is a primary bottleneck. Surveys distinguish goals as choices between objectives, target features in state space, image-based latent targets, abstract predicates, mixtures of objectives, and language descriptions. The recurring criterion is that the goal space must be compact enough for conditioning and generalization, yet expressive enough to support abstraction, recombination, and open-ended expansion. This is why language has become a major substrate for autotelic AI: language goals are described as abstract and combinatorial, and textual environments reduce the grounding problem because both observations and goals live in the same modality (Colas et al., 2020, Teodorescu et al., 2022).

Text worlds are particularly favorable because they exhibit depth, breadth, niches of progress, and direct linguistic goal grounding. Depth supplies nested multi-step skills; breadth supplies many paths and combinatorial interactions; niches of progress supply mixtures of easy, learnable, and unlearnable goals; and language observations simplify goal-satisfaction detection relative to pixel-space grounding. This motivates the study of language-conditioned autotelic agents in environments such as TextWorld, Interactive Fiction, ScienceWorld, Playground-Text, CookingWorld, and Little-Zoo (Teodorescu et al., 2022, Teodorescu et al., 2023).

Several architectures operationalize this idea. LMA3 uses a pretrained LLM as a model of cultural transmission to support three modules: a relabeler that describes goals achieved in trajectories, a goal generator that proposes new high-level goals together with decompositions into mastered subgoals, and reward functions that judge whether goals were achieved. In CookingWorld, goals and subgoals are purely textual, reward evaluation is derived from trajectory descriptions, and the agent discovers thousands of goal descriptions without hand-coded goal representations or reward functions. The same paper emphasizes that this cultural grounding shifts autotelic exploration toward more human-relevant goals rather than merely novel states (Colas et al., 2023).

For online RL with LLM agents, SAC-GLAM adapts Soft Actor-Critic and hindsight relabeling to textual, environment-level actions. Observations and goals are text, actions are discrete textual commands, and the policy is induced from the LLM’s token probabilities over the enumerated action set. Replay stores textual transitions $g \in G$ 8, batches contain 50% HER transitions in the main experiments, and off-policy training is explicitly presented as a foundation for future autotelic LLMs because it enables reuse of failures, post hoc relabeling, and goal-conditioned learning over evolving language-defined tasks (Gaven et al., 2024).

MAGELLAN adds a metacognitive layer for large language-defined goal spaces. It trains an LLM-based competence predictor $g \in G$ 9 with binary cross-entropy over recent experience, stores parameter snapshots, and estimates absolute learning progress through

$P(g)$ 0

Goals are then scheduled by $P(g)$ 1-greedy sampling proportional to $P(g)$ 2. In Little-Zoo, this semantic generalization across goals allows sample-efficient LP estimation, adaptation to evolving goal spaces, and full mastery of a large goal space without expert-defined goal groupings (Gaven et al., 11 Feb 2025).

4. Structured representations, decomposition, and hierarchical skill acquisition

A major line of work studies how representational structure constrains autotelic skill acquisition. In multi-object manipulation, object-centered graph neural networks outperform flat policies because relational inductive biases make it easier to represent interactions among objects and to generalize to unseen relational goals. This family of methods contrasts geometric goal spaces with predicate-based semantic goal spaces and shows that expressive object-centered GNNs paired with relational predicates such as close and above improve transfer to held-out goals and harder configurations (Akakzia et al., 2022).

GANGSTR extends this principle with an explicit semantic graph over discovered goals. Nodes are semantic configurations; directed edges represent transitions produced by moving a single object; each edge stores a success-rate estimate updated by exponential moving average with $P(g)$ 3 and initialized at $P(g)$ 4. Planning alternates between shortest-path decomposition and safest-path decomposition, where the safest path maximizes the product of edge success rates. In a five-block manipulation domain with a semantic goal space of roughly 70,000 reachable configurations, graph-based decomposition lets the agent master configurations such as stacks of five blocks that it fails to learn without decomposition (Akakzia et al., 2022).

Hierarchical autotelic LLM agents push the same logic further. HERAKLES uses a two-level hierarchy in Crafter: a high-level LLM selects skills from a dynamically growing skill set, while low-level policies execute them from pixels. Once a goal is mastered, successful low-level trajectories are relabeled with that goal and compiled into a callable low-level skill. The admissible skill set is filtered online by a competence estimator,

$P(g)$ 5

so the high-level plans only over skills predicted to be executable. The reported effect is effective horizon reduction, improved sample efficiency, and robust generalization to synonyms, compositional repeats, and similar-trajectory goals (Carta et al., 20 Aug 2025).

This line of work suggests a broader methodological principle: autotelic learning scales when the goal space is not only large, but structurally organized. Semantic predicates, graph representations, competence-weighted edges, and compiled skill libraries all serve the same function—turning open-ended goal pursuit into a sequence of reusable, competence-tracked subproblems rather than a flat search over primitive actions (Akakzia et al., 2022, Akakzia et al., 2022, Carta et al., 20 Aug 2025).

Social guidance has become a second major axis of autotelic AI. Help Me Explore (HME) couples autotelic exploration with minimal social interventions. A social partner proposes a discovered frontier goal and then an adjacent unknown beyond goal from an oracle semantic graph; if the beyond goal is not reached, the pair is stored in an internalization buffer and rehearsed later during autotelic episodes with probability internalization_prob = 0.5. In the five-block manipulation domain, even 0.1% social episodes create significant learning differences, and the agent masters complex configurations with only a small fraction of socially guided episodes (Akakzia et al., 2022).

Textual autotelic agents in ScienceWorld show an analogous result from hindsight feedback. A selective social peer that labels only relevant achieved language goals supports learning, whereas no feedback yields near-zero performance and exhaustive feedback collapses learning by flooding replay with too many ultra-specific goals. The same study shows that modular replay must over-sample rare goals and that following goal sequences where competence is intermediate substantially improves final performance (Teodorescu et al., 2023).

At a broader theoretical level, Vygotskian autotelic AI argues that language and culture should be internalized as cognitive tools. In this view, linguistic interactions do not merely transmit instructions; they reshape representation, support category formation, guide hierarchical decomposition, and enable inner speech, trajectory captioning, goal imagination, and explanation generation. Related sociocultural settings include the Architect–Builder Problem and ABIG, where one agent knows goals and communicates messages while another acts, yielding emergent communication protocols and transfer to harder tasks in BuildWorld (Colas et al., 2022, Srivastava et al., 6 Feb 2025).

Multi-agent autotelic RL adds a coordination problem absent from single-agent settings. In the Decentralized Intrinsically Motivated Skills Acquisition Problem, agents that sample goals independently rarely align on the same cooperative goal, making cooperative skill discovery improbable. A sufficient condition for mastering cooperative goals is group alignment, $P(g)$ 6 for all agents in the group during aligned episodes. The Goal-coordination game implements this through decentralized emergent communication: a leader samples a goal and emits a message via a communication matrix, a follower maps the message to a goal via its own matrix, and both matrices are updated by exponential moving average of experienced rewards. In cooperative navigation, this decentralized mechanism matches centralized aligned-goal baselines and induces specialization (Nisioti et al., 2022).

A plausible implication is that sociality in autotelic AI is not reducible to demonstrations. The strongest results in this line come from lighter interventions: frontier goals instead of demonstrations in HME, selective hindsight annotations in ScienceWorld, message-based goal alignment in Dec-IMSAP, and language as an abstract scaffolding device rather than a full policy specification (Akakzia et al., 2022, Teodorescu et al., 2023, Nisioti et al., 2022).

6. Evaluation, applications, and theoretical extensions

Evaluation remains an open methodological issue because open-ended agents cannot be assessed only by return on a fixed benchmark. Proposed metrics span exploration, generalization, robustness, sample efficiency, and catastrophic forgetting. Typical measures include entropy or coverage over outcomes, success on held-out goals,

$P(g)$ 7

expected success under perturbations, success versus environment steps, and continual tracking of competence on previously mastered goals. Sociocultural settings add communicative success and protocol stability (Srivastava et al., 6 Feb 2025).

Applications are correspondingly diverse. In robotics and simulation, autotelic methods target manipulation, navigation, and tool use. In text environments, they address exploration under large combinatorial action spaces. In generative systems, ACES operationalizes autotelic generation by sampling target semantic descriptor vectors, conditioning an LLM to produce puzzles matching those goals, and growing an archive over discovered semantic cells; the detailed description emphasizes diversity and archive coverage in semantic and embedding spaces, while explicitly noting that a per-puzzle difficulty function is not part of the reported method (Pourcel et al., 2023). In human–AI relations, machine companionship work extends the term “autotelic” from agent design to relational experience: machine companionship is defined as “an autotelic, coordinated connection between a human and machine that unfolds over time and is subjectively positive,” and a two-factor scale—Eudaimonic Exchange and Connective Coordination—has been proposed to operationalize this distinction (Banks et al., 22 Jun 2025, Banks, 1 Nov 2025).

Theoretical extensions have pushed autotelic AI toward questions of selfhood and embedded agency. One line uses autocatalytic networks and RAF theory to describe self-organizing, self-sustaining structures whose reactions are internally catalyzed and food-generated, suggesting a route from intrinsic motivation to bounded, self-preserving creative agency (Gabora et al., 2024). Another formalizes the autotelic agent as a tuple $P(g)$ 8, where goals are endogenous, viability is defined relative to a boundary, and the self-model operationally treats that boundary as fundamental while acknowledging that multiple valid agent-environment partitions may exist. On this account, embeddedness is necessary but not sufficient for autotelic agency, because the deepest problem is not only how an agent generates goals, but how it generates and relativizes the self to which those goals are assigned (Sarkar, 18 Jun 2026).

Limitations are recurrent and concrete. Many successful systems assume oracle graphs, curated semantics, or accurate extraction of achieved goals; failures in adjacency specification, goal parsing, or competence estimation can misguide curricula. Sparse rewards, noisy language grounding, large replay buffers, and LLM-based actor–critic training introduce computational cost and stability concerns. Safety is also unresolved: recent LLM-autotelic work explicitly notes that goal-setting agents require safe exploration and goal vetting, while theoretical accounts of self-preserving autotelicity warn that boundary maintenance and endogenous goals can become misaligned with external objectives (Akakzia et al., 2022, Gaven et al., 2024, Srivastava et al., 6 Feb 2025, Sarkar, 18 Jun 2026).

Taken together, the literature presents Autotelic AI as a research program rather than a single algorithmic family. Its unifying commitment is that agents should not merely optimize exogenous tasks, but should learn to construct goal spaces, estimate competence, generate curricula, reuse experience through hindsight, exploit structure through language and graphs, benefit from social scaffolding, and, in the strongest formulations, ground goal generation in an embedded model of their own persistence and limits (Srivastava et al., 6 Feb 2025, Colas et al., 2020)