Self-Generated Goal-Conditioned MDPs

• Self-Generated Goal-Conditioned MDPs are a reinforcement learning framework where agents dynamically generate, select, and pursue their own goals to overcome preset goal limitations.
• They employ self-supervised distance learning and automatic curriculum generation to enhance sample efficiency and adaptability in diverse environments.
• The approach addresses challenges like reward sparsity, compositional reasoning, and domain generalization, paving the way for robust autonomous learning in complex tasks.

Self-Generated Goal-Conditioned Markov Decision Processes (sG-MDPs) are a formalism and set of algorithmic and theoretical advances in reinforcement learning and planning where agents autonomously propose, select, and pursue their own goals. In contrast with traditional goal-conditioned MDPs—which typically assume a pre-specified set of goals—sG-MDPs facilitate adaptive, open-ended, and scalable learning by enabling agents to both construct and reason about novel goals at runtime. This approach directly addresses core challenges related to sample efficiency, curriculum learning, reward sparsity, and generalization across domains and tasks.

1. Formal Definition and Conceptual Framework

sG-MDPs extend the standard goal-conditioned MDP, typically denoted as $(\mathcal{S}, \mathcal{A}, T, R, \mathcal{G})$, by making the set of candidate goals $\mathcal{G}$ dynamically generated and agent-dependent.

• Standard GCRL: The policy $\pi(a|s, g)$, reward $r(s,g,a)$, and value $V_\pi(s,g)$ are all conditioned on an externally-supplied goal $g \in \mathcal{G}$. The agent faces a pre-defined or sampled set of goals, and is tasked with efficiently learning to achieve any $g$.
• sG-MDPs: The agent’s action or meta-action space is augmented such that it can propose new goals $\mathcal{A}^+$, select which goals to pursue (possibly with intrinsic motivation or curriculum-based strategies), and solve goals in ways tailored to the current state and learning context (2507.02726).

The transition function and overall system dynamics are correspondingly extended. For example, the transition may be:

$$T_\mathcal{G} : \mathcal{S} \times \mathcal{A} \times \Delta(\mathcal{G}) \to \mathcal{S} \times \Delta(\mathcal{G})$$

supporting dynamic goal stack manipulation.

A key insight is that the agent’s objective is no longer simply to optimize for a static set of goals but to maximize expected return across an evolving set of self-generated goals—potentially with multi-objective or lexicographic criteria, as explored in goal-oriented MDPs with dead ends (1210.4875).

2. Algorithms and Learning Methodologies

sG-MDP research reveals a spectrum of strategies for generation, selection, and efficient solution of self-sampled goals:

• Self-supervised Distance Learning: Agents learn task-aligned distance metrics (such as action distance or commute time) directly from trajectories to define when a goal is achieved, enable automatic curriculum generation, and support reward shaping (1907.02998). Embeddings are learned to approximate mean first passage times, with key connections to metric MDS and spectral graph theory.
• Automatic Curriculum and Goal Proposal: Several approaches employ distance predictors or uncertainty-driven sampling to adaptively focus on goals at the "boundary" of the reachable region, facilitating rapid expansion of an agent’s skill set (e.g., via Dynamical Distance Functions (2111.04120), AdaGoal (2111.12045)).
• Complex Goal Structures: Recent work explores policies conditioned not only on the current goal but on sequences or sets of upcoming goals, avoiding the "chaining problem" where subgoals may be achieved in counterproductive ways (2503.21677). Conditioning on both current and subsequent goals has been shown to yield significant gains in stability and sample efficiency.
• Imitation and Demonstration-Based sG-MDPs: Self-adaptive imitation learning algorithms such as Goal-SAGAIL introduce self-improvement by augmenting expert demonstration data with agent-generated trajectories for harder or underrepresented goals, using explicit goal-difficulty matching (2506.12676).
• Domain Invariance for Robustness: sG-MDP policies often require robustness to spurious environmental change; jointly learning domain-invariant representations and enforcing "perfect alignment" across environments significantly enhance transfer and sim2real capabilities (2110.14248).

3. Theoretical Advances and Sample Efficiency

A central challenge in sG-MDPs is ensuring efficient exploration and optimal learning when the achievable goal set is unknown or evolving:

• Extensions to SSP MDPs: Classical Stochastic Shortest Path (SSP) MDPs require all states to be able to reach the goal, forbidding dead ends. Recent theory addresses goal-oriented MDPs with dead ends (avoidable [SSPADE], unavoidable finite-penalty [fSSPUDE], and unavoidable infinite-penalty [iSSPUDE]), providing efficient value iteration and heuristic search algorithms and clarifying risk/reward trade-offs (1210.4875).
• Principled Multi-Goal Exploration: Adaptive algorithms such as AdaGoal provably optimize sample complexity for learning $\epsilon$-optimal goal-conditioned policies over the set of all reliably reachable goals, with nearly minimax optimality in both tabular and function-approximate settings (2111.12045).
• Learning with Demonstration and Curriculum: When relying on demonstration data (possibly suboptimal or narrow), self-adaptive augmentation and explicit goal-pair difficulty metrics improve both coverage and convergence speed, especially in high-dimensional spaces (2506.12676).

4. Practical Implementations and Application Domains

sG-MDPs have been deployed successfully in diverse domains:

• Robotic Manipulation: Multi-goal setups for robotic arms and dexterous hands leverage sG-MDP algorithms for curriculum learning, efficient sample use, and robust skill acquisition across goal distributions, even when demonstration data are limited or cover only easy goal instances (2506.12676).
• Continuous Control and Navigation: Agents learning in Mujoco, OpenAI Gym, or Maze environments use self-generated goals and dynamic curricula—frequently leveraging distance- or uncertainty-based goal proposal and hindsight relabeling (1907.02998, 2111.04120).
• Automated Theorem Proving: The sG-MDP framework has been applied to formalized mathematics, with agents (e.g., Bourbaki (2507.02726)) dynamically generating and stacking conjectures as new goals in proof search. Coupling sG-MDPs with MCTS-like planning and LLMs, these agents have substantially advanced the SOTA on challenging proof benchmarks like PutnamBench.

5. Robustness, Generalization, and Open Challenges

Key outstanding issues and current lines of research include:

• Domain Generalization: Ensuring policies learned via sG-MDPs transfer to unseen or perturbed environments requires learning domain-invariant latent representations. Methods such as PA-SkewFit combine aligned sampling and explicit alignment penalties (MMD, DIFF) to yield robust, transferable policies (2110.14248).
• Goal Space Design and Representation: Methods for learning, embedding, and prioritizing goals—particularly in high-dimensional, structured, or compositional spaces—remain a central challenge. Surveyed typologies include discrete, spatial, predicate-based, and compositional goals, with increasing interest in methods for unsupervised or language-conditioned goal synthesis (2012.09830).
• Reward Sparsity, Skill Composition, and Scalability: Approaches to mitigate reward sparsity (e.g., self-supervised subgoal generation, intermediate reward shaping) and methods for skill composition, transfer, and abstraction are active research frontiers. The design of value critics and policy hierarchies under sG-MDP regimes is especially pertinent for long-horizon tasks and complex, real-world problem settings (2507.02726).

6. Comparative Summary of sG-MDP Approaches

Paper / Method	Core Mechanism	Primary Application
(1210.4875)	MDPs with dead-end handling	Risk-aware planning and goal feasibility
(1907.02998)	Self-supervised action distance	Distance, curriculum, visual RL
(2111.12045)	AdaGoal, uncertainty-based success	Efficient multi-goal RL, theory+practice
(2503.21677)	Multi-goal sequential conditioning	Hierarchical RL, stability
(2506.12676)	Self-adaptive GAIL + HER	Multi-goal imitation learning, robotics
(2507.02726)	LLM-driven subgoal MCTS	Theorem proving, formal reasoning

7. Impact and Future Directions

The introduction and development of sG-MDPs have yielded marked advances in efficient, robust, and genuinely autonomous reinforcement learning:

• By supporting agents that autonomously define, select, and pursue increasingly challenging and diverse goals, sG-MDPs have led to state-of-the-art results in robotics, deep RL, and automated reasoning.
• The integration of subgoal-aware reward shaping, uncertainty-driven exploration, and domain-invariant learning under the sG-MDP paradigm suggests scalable paths toward more general, transferable, and creative AI.
• Future work will likely focus on richer goal representations, more powerful planning and abstraction mechanisms, and wider applicability to open-ended, dynamic, and compositional reasoning tasks.

A plausible implication is that sG-MDPs, by combining self-generated structure with principled learning objectives, are positioned to be a foundational framework for lifelong learning and agent autonomy across broad scientific and engineering domains.