Goal-Conditioned Supervised Learning (GCSL)

• Goal-Conditioned Supervised Learning (GCSL) is a framework that reformulates sparse goal-reaching as conditional imitation learning using hindsight relabeling of trajectories.
• The method alternates between data collection and iterated supervised policy updates, eliminating the need for explicit value-function estimation or critics.
• Extensions like WGCSL and GCSL-NF address challenges in exploration, long-horizon bias, and failure feedback, broadening its application in sparse-reward settings.

Goal-Conditioned Supervised Learning (GCSL) is a goal-conditioned reinforcement-learning paradigm that replaces explicit value-function optimization with supervised learning on hindsight-relabeled trajectories. Its defining observation is that any trajectory is a successful demonstration for the goal state it actually reaches, so an agent can iteratively collect trajectories, relabel future visited states as goals, and maximize the likelihood of the actions that led toward those achieved goals. In this way, sparse goal-reaching is recast as conditional imitation learning over self-generated data, yielding a simple and critic-free alternative to standard actor-critic or Q-learning methods for many goal-reaching tasks (Ghosh et al., 2019).

1. Formal definition and canonical objective

The original GCSL formulation studies a goal-conditioned control problem with state space $\mathcal{S}$, action space $\mathcal{A}$, transition function $p(s' \mid s,a)$, initial-state distribution $\rho(s_0)$, horizon $T$, and goal distribution $p(g)$. The policy is goal-conditioned, typically written as $\pi(\cdot \mid s,g,h)$, where $h$ denotes remaining horizon. The basic terminal goal-reaching objective is

$$J(\pi) = \mathbb{E}_{g \sim p(g)} \left[P_{\pi_g}(s_T = g)\right].$$

Equivalently, with $r_g(s)=\mathbbm{1}(s=g)$,

$\mathcal{A}$0

This is a last-timestep goal-attainment objective rather than a shortest-path objective (Ghosh et al., 2019).

A complementary formulation models the environment as a goal-conditioned MDP $\mathcal{A}$1 with absorbing goals and a policy $\mathcal{A}$2 trained to maximize goal-reaching success. In that setting, hindsight relabeling samples a current index $\mathcal{A}$3, a later index $\mathcal{A}$4, and treats $\mathcal{A}$5 as the goal for the state-action pair $\mathcal{A}$6. The resulting supervised objective is

$\mathcal{A}$7

which makes GCSL a conditional maximum-likelihood estimator over hindsight-labeled tuples rather than a Bellman-based RL algorithm (Jurgenson et al., 2023).

2. Hindsight relabeling and iterated supervised learning

The canonical GCSL algorithm alternates between data collection and supervised policy fitting. At iteration $\mathcal{A}$8, the current goal-conditioned policy collects trajectories; each trajectory is then relabeled with goals it actually achieved; all relabeled tuples are added to a replay-style dataset; and the next policy is obtained by maximizing log-likelihood on that dataset: $\mathcal{A}$9 For a single trajectory, the most general relabeling rule uses all future states: $p(s' \mid s,a)$0 Thus each action can be reinterpreted as supervision for reaching many future states, not only the originally intended goal (Ghosh et al., 2019).

This procedure is “iterated supervised learning” because it bootstraps its own supervision. There are no expert demonstrations and no critic. Old trajectories remain useful under hindsight relabeling, so the method is naturally off-policy in the sense that previously collected trajectories still provide valid training examples for achieved goals. In practice, implementations often sample relabeled tuples on the fly from stored whole trajectories rather than materializing all $p(s' \mid s,a)$1 pairs; they also often omit explicit horizon input even though the formal policy class includes $p(s' \mid s,a)$2 (Ghosh et al., 2019).

A probabilistic interpretation recasts hindsight relabeling as an approximate E-step and the supervised update as an M-step. In Hindsight Expectation Maximization, the optimality event $p(s' \mid s,a)$3 is introduced as a latent-variable model over goals and trajectories, and the variational lower bound becomes

$p(s' \mid s,a)$4

Holding $p(s' \mid s,a)$5 fixed, the M-step reduces to

$p(s' \mid s,a)$6

which is exactly a hindsight-relabeled supervised policy update (Tang et al., 2020).

3. Theoretical interpretations and guarantees

The original GCSL analysis shows that the supervised hindsight objective is not merely heuristic. For the simplified final-state-relabeled objective

$p(s' \mid s,a)$7

the paper proves

$p(s' \mid s,a)$8

where

$p(s' \mid s,a)$9

Under deterministic dynamics and full-support data collection, closeness to the GCSL optimum implies near-optimal goal-reaching performance (Ghosh et al., 2019).

Weighted variants sharpen this interpretation. Weighted GCSL (WGCSL) defines

$\rho(s_0)$0

with compound weight

$\rho(s_0)$1

The analysis introduces a surrogate $\rho(s_0)$2 and proves

$\rho(s_0)$3

so discounted relabeling and advantage weighting tighten the lower-bound relation to the underlying goal-conditioned RL objective (Yang et al., 2022).

A more recent theoretical development studies convergence and stability of GCSL in finite episodic environments through a command-extension formalism. There, goal-reaching values are written as probabilities,

$\rho(s_0)$4

$\rho(s_0)$5

and the exact infinite-sample update is

$\rho(s_0)$6

The main conclusion is that near-optimal behavior is achieved if the transition kernel lies in a sufficiently small neighborhood of a deterministic kernel, and that the relevant policies, values, and goal-reaching objective are continuous at deterministic kernels in an appropriate relative sense (Štrupl et al., 8 Feb 2025).

4. Structural limitations

GCSL’s simplicity does not remove exploration difficulty. The original paper identifies exploration as a principal limitation: if useful states are never visited, hindsight relabeling cannot create supervision for them. The method can reuse off-policy data effectively, but it still depends on the policy’s ability to generate trajectories whose future states are meaningful training goals (Ghosh et al., 2019).

A second limitation is an intrinsic bias toward short-horizon goals. When the objective is written over suffixes, the probability of updating on a goal exactly $\rho(s_0)$7 steps away satisfies

$\rho(s_0)$8

hence $\rho(s_0)$9. This monotone bias means closer goals are updated more frequently regardless of the trajectory distribution, which helps explain why vanilla GCSL often learns short-horizon behavior more reliably than long-horizon goal-reaching (Jurgenson et al., 2023).

A third limitation is the lack of trajectory stitching. In supervised goal-conditioned learning, policy updates are grounded in relabeled state-action-goal triples extracted from existing trajectories. This makes it difficult to infer optimal behavior for unseen compositional state-goal pairs that would require combining prefixes from one trajectory with suffixes from another. That limitation is emphasized in goal-conditioned weighted supervised learning and in later attempts to recover stitching either through model-based augmentation or through explicit dynamic-programming components (Lei et al., 2024).

A fourth limitation concerns failure information. Vanilla GCSL relabels future achieved states as alternative goals, thereby turning many failed attempts into positive supervision for some hindsight goal. This densifies the signal, but it also discards the fact that the original commanded goal may have been missed. The negative-feedback formulation GCSL-NF is motivated precisely by the claim that vanilla GCSL cannot learn from failures in the ordinary sense and can amplify its own behavioral biases when trained only on self-generated hindsight positives (Zhang et al., 3 Sep 2025).

5. Major extensions and variants

The GCSL literature has diversified into weighted, hierarchical, safety-aware, failure-aware, and multi-objective variants. Several influential directions are summarized below.

Variant	Core mechanism	Representative work
Weighted GCSL	Discounted relabeling, exponential advantage weighting, best-advantage filtering	WGCSL (Yang et al., 2022)
Subgoal prediction	Jointly learn $T$0 and a trajectory encoder $T$1	TraIL (Jurgenson et al., 2023)
Negative feedback	Combine hindsight positives with original-goal soft targets from learned similarity	GCSL-NF (Zhang et al., 3 Sep 2025)
Goal augmentation	Sample augmented goals with a learned dynamics model to improve stitching	MGDA (Lei et al., 2024)
Supervision + DP	Add Q-learning or $T$2-conditioned maximization to supervised goal learning	Q-WSL (Lei et al., 2024), GCReinSL (Lei et al., 1 Jun 2025)
Safety-critical extension	Split control into a nominal goal policy and a recovery policy with a cost critic	RbSL (Cao et al., 2024)
Multi-objective extension	Replace scalar goals with multi-dimensional goal vectors for recommendation	MOGCSL (Li et al., 2024)

WGCSL is the canonical extension when the issue is not data scarcity but data quality. Its compound weight

$T$3

replaces uniform imitation with weighted supervised learning, and the empirical motivation is that offline datasets mix random, suboptimal, and expert trajectories whose relabeled tuples should not contribute equally (Yang et al., 2022).

TraIL extends GCSL by learning a goal-conditioned subgoal predictor

$T$4

trained from the same replay buffer via

$T$5

At execution time the policy acts on a predicted subgoal $T$6 rather than the final goal $T$7, which is intended to shorten the effective control horizon (Jurgenson et al., 2023).

GCSL-NF changes the supervision itself. It keeps hindsight-positive tuples

$T$8

but also preserves original-goal tuples

$T$9

weighted by a learned similarity $p(g)$0. The combined objective

$p(g)$1

is explicitly designed to restore negative evidence about original goals while retaining hindsight positives (Zhang et al., 3 Sep 2025).

Two recent lines attempt to recover stitching without abandoning supervised learning. Q-WSL combines weighted supervised learning with a goal-conditioned critic: $p(g)$2 while GCReinSL replaces raw goal conditioning with a $p(g)$3-conditioned policy $p(g)$4 and uses expectile regression to approximate the maximum in-support goal-reaching value. Both are motivated by the claim that pure supervised goal-conditioned learning lacks TD-style stitching capability (Lei et al., 2024, Lei et al., 1 Jun 2025).

6. Applications, adjacent methods, and open questions

GCSL originated in sparse-reward goal-reaching control and was evaluated on tasks such as 2D navigation, robotic pushing, Lunar Lander, door opening, and claw manipulation, often showing stronger robustness than TD3-HER or PPO on challenging sparse-goal domains (Ghosh et al., 2019). Subsequent work broadened the setting to fully offline goal-conditioned RL, vision-based maze tasks, safety-critical robot manipulation, and even recommendation systems, where a multi-objective variant is described as extending goals from one-dimensional scalars to multi-dimensional vectors for next-action prediction in commercial-grade recommenders (Li et al., 2024).

A parallel line of work focuses less on the supervised policy update and more on the geometry of goals. One approach learns a state representation $p(g)$5 whose latent distance approximates minimum action distance, using only offline unlabeled trajectories and temporal separations: $p(g)$6 This is not GCSL, but it directly addresses questions that GCSL leaves implicit: how far goals are, which relabeled goals are meaningful, and how to estimate horizon or reachability from offline data (Steccanella et al., 2022). A related earlier line learns self-supervised distance functions from trajectories to define success and curricula when Euclidean distance in goal space is inadequate (Venkattaramanujam et al., 2019).

Other neighboring methods propose alternatives to the GCSL objective itself. Offline goal-conditioned reward shaping first learns a self-supervised representation or predictive structure from trajectories and then trains a goal-conditioned policy with offline RL using a learned dense reward rather than pure action imitation (Mezghani et al., 2023). SMORe goes further by treating offline goal-conditioned RL as mixture-distribution matching instead of supervised cloning, explicitly positioning itself against GCSL-style methods when the offline data are suboptimal (Sikchi et al., 2023).

A final open direction concerns the policy-conditioning interface. Recent online work argues that the optimal goal-conditioned action depends on the goal only through a directional signal in a learned goal-reaching geometry, and proposes conditioning the actor on a unit direction and magnitude rather than the raw goal. This is not a GCSL method, but it suggests that part of GCSL’s long-horizon difficulty may lie not only in the supervised objective but also in the choice of conditioning variable (K et al., 15 Jun 2026).

Taken together, the literature presents GCSL as more than a behavioral-cloning baseline. It is a self-supervised control paradigm with a clear hindsight mechanism, a lower-bound interpretation, and strong empirical appeal in sparse-reward settings, but also with persistent questions about exploration, long-horizon bias, failure information, representation, and compositional generalization. Subsequent work has largely preserved the central hindsight-supervised insight while altering one of four axes: sample weighting, goal representation, auxiliary prediction, or integration with value-based reasoning.