Hindsight Goal-Conditioned Regularization

• HGR is a method that constructs a goal-conditioned action prior from hindsight goals and regularizes the policy using a KL divergence term.
• It extends standard HER by influencing the optimization objective to enhance action coverage beyond exact achieved-goal pairs.
• Empirical results demonstrate that HGR improves sample efficiency and learning robustness in sparse-reward robotic tasks compared to baseline methods.

Hindsight Goal-conditioned Regularization (HGR) is a hindsight-based policy regularization mechanism for sparse-reward goal-conditioned reinforcement learning (GCRL). In the formulation introduced in "GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning," HGR is not the entire algorithm but one of two hindsight regularizers inside the broader GCHR framework, the other being Hindsight Self-imitation Regularization (HSR). Its defining operation is to construct a goal-conditioned action prior from hindsight goals visited along trajectories and to regularize the current policy toward that prior through a KL term, thereby going beyond ordinary Hindsight Experience Replay (HER), which only relabels trajectories with achieved goals (Lei et al., 8 Aug 2025).

1. Definition and conceptual scope

In the paper’s terminology, GCHR denotes the full off-policy framework, while HGR denotes the specific regularizer that uses hindsight goals to build an action prior. The paper’s central claim is that HER-style relabeling alone does not fully exploit the information contained in collected trajectories: relabeling turns failed rollouts into successful examples for relabeled goals, but still uses those trajectories mainly as supervised targets for the exact relabeled goals. HGR instead attempts to extract a broader goal-conditioned action prior from the same trajectories and to use that prior during policy optimization (Lei et al., 8 Aug 2025).

The setting is a standard goal-conditioned MDP

$$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$

with goal-conditioned policy

$\pi(a \mid s, g),$

and sparse binary goal-reaching rewards defined through a deterministic state-to-goal map $\phi:\mathcal{S}\to\mathcal{G}$: $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$ The paper adopts an absorbing-goal formulation: once the agent reaches any state in

$\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$

that state is absorbing and the reward remains $1$ at each subsequent step (Lei et al., 8 Aug 2025).

A useful conceptual distinction follows directly from this formulation. HER modifies the training data by changing which goal a transition is paired with. HGR, by contrast, modifies the optimization objective through an explicit policy regularizer derived from hindsight goals. This places HGR closer to objective-level regularization than to replay-only relabeling.

2. Problem setting and motivation in sparse-reward GCRL

The paper motivates HGR from the difficulty of sparse binary rewards in off-policy GCRL. Under the absorbing-goal formulation, the goal-conditioned action-value function can be written as discounted goal reachability: $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$ This makes explicit that value estimation is fundamentally a reachability problem over goals rather than a dense-reward regression problem (Lei et al., 8 Aug 2025).

HER provides the standard remedy. For a trajectory

$\tau = (s_0, a_0, \ldots, s_T),$

the set of future-achievable goals from time $t$ is

$$\mathcal{G}_t^{\text{future}}(\tau)=\{\phi(s_{t'}) : t \leq t' \leq T\},$$

and a relabeling rule $\pi(a \mid s, g),$0 chooses a hindsight goal $\pi(a \mid s, g),$1 from this set, yielding a HER dataset

$\pi(a \mid s, g),$2

The paper’s critique is not that HER is ineffective, but that it only uses actions as positives for goals that were directly achieved in the trajectory. This leads to what the authors describe as limited action coverage: the learner exploits exact relabeled state-goal-action triples, but not the broader structure induced by the sequence of visited goals in the trajectory (Lei et al., 8 Aug 2025).

This motivates HGR. The paper’s key intuition is compositional: actions that help from state $\pi(a \mid s, g),$3 toward some hindsight goal $\pi(a \mid s, g),$4 encountered along a trajectory may also be useful stepping stones toward a desired goal $\pi(a \mid s, g),$5. HGR is designed to capture that broader reuse.

3. Mechanism of Hindsight Goal-conditioned Regularization

HGR begins from the set of hindsight goals visited in a trajectory: $\pi(a \mid s, g),$6 where $\pi(a \mid s, g),$7. For a current state $\pi(a \mid s, g),$8 and desired goal $\pi(a \mid s, g),$9, HGR samples hindsight goals $\phi:\mathcal{S}\to\mathcal{G}$0 from a distribution $\phi:\mathcal{S}\to\mathcal{G}$1 over this set and forms a hindsight-goal prior

$\phi:\mathcal{S}\to\mathcal{G}$2

where $\phi:\mathcal{S}\to\mathcal{G}$3 is a delayed target policy (Lei et al., 8 Aug 2025).

In implementation, the prior is approximated by averaging over $\phi:\mathcal{S}\to\mathcal{G}$4 sampled hindsight goals: $\phi:\mathcal{S}\to\mathcal{G}$5 This prior is then used in a KL regularizer of the form

$\phi:\mathcal{S}\to\mathcal{G}$6

The paper presents the associated loss as

$\phi:\mathcal{S}\to\mathcal{G}$7

because the full optimization is written as a maximization objective (Lei et al., 8 Aug 2025).

The direction of the KL is central. Since it is

$\phi:\mathcal{S}\to\mathcal{G}$8

the current policy is encouraged to cover the support of the hindsight-goal prior rather than merely imitate a single historical action. This is the core difference from HSR, whose role is narrower: $\phi:\mathcal{S}\to\mathcal{G}$9 HSR treats HER-relabeled actions as self-demonstrations; HGR instead regularizes against a policy-generated mixture over multiple hindsight goals (Lei et al., 8 Aug 2025).

The paper formalizes this distinction through action support. It defines $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$0 as the actions that directly led to $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$1 in replay and $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$2 as the union of actions with positive probability under the delayed policy for hindsight goals from replay. Theorem 1 states

$r(s, g) = \mathbf{I}\{\phi(s) = g\}.$3

which is the paper’s formal statement that HGR expands action coverage beyond HSR (Lei et al., 8 Aug 2025).

4. Full GCHR objective, theoretical rationale, and implementation

The complete GCHR objective combines ordinary off-policy goal-conditioned RL with HSR and HGR: $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$4 Here $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$5 is the original replay buffer, $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$6 is the HER-relabeled replay buffer, $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$7 controls HSR, and $r(s, g) = \mathbf{I}\{\phi(s) = g\}.$8 controls HGR (Lei et al., 8 Aug 2025).

The paper’s theoretical rationale for HGR is based on a via-goal decomposition. It introduces a via-goal value

$r(s, g) = \mathbf{I}\{\phi(s) = g\}.$9

which measures a compositional strategy: first reach an intermediate goal $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$0, then continue to the final goal $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$1. Under the paper’s Uniform Reachability assumption, if the base policy improves monotonically, then the expected via-goal value under the delayed policy improves as well. The intended implication is that the HGR prior is adaptive rather than static: as the policy improves, the prior induced by visited hindsight goals becomes better aligned with useful behavior for desired goals (Lei et al., 8 Aug 2025).

Algorithmically, GCHR uses a standard off-policy actor-critic backbone, with SAC chosen in the experiments because of its off-policy nature and stochastic policy representation. Replay stores both original and HER-relabeled trajectories. Critic updates use ordinary target-network TD learning, while actor updates optimize the RL objective minus weighted HSR and HGR penalties. The paper characterizes the implementation as simple, stating that GCHR can be added in “five lines of code” because it requires neither a learned dynamics model nor extra planning machinery (Lei et al., 8 Aug 2025).

A concise summary of the components is as follows.

Component	Mechanism	Intervention locus
HER	Relabel transitions with future achieved goals	Replay/data distribution
HSR	Maximize log-likelihood of hindsight-successful actions	Actor regularization
HGR	KL regularization toward a hindsight-goal action prior	Actor regularization
GCHR	RL objective plus HSR and HGR	Full off-policy training

The implementation details reported for GCHR include $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$2 by default for HGR, $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$3 by default for HSR, discount $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$4, batch size $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$5, replay buffer size $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$6, learning rate $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$7, Polyak coefficient $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$8, warmup steps $\mathcal{S}_g = \{s \in \mathcal{S}: \phi(s)=g\},$9, random action probability $1$0, additive Gaussian noise scale $1$1, and HER replay probability $1$2. For each state, relabeled goals are uniformly sampled from future states within the trajectory (Lei et al., 8 Aug 2025).

5. Empirical behavior and reported performance

The reported experiments use sparse-reward goal-conditioned robot tasks from the OpenAI multi-goal benchmarks: FetchReach, FetchPick, FetchPush, FetchSlide, HandReach, BlockRotateZ, BlockRotateXYZ, and BlockRotateParallel. The paper also includes a qualitative exploration study in L-Antmaze 2D (Lei et al., 8 Aug 2025).

The stated headline result is that GCHR performs better than all baseline methods in all environments, coupled with a faster learning speed. The main baselines are DDPG, DDPG+HER, MHER, GCSL, WGCSL, GoFar, and DWSL. The paper also reports an actor-critic comparison against DDPG+HER and SAC+HER. Selected final mean success rates reported in the paper are:

Environment	GCHR	Selected baselines
FetchReach	$1$3	—
FetchPick	$1$4	DDPG+HER $1$5, SAC+HER $1$6
FetchPush	$1$7	DDPG+HER $1$8, SAC+HER $1$9
HandReach	$$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$0	DDPG+HER $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$1, SAC+HER $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$2

The appendix gives a direct sample-efficiency example on FetchPush: to reach mean success rate $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$3, DDPG+HER requires over 6000 training samples while GCHR requires about 4000, summarized in the paper as about 1.5× better sample efficiency (Lei et al., 8 Aug 2025).

The paper also reports ablations isolating HGR from HSR. The key conclusion is explicit: HGR is more important than HSR in the GCHR framework. Additional ablations vary $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$4 over $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$5, $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$6 for HSR, the number of hindsight goals used in the prior, and the relabel ratio over $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$7. The reported interpretation is that GCHR is robust across these settings, that broader use of relabeled goals tends to help, and that the HGR prior behaves as an implicit curriculum: early in training it reflects nearby achieved goals, and later it shifts as the delayed policy improves (Lei et al., 8 Aug 2025).

The paper also reports that GCHR is the most robust method under Gaussian action noise in FetchPush. This suggests, in the paper’s interpretation, that a policy-generated hindsight prior can be more tolerant to stochasticity than pure self-imitation based on deterministic replayed actions (Lei et al., 8 Aug 2025).

6. Position within the hindsight RL literature

HGR belongs to a broader family of hindsight methods, but its intervention point is specific. It should not be conflated with methods that only change the hindsight-goal distribution. For example, Hindsight Goal Generation (HGG) selects intermediate goals by balancing current achievability against distance to the target-goal distribution through a Wasserstein-inspired assignment problem, thereby shaping exploration-time goal selection rather than adding an explicit regularization term to the learner (Ren et al., 2019). By contrast, HGR keeps hindsight goals as inputs to an explicit policy regularizer.

Earlier work on Hindsight Policy Gradients showed how trajectories collected under one goal can be reused for alternative goals through importance-corrected policy-gradient estimators, establishing a principled cross-goal reuse mechanism but not an explicit regularizer of the HGR type (Rauber et al., 2017). Hindsight Expectation Maximization provided a different foundation, interpreting hindsight relabeling as an approximate E-step over latent goals and reducing the policy update to supervised fitting on relabeled trajectories; this offers a variational view of hindsight reuse but again does not define an HGR-style action-prior regularizer (Tang et al., 2020).

Several related methods intervene on other aspects of hindsight learning. Universal Value Density Estimation replaces hindsight-relabeled Bellman targets with a learned density $$Q^\pi(s, a, g) = \mathbb{E}\left[\sum_{t=0}^{\infty} \gamma^t r(s_t, g) \,\Big|\, s_0 = s, a_0 = a, \pi, g\right] = \sum_{\Delta=1}^{\infty} \gamma^\Delta \Pr_\pi(\phi(s_\Delta) = g \mid s_0=s,a_0=a).$$8 over future achieved goals, explicitly motivated by avoiding hindsight bias in stochastic domains (Schroecker et al., 2020). Maximum Entropy-Regularized Multi-Goal Reinforcement Learning regularizes the achieved-goal distribution in replay toward higher entropy through trajectory-level prioritization, which is a replay-distribution regularizer rather than an actor prior over actions (Zhao et al., 2019). Efficient Sparse-Reward Goal-Conditioned Reinforcement Learning with a High Replay Ratio and Regularization combines HER with REDQ and bounded target Q-values, showing that hindsight relabeling can destabilize value learning and that target-level regularization can materially improve sample efficiency (Hiraoka, 2023).

More recent work also generalizes the semantics of hindsight. Goal-Set Hindsight Relabeling (GS-HER) replaces singleton full-state goals with query-defined goal sets, so that a future achieved state certifies a whole set of equivalent successful goals rather than an exact goal state (García et al., 8 Jun 2026). Hindsight Relabeling using Interactions (HInt) filters hindsight relabels using inferred null-counterfactual interactions, restricting replay to trajectories in which the target outcome lies on an inferred action-to-target interaction chain (Chuck et al., 6 May 2025). In relational planning settings, First-Order Representation Languages for Goal-Conditioned RL shows that propositional and lifted hindsight goals can create a curriculum of easier goals and substantially outperform full-state HER, which is a representational change in the goal language rather than an optimization regularizer (Ståhlberg et al., 22 Dec 2025).

Taken together, these comparisons locate HGR precisely. HER relabels achieved goals; HGG optimizes which hindsight goals are generated; GS-HER changes what counts as goal satisfaction; HInt restricts which relabels are admitted; UVD models future-goal density; REDQ+HER+BQ regularizes value targets; hEM casts hindsight reuse as EM. HGR’s distinctive contribution is narrower and more explicit: it turns hindsight goals into a goal-conditioned action prior and regularizes the policy toward that prior through KL divergence (Lei et al., 8 Aug 2025).