Hindsight Self-Imitation Regularization (HSR)

• The paper introduces HSR as a policy regularizer that converts hindsight transitions into an explicit behavior cloning loss on the actor.
• HSR extends HER by providing a direct actor-side signal, thereby enhancing sample efficiency in sparse-reward, goal-conditioned reinforcement learning.
• Empirical evaluations in GCHR show that while HSR improves over plain HER, its best performance emerges when combined with Hindsight Goal Regularization (HGR).

Searching arXiv for the named paper and closely related hindsight/self-imitation work to ground the article in current arXiv records. arXiv search query: ti:"Goal-Conditioned Hindsight Regularization" OR all:"Hindsight Self-imitation" Hindsight Self-imitation Regularization (HSR) is a policy regularizer for sparse-reward goal-conditioned reinforcement learning that was introduced as one component of Goal-Conditioned Hindsight Regularization (GCHR). In the dominant naming used in the paper, HSR converts hindsight-relabeled transitions produced by hindsight experience replay (HER) into an explicit behavior-cloning term on the actor: for a state $s_t$ and relabeled hindsight goal $g'_t$, the observed action $a_t$ is treated as a successful action to reproduce. HSR therefore extends HER from reward relabeling for value learning to direct policy-side supervision, while remaining part of a broader framework in which it is paired with Hindsight Goal Regularization (HGR) (Lei et al., 8 Aug 2025).

1. Definition and nomenclature

HSR is defined in the GCHR framework as Hindsight Self-imitation Regularization, although the paper’s terminology is slightly inconsistent. The overall method is called GCHR (“Goal-Conditioned Hindsight Regularization”), and it contains two policy regularizers: HSR and HGR. In the introduction and figure caption, HSR is called Hindsight Self-imitation Regularization and HGR is called Hindsight Goal Regularization; in the method section, the subsection title says “Hindsight Action Regularization (HSR),” but the text and the rest of the paper clearly use HSR to mean the hindsight self-imitation / behavior-cloning term. The dominant naming is therefore HSR $=$ Hindsight Self-imitation Regularization (Lei et al., 8 Aug 2025).

At a high level, HSR is the paper’s mechanism for turning hindsight-relabeled transitions into an explicit policy regularizer rather than using them only for value-function learning. Standard HER increases the density of reward-bearing training targets by relabeling a trajectory with goals it actually achieved. HSR adds the missing actor-side signal: it tells the policy that if a trajectory later achieved a relabeled goal, then the earlier action on that trajectory is a good action to imitate under that hindsight goal. The paper’s claim is that this improves sample efficiency in sparse-reward goal-conditioned settings because the actor can directly imitate actions that empirically succeeded under hindsight goals instead of relying only on bootstrapping through relabeled rewards.

2. Formal setting and objective

The underlying problem is a goal-conditioned MDP

$$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$

with sparse binary reward

$r(s, g) = \mathbf{I}\{\phi(s) = g\},$

where $\phi:\mathcal{S}\to\mathcal{G}$ maps states to goal features, $\mathcal{S}$ is the state space, $\mathcal{A}$ the action space, $\mathcal{G}$ the goal space, $g'_t$0 the transition kernel, and $g'_t$1 the discount factor. A goal-conditioned policy is $g'_t$2. The paper adopts an absorbing-goal formulation and writes

$g'_t$3

This interpretation matters because goal-reaching is expressed as future occupancy or future-goal probability, so hindsight relabeling exposes more positive reachability information than the original sparse task labels alone (Lei et al., 8 Aug 2025).

HER is defined over a trajectory

$g'_t$4

For timestep $g'_t$5, the set of future achieved goals is

$g'_t$6

and a relabeling strategy $g'_t$7 picks a hindsight goal $g'_t$8 from that set. The HER dataset is

$g'_t$9

HSR is built directly on this hindsight-relabeled data. If a future state $a_t$0 induces hindsight goal $a_t$1, then the transition $a_t$2 is treated as evidence that $a_t$3 is a good action for state $a_t$4 under goal $a_t$5. The paper explicitly describes this as a “self-imitation prior” given by a Dirac distribution $a_t$6. The intended HSR loss is the behavioral-cloning negative log-likelihood

$a_t$7

The appendix algorithm implements the per-sample form

$a_t$8

In direct terms, HSR is an unweighted HER-conditioned behavior-cloning loss on relabeled transitions. Its symbols are the replayed state $a_t$9, the replayed action $=$0, a hindsight goal $=$1 or $=$2 chosen from future achieved goals on the same trajectory, and the current goal-conditioned actor $=$3. The paper does not define any threshold, advantage weight, return score, or success weighting inside the HSR term.

3. Role within GCHR and the training pipeline

In the full GCHR objective, HSR appears as one of two policy regularizers on top of the base off-policy RL objective: $=$4 Here $=$5 is the replay buffer of original transitions, $=$6 is the hindsight-relabeled replay buffer, $=$7 is the HSR weight, and $=$8 is the HGR weight. The appendix pseudocode makes the actor update explicit: $=$9 HSR therefore modifies the actor objective as an additive imitation regularizer; the critic update remains standard TD learning (Lei et al., 8 Aug 2025).

The critic target is a target-network TD target of the form

$$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$0

and the critic minimizes

$$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$1

This is a structural distinction: HSR does not alter the Bellman target, reward function, or TD recursion. It adds supervised pressure on the actor toward replayed hindsight-successful actions.

Algorithmically, the pipeline is local and transition-level. A trajectory $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$2 is collected under the current goal-conditioned policy for some desired goal $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$3 and stored in replay buffer $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$4. A relabeled buffer $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$5 is then created using HER: for each state-action pair $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$6, one or more future states $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$7, $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$8, are chosen, the hindsight goal is set to $$(\mathcal{S}, \mathcal{A}, \mathcal{G}, P, r, \gamma),$$9, and $r(s, g) = \mathbf{I}\{\phi(s) = g\},$0 is added to $r(s, g) = \mathbf{I}\{\phi(s) = g\},$1. The implementation details state that “for each state, relabeled goals are uniformly sampled from all future states within its trajectory.” During training, a relabeled transition $r(s, g) = \mathbf{I}\{\phi(s) = g\},$2 is sampled, the loss

$r(s, g) = \mathbf{I}\{\phi(s) = g\},$3

is computed, and this term is added to the actor loss with weight $r(s, g) = \mathbf{I}\{\phi(s) = g\},$4. The signal is therefore transition-level, not a trajectory-level return-weighted objective.

A central consequence is that HSR is plain HER-based behavior cloning. The paper explicitly notes that it does not use advantage weights, return thresholds, or selective filtering. The success criterion is simply that, under hindsight relabeling, the action belonged to a trajectory that eventually achieved the relabeled goal.

4. Relationship to HER, self-imitation learning, and adjacent hindsight methods

HSR is best understood by contrast with neighboring methods. Relative to standard HER, it is a policy-side extension. Relative to classic self-imitation learning, it is narrower and simpler: it is self-imitation specifically in hindsight-relabeled goal-conditioned space, without explicit return-minus-value gating. Relative to later or parallel hindsight-augmented imitation methods, it occupies the specific design point of off-policy, transition-level, unweighted behavior cloning on HER-relabeled replay.

Method	Hindsight unit	Policy signal
HER	Relabeled transition	Mainly critic / TD targets
HSR	Relabeled transition	Unweighted $r(s, g) = \mathbf{I}\{\phi(s) = g\},$5
ESIL	Relabeled whole episode	Log-likelihood gated by $r(s, g) = \mathbf{I}\{\phi(s) = g\},$6
VHS	Relabeled failed trajectory in visual POMDP	Auxiliary self-imitation loss
HCA	Future-state or return-conditioned outcome	Hindsight-weighted credit assignment

The contrast with Self-Imitation Learning is especially direct. SIL stores triples $r(s, g) = \mathbf{I}\{\phi(s) = g\},$7 and imitates only when the realized return exceeds the current value estimate, using

$r(s, g) = \mathbf{I}\{\phi(s) = g\},$8

together with a clipped value loss. HSR does not relabel by return quality under the original goal, and it does not use the SIL gate $r(s, g) = \mathbf{I}\{\phi(s) = g\},$9; instead it treats hindsight goal achievement itself as sufficient evidence for imitation (Oh et al., 2018).

A closer precursor is “Episodic Self-Imitation Learning with Hindsight”, which combines PPO with a hindsight self-imitation term over whole episodes and a return-comparison filter

$\phi:\mathcal{S}\to\mathcal{G}$0

That method is on-policy, episodic, and selectively filtered, whereas HSR is off-policy, transition-level, and unweighted (Dai et al., 2020).

In visual partially observable environments, “Visual Hindsight Self-Imitation Learning for Interactive Navigation” adds a hindsight self-imitation loss to A3C after relabeling failed episodes with a visual goal representation, using the total loss

$\phi:\mathcal{S}\to\mathcal{G}$1

This places VHS close to HSR in spirit, but its core novelty is visual goal relabeling through prototypical goal embeddings rather than HER-style goal-state relabeling (Kim et al., 2023).

At the level of credit assignment, “Hindsight Credit Assignment” offers a more selective alternative perspective. It introduces hindsight action posteriors such as

$\phi:\mathcal{S}\to\mathcal{G}$2

and derives hindsight-weighted advantage expressions based on posterior-to-prior ratios. This suggests a more causally selective version of hindsight imitation, although HSR itself does not use such weights and remains plain log-likelihood cloning on relabeled replay (Harutyunyan et al., 2019).

A later related system, “Translating Flow to Policy via Hindsight Online Imitation,” performs hindsight relabeling and self-imitation in a hierarchical manipulation setting by training a low-level policy to imitate actions conditioned on achieved point-flow goals. That method is better understood as an online goal-conditioned imitation scheme than as an explicit regularizer, but it shares the same basic logic: retrospectively treat achieved outcomes as goals and imitate the corresponding actions (Zheng et al., 22 Dec 2025).

5. Empirical role inside GCHR

The main empirical claims in the GCHR paper concern the combined method rather than HSR in isolation. The key ablation compares three variants: full GCHR with both HSR and HGR, HGR-only, and HSR-only. In Figure 1, the paper states that “HGR are more important than HSR in the GCHR framework.” The reported trend is that HSR-only improves over plain HER baselines such as SAC+HER or DDPG+HER in some settings, but the full combination performs best, and HGR-only is generally stronger than HSR-only. The paper’s explanation is that HSR learns a useful prior policy from hindsight-optimal relabeled data, whereas HGR broadens action support by exploiting relationships among goals (Lei et al., 8 Aug 2025).

This distinction is formalized by the paper’s action-support analysis. HSR is tied to observed replay actions: $\phi:\mathcal{S}\to\mathcal{G}$3 whereas HGR induces a broader support through a hindsight-goal policy prior. The theorem states

$\phi:\mathcal{S}\to\mathcal{G}$4

The intended interpretation is that HSR is anchored to exact replayed actions that previously worked under relabeled goals, while HGR can compose information from multiple visited goals and therefore generalize beyond those exact actions.

On eight robotics tasks from the Fetch and Hand suites, full GCHR achieves the best sample efficiency and final performance. The reported mean success rates include: Fetch Pick, SAC+HER $\phi:\mathcal{S}\to\mathcal{G}$5, DDPG+HER $\phi:\mathcal{S}\to\mathcal{G}$6, GCHR $\phi:\mathcal{S}\to\mathcal{G}$7; Fetch Push, SAC+HER $\phi:\mathcal{S}\to\mathcal{G}$8, DDPG+HER $\phi:\mathcal{S}\to\mathcal{G}$9, GCHR $\mathcal{S}$0; and Hand Reach, SAC+HER $\mathcal{S}$1, DDPG+HER $\mathcal{S}$2, GCHR $\mathcal{S}$3. These are full-method results, not HSR-only results, and the paper explicitly attributes the larger gains to the “synergistic interaction between HSR and HGR.”

The sample-efficiency claim is similarly framed at the level of GCHR as a whole. In FetchPush, to reach a mean success rate of $\mathcal{S}$4, DDPG+HER needs over 6000 training samples whereas GCHR needs about 4000, about $\mathcal{S}$5 more sample efficient. A careful reading therefore yields a narrower conclusion about HSR itself: HSR is empirically useful, but the strongest quantitative evidence in the paper supports the combined GCHR objective rather than HSR in isolation.

6. Implementation details, limitations, and significance

For reproduction, the paper’s main experiments choose SAC as the base method “due to its off-policy nature and stochastic policy representation,” although some baseline descriptions use a DDPG-style framework. HSR is applied only to the actor update, not the critic. Hindsight goals are sampled uniformly from future achieved states in the trajectory. The HER replay probability is $\mathcal{S}$6 in the main hyperparameter table. The appendix reports HSR-weight sweeps: $\mathcal{S}$7 gives FetchPush $\mathcal{S}$8, HandReach $\mathcal{S}$9; $\mathcal{A}$0 gives FetchPush $\mathcal{A}$1, HandReach $\mathcal{A}$2; $\mathcal{A}$3 gives FetchPush $\mathcal{A}$4, HandReach $\mathcal{A}$5; and $\mathcal{A}$6 gives FetchPush $\mathcal{A}$7, HandReach $\mathcal{A}$8. The default choice is therefore $\mathcal{A}$9, while the full method uses $\mathcal{G}$0 for HGR. Other reported hyperparameters are batch size $\mathcal{G}$1, replay buffer size $\mathcal{G}$2, learning rate $\mathcal{G}$3, discount $\mathcal{G}$4, Polyak coefficient $\mathcal{G}$5, warmup $\mathcal{G}$6 steps, random-action probability $\mathcal{G}$7, and Gaussian exploration noise scale $\mathcal{G}$8 (Lei et al., 8 Aug 2025).

The paper also specifies what HSR is not. It is not weighted behavior cloning; it does not use advantage weights, return thresholds, or selective filtering; and it does not alter the critic’s TD structure. This makes faithful implementation comparatively straightforward: an unweighted HER-conditioned behavior-cloning loss on relabeled replay, added to the actor objective with coefficient $\mathcal{G}$9.

Its main limitation follows directly from that simplicity. Because HSR only imitates actions that were actually observed on trajectories that later achieved the hindsight goal, it is “biased toward past experience” and has limited action coverage. The paper therefore argues that HSR alone does not fully exploit collected experience. It also notes that methods “based solely on self-imitation are highly sensitive to noise” in stochastic FetchPush variants, and cites the broader concern that “Imitating past successes can be very suboptimal.” A plausible implication is that HSR’s strength lies less in being a complete standalone algorithm than in serving as a stable policy-side mechanism for extracting more learning signal from HER-relabeled experience.

In that sense, HSR occupies a specific place in the development of hindsight-based RL. It takes the basic HER insight that many failed trajectories still contain achieved goals, and turns that hindsight success signal into direct supervised action learning. The resulting regularizer is technically simple, actor-local, and easy to integrate into off-policy actor-critic training. Its historical significance lies not in replacing HER, SIL, or goal-conditioned RL, but in making explicit a policy-learning step that those frameworks often leave implicit: if an action helped realize a hindsight goal, the actor can be trained to reproduce that action under that goal.