Latent Self-Imitation in Reinforcement Learning

• Latent self-imitation is a family of methods that reuses an agent’s own inferred behaviors, latent codes, or replayed memories to reinforce high-return trajectories.
• Mechanisms such as return-thresholding, off-policy optimistic updates, and adversarial reward shaping enable nuanced imitation without external expert signals.
• These approaches improve exploration and stabilize learning by converting self-generated internal representations into precise supervisory signals.

Searching arXiv for the cited papers and closely related work on self-imitation and latent-action imitation. “Latent self-imitation” is not a standardized term in the cited literature. As an Editor’s term, it denotes a family of methods in which an agent reuses its own successful behavior, inferred action structure, or compressed internal representations through an indirect carrier—optimistic TD targets, learned shaped rewards, lower-bound value targets, relabeled visual goals, latent action codes, or latent replay—rather than by directly cloning externally labeled expert actions. In the narrow reinforcement-learning sense, the prototype is self-imitation learning, which reinforces past state-action pairs only when their realized return exceeds the agent’s current estimate; in broader imitation-from-observation and lifelong-learning settings, analogous mechanisms appear when videos or trajectories are relabeled with inferred latent actions or when latent memories are replayed to preserve behavior across tasks (Ferret et al., 2020, Tang, 2020, Guo et al., 2018, Edwards et al., 2018, Ayalew et al., 2 Jun 2026, Kim et al., 2023, Yu et al., 11 Mar 2026).

1. Conceptual scope and taxonomy

The literature spans several distinct mechanisms that can be grouped under latent self-imitation only at the level of abstraction. Some are strict self-imitation methods in RL, because they explicitly reinforce the agent’s own past high-return trajectories. Others are broader self-supervised or lifelong-learning systems in which the “imitation” target is an internal latent code, a hindsight-reinterpreted goal, or a replayed latent memory.

Family	Carrier of imitation signal	Representative work
Return-thresholded self-imitation	Positive-part advantage against current value estimate	SIL and generalized lower-bound Q-learning (Tang, 2020)
Off-policy optimistic self-imitation	Modified reward and TD target using $\max(G_t, Q)$	SAIL (Ferret et al., 2020)
Adversarial self-imitation	Discriminator-shaped reward over good self-generated trajectories	GASIL/GARL (Guo et al., 2018)
Latent-action imitation from observation	Learned latent actions later aligned to real actions	ILPO (Edwards et al., 2018)
Latent-action world modeling	Inferred latent actions reused for cloning and planning	CLAW (Ayalew et al., 2 Jun 2026)
Visual hindsight relabeling	Failed trajectories reinterpreted as pseudo-successes in latent goal space	VHS (Kim et al., 2023)
Lifelong latent replay	Stored multimodal latent features reused for future policy refinement	MLR + IFA (Yu et al., 11 Mar 2026)

The strictest uses are SIL, SAIL, and GASIL, because they directly reuse self-generated trajectories as training signal. ILPO, CLAW, VHS, and MLR+IFA broaden the notion by relocating imitation into latent actions, visually relabeled goals, or stored latent embeddings. This suggests that “latent” in this area does not identify a single latent-variable formalism; it identifies the hidden or indirect carrier through which self-generated information is turned back into supervision.

2. Return-thresholded self-imitation and lower-bound value learning

The canonical self-imitation mechanism is selective replay of “good surprises.” In the on-policy SIL formulation summarized by SAIL, a transition is stored with Monte Carlo return $G_t$, and updates are applied only when the observed return exceeds the current value estimate: $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$ with $(x)_+ = \max(0,x)$. The positive-part truncation is the defining filter: unexpectedly good past behavior is reinforced, while non-beneficial or mediocre behavior is ignored (Ferret et al., 2020).

"Self-Imitation Learning via Generalized Lower Bound Q-learning" recasts this idea as lower-bound Q-learning and then generalizes it to an $n$-step bootstrapped form (Tang, 2020). Its practical lower-bound objective is

$$\min_\theta \mathbb{E}_\mathcal{D}\Big(\big[Q^\mu(x,a)-Q_\theta(x,a)\big]_+^2\Big),$$

and the generalized lower bound is

$$Q^{\pi^\ast}(x_0,a_0) \ge L^{\pi,\mu,n}(x_0,a_0) \coloneqq \mathbb{E}_\mu\!\left[\sum_{t=0}^{n-1}\gamma^t r_t + \gamma^n Q^\pi(x_n,a_n)\right].$$

This formulation makes two points explicit. First, original return-based SIL is recovered as the $n\to\infty$ limit. Second, self-imitation can be interpreted as a biased off-policy update whose bias is constrained to be positive and upper-bounded by $Q^{\pi^\ast}$.

The paper’s theoretical emphasis is the trade-off between fixed-point bias and contraction rate. It draws a close analogy to uncorrected $n$-step Q-learning, which propagates information rapidly but can introduce arbitrary bias. Generalized SIL preserves fast propagation while constraining the bias through the lower-bound construction. In that sense, latent self-imitation in value learning is not merely replay; it is a selective operator modification that lets past success act as a structured bootstrapping signal.

3. Optimistic off-policy self-imitation: SAIL

"Self-Imitation Advantage Learning" generalizes self-imitation to off-policy action-value methods such as DQN and IQN by embedding imitation directly into the reward and Bellman operator (Ferret et al., 2020). The core obstacle is that standard SIL assumes an explicit policy-gradient update and full-return targets, whereas off-policy control typically learns $G_t$0 with only an implicit greedy or exploratory policy. A second obstacle is stale returns: an old Monte Carlo return may become pessimistic relative to what the current agent can now achieve from the same state.

SAIL resolves both issues through a modified reward: $G_t$1 The mechanism is “latent” in the sense that self-imitation is carried by an optimistic TD target rather than by explicit policy cloning. If $G_t$2 exceeds the target critic’s estimate, the method behaves like self-imitation and boosts the sampled action. If the stored return is stale and pessimistic, the current $G_t$3 replaces it, preventing old trajectories from suppressing now-promising behavior. The same reward decomposes into an Advantage Learning term plus an imitation bonus: $G_t$4 The resulting loss remains ordinary TD regression,

$G_t$5

so no separate actor is required.

Algorithmically, SAIL stores transitions with placeholder returns, computes Monte Carlo returns at episode end,

$G_t$6

samples minibatches from replay, forms $G_t$7 using $G_t$8, and updates the critic by gradient descent. A notable design choice is the absence of prioritization; the authors explicitly avoid it to prevent domination by stochastic high-return outliers.

Empirically, SAIL is evaluated on the Arcade Learning Environment and is strongest on hard exploration and sparse-reward games. With DQN, it improves over vanilla DQN and AL-DQN, with especially large gains on Frostbite, Venture, Gravitar, PrivateEye, and Montezuma’s Revenge. On PrivateEye it outperforms stronger baselines such as C51 and IQN. Applied to IQN, SAIL-IQN outperforms IQN, AL-IQN, and Rainbow across aggregated Atari results, and it also performs better overall than RND while remaining combinable with RND for additional gains. The paper’s central claim is therefore that off-policy latent self-imitation can be realized as optimistic replay in the TD target rather than as explicit behavioral cloning.

4. Adversarial reward shaping as implicit self-imitation

"Generative Adversarial Self-Imitation Learning" frames self-imitation as a GAIL-like adversarial game in which the “expert” is the agent’s own good trajectory buffer (Guo et al., 2018). The method is motivated by temporal credit assignment under sparse, delayed, or noisy rewards. Instead of directly maximizing the environment reward, it trains a discriminator to distinguish current policy trajectories from stored good trajectories, typically the top-$G_t$9 episodes by discounted return. The policy then learns to fool the discriminator, which implicitly pushes it toward reproducing its own previously successful behavior.

The key shaped reward is

$$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$0

so the discriminator becomes a learned internal reward function. This is the paper’s implicit or latent self-imitation signal: the policy is not supervised to copy a particular trajectory, but it receives denser reward along state-action sequences resembling past good trajectories. The alternating training loop updates the good-trajectory buffer, trains the discriminator on current versus buffered rollouts, and then updates the policy with the discriminator-shaped reward. The paper’s main practical instantiation is PPO+GARL, which computes PPO advantages under $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$1.

The paper argues that this helps in four related ways: it provides dense intermediate guidance, improves long-horizon credit assignment, biases exploration toward previously promising regions of state-action space, and can reduce commitment to misleading short-term rewards. In the 2D Point Mass environment, PPO converges to a suboptimal policy that collects one green object, whereas PPO+GARL eventually learns a better policy that collects two green objects. In MuJoCo, GARL improves PPO on most tasks and improves PPO on all delayed-reward tasks by a large margin; the delayed setting delivers accumulated rewards every 20 steps or at episode end. The method also outperforms PPO+BC on most tasks and outperforms PPO+SIL in the reported experiments.

The paper is explicit about limitations. Performance depends on buffer design, discriminator-policy balance, and the multi-modal structure of the good-trajectory buffer. The authors also note that stronger theoretical guarantees remain future work, and that determinism assumptions are common in self-imitation-style improvement arguments. A common misconception is therefore that adversarial self-imitation is merely behavior cloning on top trajectories. In GASIL, the imitation signal is mediated by a learned reward; the discriminator is the latent carrier of self-imitation, not a direct supervised action label.

5. Latent actions as the medium of imitation from observation

"Imitating Latent Policies from Observation" moves imitation into a learned latent action space inferred from state-only demonstrations (Edwards et al., 2018). The method assumes discrete latent causes $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$2 of transitions, learns a latent forward model $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$3 and latent policy $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$4 from observation pairs, and only later aligns latent actions to real environment actions using a small amount of interaction data. The transition model predicts $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$5, and multimodality is induced by a winner-take-all objective,

$$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$6

combined with an expectation loss over the latent policy. The resulting policy learning problem is decomposed into $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$7 and $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$8 rather than direct $$\mathcal{L}^\text{sil}_{PG} = - \log \pi_{\theta}(a_t \mid s_t)\,(G_t - V_{\theta}(s_t))_+, \qquad \mathcal{L}^\text{sil}_{V} = \frac{1}{2}(G_t - V_{\theta}(s_t))_+^2,$$9. In classic control, ILPO is trained on 50,000 expert state observations and reaches expert-level performance in Cartpole and Acrobot after fewer than about 100 environment steps for action alignment; it outperforms BCO in all classic control settings. In CoinRun, trained on 1000 expert episodes, it also performs better than BCO, though visual alignment is slower and more initialization-sensitive.

CLAW extends the latent-action idea into a fully end-to-end self-supervised world model learned from action-free videos (Ayalew et al., 2 Jun 2026). Its Latent Action Model infers a continuous latent action

$(x)_+ = \max(0,x)$0

and a diffusion-based world model learns $(x)_+ = \max(0,x)$1 rather than action-conditioned dynamics. The LAM uses a ViT-based transition encoder whose output is $(x)_+ = \max(0,x)$2-normalized and typically has dimension 32; the reported implementation uses 64×64 inputs, 4×4 patches, embedding dimension 512, depth 8, and 8 heads. The world model is a U-ViT of depth 12 and 8 heads with a probability-flow ODE sampler using 50 denoising steps.

Its central anti-collapse device is adversarial latent regularization. The world model receives both an $(x)_+ = \max(0,x)$3 branch and a $(x)_+ = \max(0,x)$4-only branch connected through a gradient reversal layer,

$(x)_+ = \max(0,x)$5

so accurate prediction from $(x)_+ = \max(0,x)$6 alone is penalized. This prevents the latent code from leaking future visual content and pushes it toward action-like semantics. During imitation-from-observation, CLAW infers latent actions from expert videos, trains a policy with an $(x)_+ = \max(0,x)$7 regression objective to predict those latent codes, $(x)_+ = \max(0,x)$8, and grounds them to executable actions by nearest-neighbor retrieval against a small labeled latent-action/action reference set. Planning uses MPC-CEM over latent action sequences.

CLAW is not self-imitation in the strict RL sense of replaying one’s own successful trajectories, but it is self-imitation-like because the model’s own learned latent codes later become the supervision signal for cloning and planning. Experimentally, on Crafter it reaches 86.0% wood-harvest success and 4.46 average wood, versus 43.5% / 1.60 for AdaWorld and 35.5% / 1.12 for LAPO; it also improves open-drawer and close-drawer success on OGBench and is best overall on the real-world tabletop benchmark, especially on Push-Cover Duck. The ablation on adversarial regularization is particularly strong: on Crafter, the full model has PSNR 8.31 and $(x)_+ = \max(0,x)$9PSNR 8.90, whereas “CLAW (No Adv.)” drops to 1.33 and 0.79.

6. Visual hindsight, latent replay, and broader interpretations

"Visual Hindsight Self-Imitation Learning for Interactive Navigation" relocates self-imitation into a latent visual goal space (Kim et al., 2023). The setting is instruction-based interactive visual navigation in a POMDP with egocentric RGB observations and sparse success signals. VHS combines A3C, supervised contrastive goal-aware representation learning, and hindsight self-imitation. Its crucial representational device is the prototypical goal embedding

$n$0

built from stored successful terminal goal observations. When an episode fails, the terminal observation embedding $n$1 and final action $n$2 are used to relabel the entire trajectory as a pseudo-success under the new latent goal. The VHS loss is

$n$3

This differs from standard SIL: trajectories need not have been successful under the original instruction, because failure can be converted into a useful training target through visual hindsight relabeling. The method is applied only with probability $n$4 to avoid overfitting to noisy suboptimal paths. On three tasks, VHS achieves 91.4 ± 0.8% on Task1, 92.0 ± 0.8% on Task2, and 89.5 ± 0.9% on Task3. Using GDAN as reference in Task2, VHS reports SRR = 20.6% and SEI = 385.4%, while PG embedding alone reports SRR = 69.0% and SEI = 222.8%. Failure-only relabeling works best, and 80% triggering is slightly more stable than 100%.

"Lifelong Imitation Learning with Multimodal Latent Replay and Incremental Adjustment" pushes the concept further away from classical reward-based self-imitation and toward latent self-rehearsal (Yu et al., 11 Mar 2026). In lifelong imitation learning, the replay buffer stores latent multimodal features rather than raw trajectories: $n$5 where $n$6 is the concatenated frozen-encoder output for vision, language, and robot state. The policy uses a CLIP image encoder, CLIP text encoder, a 2-layer MLP for state, FiLM conditioning, a GPT-2 temporal decoder, and a policy head that predicts the action. To preserve inter-task distinctiveness, the method adds Incremental Feature Adjustment with angular distance

$n$7

and adaptive margin $n$8, using the language embedding as the reference. The method establishes a new state of the art on LIBERO: on LIBERO-OBJECT, FWT 84.6, NBT 11.4, AUC 79.4; on LIBERO-GOAL, FWT 80.0, NBT 6.9, AUC 77.2; on LIBERO-50, FWT 60.8, NBT 8.6, AUC 56.1. The abstract summarizes this as 10–17 AUC points over previous leading methods and up to 65% less forgetting. Even MLR alone outperforms prior SOTA methods, and removing FiLM drops LIBERO-OBJECT AUC from about 79.4 to about 41.6.

A useful distinction emerges from these works. In strict self-imitation, the reused object is a high-return trajectory or optimistic value target, as in SIL, SAIL, and GASIL. In broader latent self-imitation, the reused object is an internally generated representation: a latent action code inferred from observation, a relabeled visual goal, or a compact multimodal embedding stored for future rehearsal. The unifying pattern is not the presence of a particular latent variable, but the conversion of self-generated or self-inferred internal structure into downstream supervision.