Temporal Self-Imitation Learning

Abstract: Long-horizon robot manipulation policies trained with reward shaping can still achieve high return through inefficient interactions, while rare efficient behaviors discovered during training may be forgotten. We argue that temporal efficiency itself provides a powerful and underutilized source of self-supervision for reinforcement learning. We introduce Temporal Self-Imitation Learning (TSIL), a reinforcement learning framework that mines temporally efficient successful trajectories generated during learning and converts them into reusable supervision for future policy improvement. TSIL progressively refines learning using configuration-conditioned adaptive temporal targets derived from fast successful trajectories, while preserving and replaying efficient behaviors through efficiency-weighted self-imitation learning. Across 15 distinct long-horizon manipulation tasks, TSIL consistently improves learning efficiency, task-completion efficiency, revisitation of fast successful behaviors, and robustness to unstable training conditions. More broadly, our results suggest that the temporal structure of successful behavior itself provides a scalable self-supervisory signal for reinforcement learning beyond manually engineered reward shaping alone.

• The paper introduces a novel approach using adaptive temporal targets and efficiency-weighted self-imitation replay to enhance learning speed and behavioral efficiency.
• The paper's methodology integrates on-policy PPO with an auxiliary replay buffer to outperform baseline methods on long-horizon robotic tasks.
• Empirical results on Meta-World benchmarks show that TSIL achieves high success rates, improved sample efficiency, and reduced episode completion times.

Temporal Self-Imitation Learning: A Formal Essay

Motivation and Problem Formulation

Temporal Self-Imitation Learning (TSIL) introduces a reinforcement learning (RL) framework structured to mine and leverage temporally efficient successful trajectories for robot manipulation. The motivation arises from observations that standard reward-shaped RL, even with dense rewards and success signals, can drive agents toward task completion without guaranteeing behavioral efficiency or robustness. Agents frequently discover rare efficient behaviors during training but tend to discard them in subsequent updates, particularly in long-horizon manipulation regimes characterized by reward distraction, distribution shift, or unstable optimization. The paper clearly distinguishes between mere task success and temporally efficient task completion as separate evaluative axes, positing temporal efficiency as an underutilized self-supervisory signal.

Methodological Innovations

TSIL is structured around two principal mechanisms: adaptive temporal targets and efficiency-weighted self-imitation.

• Configuration-Conditioned Adaptive Temporal Targets: Rather than imposing global timing constraints or penalizing per-step duration, TSIL maintains a per-configuration temporal target reflecting the fastest successful trajectory for each particular environment instantiation. This target is dynamically updated as faster behaviors emerge, conditioning both observations and reward structure. The temporal target acts not as a hard constraint but as an adaptive bias on the policy, shaping reinforcement toward faster solutions while preserving the incentive for task completion.
• Efficiency-Weighted Self-Imitation Replay: TSIL supplements on-policy PPO with a replay buffer that retains the top-k fastest successful trajectories for each configuration. Replay is trajectory-level, relabeled according to the latest temporal target, and weighted by completion speed. Imitation updates are gated by advantage gaps and prioritized by speed, as opposed to generic high-return replay which favors reward but may preserve suboptimal behavior.

Crucially, TSIL does not modify the PPO algorithm itself; it augments the policy update by conditioning input and reward and integrates an auxiliary imitation loss over the temporally prioritized buffer.

Empirical Evaluation and Numerical Results

TSIL was evaluated on 15 long-horizon tasks from the Meta-World benchmark using the Isaac Gym robotic simulator. Comparative baselines included dense reward PPO (IH), per-step penalties (Step-cost IH), dense-to-sparse reward schedules (D2S IH), fixed temporal targets (FTTL), adaptive targets without replay (ATTL), ATTL with standard self-imitation (ATTL+SIL), and TSIL.

Key numerical results:

• Final Success Rate: TSIL obtained a mean success rate of $0.986 \pm 0.022$, outperforming all baselines.
• Sample Efficiency: TSIL consistently achieved highest area under the success curve (AUC), lowest steps-to-80% success, and greatest number of collected successful episodes.
• Behavioral Efficiency: TSIL produced lowest mean episode completion times for successful evaluation runs, indicating minimal wasted interaction.
• Robustness: Across policy gradient noise, reward dropout, aggressive PPO clipping, and learning rate sweeps, TSIL maintained strongest tradeoff between effectiveness, efficiency, and behavioral efficiency, anchoring optimization to fast successful behaviors and mitigating drift.

These results robustly support the claim that reward alone is insufficient to distinguish temporally efficient solutions from slower, reward-distracted trajectories.

Analysis of Optimization Dynamics

The analysis demonstrates that adaptive temporal targets systematically concentrate PPO update pressure on fast-success trajectories, suppressing reinforcement of reward-distracted and horizon-consuming failures. Replay diagnostics show TSIL’s efficiency-weighted buffer preserves the fastest successful behaviors and enhances their revisitation probability, aligning replay-induced updates more tightly with regions of fast-success memory likelihood compared to generic self-imitation replay.

This empirical alignment supports the theoretical premise that temporal structure discovered during learning is a scalable self-supervisory signal available for policy refinement beyond what is provided by reward shaping or generic imitation alone.

Implications and Limitations

TSIL has practical implications for skill acquisition in robotic manipulation, where temporal efficiency is critical for operational utility and safety. By transforming temporal efficiency into reusable supervision, TSIL enables RL agents to progressively refine their behavioral repertoire toward compact, reproducible strategies, thereby enhancing exploitation and consolidation stages post-exploration.

The framework’s neglect of initial exploration is acknowledged: TSIL depends on the base learner to provide occasional successful trajectories, after which it strengthens exploitation. For domains requiring behavioral diversity, such as preference for slower or safer maneuvers, TSIL could be augmented with additional metrics or multi-objective conditioning.

Memory efficiency is another concern, with the replay buffer maintained at modest capacity; future work could integrate more advanced replay compression or hierarchical memory to further optimize buffer composition.

Future Directions

Potential future developments include integrating temporal self-imitation with skill sequencing, hierarchically conditioned RL, multi-objective behavioral metrics, and broader applications to embodied foundation models. Extensions into real hardware deployment, adaptive memory compression, and policy distillation could yield further scalability and generalization.

Conclusion

Temporal Self-Imitation Learning establishes temporal efficiency as a central, reusable self-supervisory signal in RL for long-horizon manipulation. By leveraging adaptive configuration-conditioned temporal targets and efficiency-weighted trajectory replay, TSIL materially improves task effectiveness, sample efficiency, behavioral efficiency, and robustness on complex robotic benchmarks. The approach refines the direction of policy optimization and enhances preservation of efficient behaviors, charting promising new methodological grounds for reinforcement learning and robot autonomy (2606.19752).