Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

Abstract: Exploration is a prerequisite for learning useful behaviors in sparse-reward, long-horizon tasks, particularly within 3D environments. Curiosity-driven reinforcement learning addresses this via intrinsic rewards derived from the mismatch between the agent's predictive model of the world and reality. However, translating this intrinsic motivation to complex, photorealistic environments remains difficult, as agents can become trapped in local loops and receive fresh rewards for revisiting forgotten states. In this work, we demonstrate that this failure stems from a lack of spatial persistence and episodic context. We show that effective curiosity requires a model of the world that is persistent and continuously updated, paired with an agent that maintains an episodic trajectory history to navigate toward novel regions. We achieve this using an online 3D reconstruction as a persistent model of the world, while the agent policy is parameterized as a sequence model over RGB observations to maintain episodic context. This design enables effective exploration during training while allowing the agent to navigate using solely RGB frames at deployment. Trained purely via curiosity on HM3D, our agent outperforms RL-based active mapping baselines and generalizes zero-shot to Gibson and AI-generated worlds. Our end-to-end policy enables efficient adaptation to downstream tasks, such as apple picking and image-goal navigation, outperforming from-scratch baselines. Please see video results at https://recuriosity.github.io/.

• The paper presents a persistent 3D Gaussian Splatting model that continuously updates environmental maps and generates stable intrinsic rewards for exploration.
• It employs a transformer-based episodic policy with sliding-window causal and global linear attention to leverage long-horizon memory and avoid local loops.
• Empirical results demonstrate superior scene coverage and zero-shot transfer to downstream tasks, underscoring the model’s adaptability and intrinsic motivation efficiency.

Episodic Context and Persistent World Models for Curiosity-Driven Exploration in 3D Environments

Introduction

Curiosity-driven exploration has historically offered a principled solution for addressing sparse or delayed reward scenarios in reinforcement learning (RL), particularly in complex 3D environments. However, standard intrinsic motivation methods often suffer from a lack of persistent world memory and insufficient episodic trajectory context, leading to premature convergence to local behavior cycles and spurious novelty signals. In "Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration" (2605.22814), the authors advance the field by demonstrating that effective large-scale exploration is predicated on two requirements: a persistent, incrementally updated model of the environment, and a policy architecture capable of leveraging full episodic context via long-horizon memory.

Methodological Contributions

Persistent 3D World Model

The primary innovation is the use of an online 3D Gaussian Splatting (3DGS) system as an explicit, continuously updated forward model of the environment. This world model consumes privileged RGB-D and camera pose streams during training to reconstruct the encountered 3D scene in real-time. Intrinsic curiosity rewards are issued based on the prediction error between the rendered view from this model and the agent’s actual RGB observation, with a low-pass filter and downsampling mitigating incidental high-frequency error. Unlike traditional forward dynamics models with short-term or non-episodic memory, the 3DGS-based model produces stable novelty signals, is resistant to catastrophic forgetting within an episode, and enables spatial persistence necessary for exploration.

Episodic Sequence Policy for Novelty Seeking

The agent is parameterized as a transformer with per-step input streams comprising RGB observations and action representations (as Plücker-ray images). A novel fusion layer cross-attends learned frame queries to both image patches and DINOv2 visual features. The sequence model employs sliding-window causal attention for tractable training and a global linear-attention memory module to support very long-term context. This enables the policy to condition actions on full episodic trajectory information, moving beyond map-based RL approaches that bypass semantic context, or limited-memory policies that cannot coordinate back-tracking or strategic exploration when rewards are locally sparse.

Training Procedure and Regularization

Policy optimization is executed with PPO, using the intrinsic reward as the sole signal. To prevent policy entropy collapse, a scheduled mixture of learned and random (uniform) policy actions is annealed through training, guaranteeing persistent exploratory behavior and robustness as intrinsic rewards become sparse. Critically, during deployment, the agent operates entirely from RGB; persistent mapping and privileged information are only utilized during training, aligning with real-world deployment constraints.

Empirical Results

Superior Exploration in Static Realistic 3D Environments

Evaluated on the HM3D and Gibson datasets, the proposed agent achieves higher 3D completeness (scene coverage) than baselines such as Active Neural SLAM (ANS) and Occupancy Anticipation (OccAnt). Notably, performance exceeds even methods utilizing privileged depth at test time, while maintaining a strict RGB-only test regime. The agent rapidly explores and covers more unique scene surface area across rollout horizons (256, 512, 1024 steps), with substantially reduced mean distance between observed and ground-truth points.

Ablative Analysis of Memory and Persistence

Ablation studies confirm strong dependence on both persistent world models and sufficient agent memory. Removing persistence from the 3DGS or limiting policy sequence length degrades exploration, with agents devolving into local loops or failing to navigate away from explored territories. Asymmetric memory (e.g., memoryless actor with memoryful critic) performs better than no memory, but full transformer sequence models yield the best results.

Transfer to Downstream Tasks and Zero-Shot Generalization

The agent, pretrained purely via curiosity on exploration, can be rapidly fine-tuned for sparse-reward navigation tasks—such as apple-picking and image-goal navigation—outperforming from-scratch policies, especially in extremely sparse reward settings. In addition, the agent generalizes zero-shot to AI-generated scenes with distinct appearance and rendering pipelines, demonstrating robustness to distribution shift.

Theoretical and Practical Implications

By explicitly separating deployment from training requirements, and decoupling spatial persistence from end-to-end policy memory, this work identifies two fundamental ingredients for scalable RL exploration: persistent closed-loop world models that mitigate state revisitation artifacts, and policy architectures that support long-horizon planning. The persistent 3DGS proxy grounds curiosity in observed environment structure, which, although restricted to static scenes, is an important baseline for the evaluation of future action-conditioned generative models.

The demonstrated flexibility of the policy to adapt to arbitrary semantic tasks post hoc, without pre-assumed geometric mapping at inference, underscores the potential for unified exploration architectures in multi-task embodied AI. Further, the empirical findings challenge the reliance on explicit map-based policies for RL exploration, advocating for end-to-end visually conditioned exploration as a more adaptable, scalable paradigm.

Future Directions

The authors identify a major limitation: the static world assumption of their current 3DGS-based forward model. Transitioning to learned, action-conditioned generative world models with persistent spatial memory will be crucial for extending these results to dynamic environments. As generative scene models mature, integrating intrinsic motivation grounded in persistent, adaptively updated representations will be necessary for robustness in the presence of environmental non-stationarity and temporal drift. Future work should also explore the implications for lifelong and continual learning, open-world task generalization, and embodied agents in real-world robotics.

Conclusion

This study rigorously demonstrates that curiosity-driven exploration in visually rich 3D environments requires both spatially persistent world models for reliable intrinsic reward and episodic sequence policies for long-horizon novelty planning. The presented system outperforms state-of-the-art RL map-based baselines, generalizes across domains, and is highly adaptable to downstream tasks with only RGB input. These findings set a new standard for scalable, task-agnostic exploration agents in embodied AI, delineating theoretical requirements for intrinsic motivation that must be met by future dynamic world model research.