BYOL-Explore: Latent Predictive RL

• The paper introduces BYOL-Explore as a curiosity-driven RL method that learns world representations, latent dynamics, and an exploration policy simultaneously using a single latent prediction loss.
• BYOL-Explore employs a latent predictive architecture with an online/target network pair, using closed-loop and open-loop recurrent cells for representing temporal dynamics in high-dimensional, partially observable environments.
• Empirical results demonstrate superhuman performance on hard Atari exploration tasks and significant advancements on complex benchmarks like DM-HARD-8 using intrinsic rewards.

Searching arXiv for the main BYOL-Explore paper and closely related BYOL/RL theory papers.

BYOL-Explore is a curiosity-driven reinforcement learning method for hard exploration from pixels that adapts Bootstrap Your Own Latent (BYOL) from self-supervised vision to partially observable, visually complex control. Its defining idea is to learn a world representation, latent dynamics, and an exploration policy jointly by optimizing a single prediction loss in latent space, with no additional auxiliary objective, and then to use that same latent prediction error as intrinsic reward (Guo et al., 2022). In BYOL-Explore, the analogue of BYOL’s two augmented views is no longer two transformations of the same image, but rather the pair consisting of a present history state with future actions and the actual future observation encoded by an exponential-moving-average target network. The method was introduced as a general exploration mechanism for sparse-reward domains, and was evaluated on the ten hardest exploration games in Atari and on DM-HARD-8, a visually rich, partially observable, continuous-action benchmark (Guo et al., 2022).

1. Problem setting and conceptual position

BYOL-Explore targets sparse-reward exploration in environments where random behavior rarely reaches extrinsic reward and where the observational stream is high-dimensional, partially observable, and often procedurally varied. The original paper emphasizes visually complex environments, long-horizon tasks, and settings in which useful behavior depends on understanding interaction dynamics rather than merely visiting unseen pixels (Guo et al., 2022). This places the method within the curiosity-driven RL tradition, but with a specifically latent-predictive formulation: states are interesting to the agent to the extent that their future latent representations remain hard to predict from its current latent history and the actions taken.

This design distinguishes BYOL-Explore from both pixel-prediction curiosity and static-domain BYOL. Standard BYOL learns image representations by predicting one augmented view of an image from another using an online network, a target network updated by exponential moving average, a predictor on the online branch, and a normalized regression loss (Grill et al., 2020). BYOL-Explore preserves that bootstrap structure, but replaces view agreement under augmentation with multistep future prediction under action-conditioned latent dynamics (Guo et al., 2022). The result is not a general world model in the reconstruction sense, since there is no pixel decoder and no separate auxiliary representation loss; instead, the single latent prediction loss is meant to simultaneously induce useful features, a predictive recurrent state, and an intrinsic reward.

A plausible implication is that BYOL-Explore belongs to a class of methods in which representation learning and exploration are not merely co-trained but operationally identified: the same error signal shapes both the model and the policy. The original paper makes this point explicitly by describing the method as learning “a world representation, the world dynamics, and an exploration policy all-together by optimizing a single prediction loss in the latent space with no additional auxiliary objective” (Guo et al., 2022).

2. Latent predictive architecture

The BYOL-Explore world model contains an online network and a target network. The online side consists of an encoder $f_\theta : \mathcal{O} \to \mathbb{R}^N$, a closed-loop recurrent cell $h^c_\theta$, an open-loop recurrent cell $h^o_\theta$, and a predictor $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$. The target side contains only an encoder $f_\phi$, updated by exponential moving average according to

$\phi \leftarrow \alpha \phi + (1-\alpha)\theta.$

This inherits the online/target asymmetry of BYOL, where the target branch supplies a bootstrap latent target and is not updated by direct gradient descent (Grill et al., 2020).

At time $t$, the encoder maps the current observation $o_t$ to $f_\theta(o_t)$, and the closed-loop recurrent state summarizes trajectory history: $b_t = h^c_\theta(b_{t-1}, a_{t-1}, f_\theta(o_t)).$ From that history state, the model performs open-loop rollout using future actions only: $h^c_\theta$0 The predictor $h^c_\theta$1 maps $h^c_\theta$2 to a predicted future latent, while the target encoder $h^c_\theta$3 encodes the actual future observation $h^c_\theta$4 (Guo et al., 2022).

This architecture is specifically designed for partial observability. The closed-loop state $h^c_\theta$5 is a latent belief or history representation rather than a Markov state inferred from a single frame, and the open-loop recurrent cell models future latent evolution without access to future observations. The paper also allows parameter sharing between the exploration model and the RL agent: $h^c_\theta$6 This means that the encoder and recurrent state used for policy optimization can be the same ones used for self-supervised predictive modeling (Guo et al., 2022).

The concrete instantiations differ across domains. On Atari, the paper uses $h^c_\theta$7 and $h^c_\theta$8, a Deep ResNet encoder on grayscale $h^c_\theta$9 images, GRUs for both recurrent modules, and one-hidden-layer MLPs of width 256 for policy, value, and predictor. On DM-HARD-8, it uses $h^o_\theta$0 and $h^o_\theta$1, a Deep ResNet on RGB observations, LSTMs for both recurrent modules, and a discretization of the continuous action space into 76 bins for open-loop rollout. The DM-HARD-8 recurrent state additionally incorporates hand force, a holding-object Boolean, hand-body distance, last action, previous reward under a signed hyperbolic transform, and an instruction encoded by an LSTM (Guo et al., 2022).

3. Loss, intrinsic reward, and optimization

The core BYOL-Explore objective is a normalized squared error between the online prediction of a future latent and the stop-gradient target latent from the EMA encoder: $h^o_\theta$2 For a batch of trajectories, the loss averages over trajectory index $h^o_\theta$3, start time $h^o_\theta$4, and prediction horizon $h^o_\theta$5: $h^o_\theta$6 with

$h^o_\theta$7

There is no separate reconstruction term, inverse-dynamics term, or contrastive term (Guo et al., 2022).

The intrinsic reward for transition $h^o_\theta$8 is defined by aggregating every latent prediction error whose target is $h^o_\theta$9: $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$0 Thus a transition receives high curiosity reward when many past latent rollouts fail to anticipate its resulting observation. This makes the intrinsic signal multistep by construction rather than restricted to one-step prediction (Guo et al., 2022).

Because this signal is highly nonstationary, the paper normalizes it using an EMA estimate of its standard deviation $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$1, with $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$2, so that the normalized reward is $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$3. The method can also apply reward prioritization by subtracting a running mean threshold and flooring at zero, retaining only higher-than-typical uncertainty. The policy is then trained either on pure intrinsic reward or on a mixture

$g_\theta : \mathbb{R}^M \to \mathbb{R}^N$4

where $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$5 is normalized extrinsic reward and $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$6 is BYOL-Explore intrinsic reward (Guo et al., 2022).

The experimental system uses VMPO, which the paper describes as an on-policy RL algorithm. For Atari, $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$7; for DM-HARD-8, $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$8. Value learning uses VTrace targets without off-policy corrections, with an MSE loss of weight 0.5. Entropy regularization is used on Atari but not on DM-HARD-8. The BYOL loss is scaled by 5.0 on Atari and 1.0 on DM-HARD-8, and optimization uses Adam with learning rate $g_\theta : \mathbb{R}^M \to \mathbb{R}^N$9 (Guo et al., 2022).

A notable methodological point is that BYOL-Explore uses no handcrafted visual data augmentation. The supervision comes from temporal structure: the present history plus future actions must predict the target encoder’s representation of the future observation (Guo et al., 2022).

4. Empirical performance

The method was evaluated on the ten hardest exploration games in Atari—Alien, Freeway, Gravitar, Hero, Montezuma’s Revenge, Pitfall, Private Eye, Qbert, Solaris, and Venture—and on DM-HARD-8, which comprises Drawbridge, Remember Sensor, Navigate Cubes, Wall Sensor, Wall Sensor Stack, Push Blocks, Throw Across, and Baseball (Guo et al., 2022). Atari performance is reported using human-normalized score,

$f_\phi$0

and clipped HNS, while DM-HARD-8 is summarized analogously with clipped and unclipped human-normalized measures (Guo et al., 2022).

On Atari, BYOL-Explore achieved a mean CHNS of $f_\phi$1, eight superhuman games, and a mean HNS of $f_\phi$2. A larger-sequence variant, “BYOL-Explore (big),” reached mean CHNS $f_\phi$3 and was superhuman on all ten hard-exploration Atari games (Guo et al., 2022). Per-game maxima included 124,524 on Alien, 153,124 on Hero, 13,518 on Montezuma’s Revenge, 25,175 on Pitfall, and 358,269 on Qbert, all above the cited human reference scores for those games (Guo et al., 2022).

In pure intrinsic-reward training on Montezuma’s Revenge, the method explored more than 20 rooms on average, compared with a best published RND result of 17 rooms. The paper also states that, in the pure intrinsic setting across Atari, BYOL-Explore was the only compared method to obtain positive reward in Pitfall (Guo et al., 2022).

On DM-HARD-8, a single agent and a single world model were trained jointly across all eight tasks, with tasks sampled uniformly at episode start. BYOL-Explore achieved mean CHNS $f_\phi$4, four superhuman tasks, mean HNS $f_\phi$5, and median HNS $f_\phi$6, strongly outperforming RND, ICM, and pure RL baselines (Guo et al., 2022). It exceeded human scores on Navigate Cubes, Wall Sensor, Throw Across, and Baseball, and performed strongly on five of the eight tasks overall. The paper highlights Throw Across in particular because prior work reportedly required human demonstrations to make substantial progress, whereas BYOL-Explore solved the majority of DM-HARD-8 tasks using intrinsic reward augmentation alone (Guo et al., 2022).

Two tasks remained especially difficult: Remember Sensor and Push Blocks. The paper attributes this to large variability from procedural generation and difficult memory or credit-assignment demands. This suggests that BYOL-Explore’s latent predictive curiosity is powerful but not uniformly sufficient across all partially observable manipulation problems (Guo et al., 2022).

5. Ablations and methodological interpretation

The most important ablation concerns the target network. When the target is fixed, corresponding to $f_\phi$7 and no EMA learning, performance drops substantially. The paper treats this as evidence that the learned bootstrapped target representation is crucial, and not merely a convenience inherited from BYOL (Guo et al., 2022). This suggests that target-network bootstrapping contributes directly to representation quality rather than only to numerical stability.

Prediction horizon matters differently across domains. On Atari, $f_\phi$8 can already be strong and may perform similarly to longer horizons in aggregate. On DM-HARD-8, however, one-step prediction learns more slowly and ends worse; appendix results show that increasing $f_\phi$9 from 1 to 16 steadily improves performance on the benchmark (Guo et al., 2022). This is consistent with the benchmark’s partial observability and long-horizon interaction structure.

Reward clipping or prioritization is not the dominant effect. On Atari it helps achieve especially high scores on games such as Montezuma’s Revenge and Hero, but the paper emphasizes that BYOL-Explore remains strong without it, and on DM-HARD-8 the unclipped variant is similar in aggregate. The main gain therefore appears to come from the latent bootstrapped prediction mechanism itself, not from thresholding the curiosity signal (Guo et al., 2022).

Representation sharing is also beneficial. A no-sharing variant remains viable on Atari, but the default shared configuration is generally stronger and simpler, because the same latent state supports both control and predictive modeling (Guo et al., 2022). The broader interpretation offered by the paper is that random-target methods such as RND can generate useful navigation novelty, but BYOL-Explore’s learned target encoder supplies a richer representation for interaction-heavy tasks, especially in environments where object manipulation and partial observability are central (Guo et al., 2022).

6. Relation to the BYOL family, limitations, and later developments

BYOL-Explore retains the structural ingredients that later analyses identify as central to BYOL itself: online/target asymmetry, a predictor on the online branch, stop-gradient on the target branch, and EMA target updates (Grill et al., 2020). Subsequent work on BYOL’s mechanisms argued that batch statistics are not essential for non-collapse, since BYOL can work with batch-independent normalization schemes (Richemond et al., 2020), and other analyses interpreted the predictor and EMA target as an approximate orthonormalization mechanism (Richemond et al., 2023). These papers are not about exploration, but they are directly relevant to understanding why a BYOL-style latent predictive loss can remain stable when used as an intrinsic-reward signal.

A later RL-specific extension, “Curiosity in Hindsight,” takes BYOL-Explore as its starting point and argues that predictive-error curiosity is fragile in stochastic environments because irreducible noise can generate persistent intrinsic reward. That paper instantiates its framework as BYOL-Hindsight, replacing plain forward prediction with hindsight-conditioned reconstruction and reporting improvements over BYOL-Explore on grid worlds, sticky-action Atari, and Montezuma’s Revenge with sticky actions, while preserving performance in the non-sticky setting (Jarrett et al., 2022). This establishes an important limitation of the original BYOL-Explore: the original paper shows particularly strong performance in fairly deterministic dynamics, and explicitly notes future work for more stochastic or highly procedurally generated environments such as NetHack (Guo et al., 2022).

Theoretical work on action-conditional self-predictive RL later analyzed BYOL-style objectives that condition on actions, showing that action-conditional objectives are aligned with per-action dynamics $\phi \leftarrow \alpha \phi + (1-\alpha)\theta.$0 and Q-like or control-relevant structure, and reporting that BYOL-AC is better overall in a variety of settings (Khetarpal et al., 2024). This is not a direct analysis of BYOL-Explore, but it is closely aligned with its design, since BYOL-Explore’s open-loop recurrent rollout is explicitly action-conditioned (Guo et al., 2022).

Within the broader BYOL literature, BYOL-Explore should also be distinguished from non-RL domain adaptations such as BYOL-A, which uses BYOL for general-purpose audio representation learning and is not an exploration method at all (Niizumi et al., 2022). BYOL-Explore is instead a latent world-model curiosity algorithm whose distinctive contribution is to make a single BYOL-style future-latent prediction loss serve simultaneously as representation learner, dynamics learner, and intrinsic reward generator (Guo et al., 2022). Its enduring significance lies in demonstrating that this unusually compact formulation can achieve superhuman exploration performance on the ten hardest Atari games and solve the majority of DM-HARD-8 tasks without demonstrations, while also exposing the central limitation that later hindsight-based methods target: plain latent prediction error is powerful, but it is not automatically robust to stochasticity (Guo et al., 2022).