Cause of instability in individual goal performance on Frozen Lake

Determine whether the observed instability in individual goal success rates during training with autonomous goal conditioning in the Frozen Lake environment is caused by policy forgetting, by the stochastic transition dynamics of Frozen Lake, or by other factors.

Background

The paper evaluates an environment-agnostic, reward-free goal-conditioning approach across Cliff Walking, Frozen Lake, and Pathological Mountain Car. Frozen Lake is a discrete grid environment with stochastic transitions, which introduces uncertainty into both learning and evaluation.

The authors report that although the average goal success rate on Frozen Lake stabilizes, the success rates for individual goals fluctuate substantially during training. They explicitly state that the underlying cause of this instability—whether policy forgetting, environmental stochasticity, or another factor—remains unresolved.