No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery (2408.15099v3)

Abstract: What data or environments to use for training to improve downstream performance is a longstanding and very topical question in reinforcement learning. In particular, Unsupervised Environment Design (UED) methods have gained recent attention as their adaptive curricula promise to enable agents to be robust to in- and out-of-distribution tasks. This work investigates how existing UED methods select training environments, focusing on task prioritisation metrics. Surprisingly, despite methods aiming to maximise regret in theory, the practical approximations do not correlate with regret but with success rate. As a result, a significant portion of an agent's experience comes from environments it has already mastered, offering little to no contribution toward enhancing its abilities. Put differently, current methods fail to predict intuitive measures of ``learnability.'' Specifically, they are unable to consistently identify those scenarios that the agent can sometimes solve, but not always. Based on our analysis, we develop a method that directly trains on scenarios with high learnability. This simple and intuitive approach outperforms existing UED methods in several binary-outcome environments, including the standard domain of Minigrid and a novel setting closely inspired by a real-world robotics problem. We further introduce a new adversarial evaluation procedure for directly measuring robustness, closely mirroring the conditional value at risk (CVaR). We open-source all our code and present visualisations of final policies here: https://github.com/amacrutherford/sampling-for-learnability.

• The paper reveals that conventional UED scoring functions poorly correlate with true task learnability, limiting their practical effectiveness.
• The paper introduces Sampling For Learnability (SFL), a method that dynamically selects training levels with balanced success rates to optimize learning.
• Experimental results in single- and multi-agent JaxNav and Minigrid domains demonstrate that SFL significantly outperforms Domain Randomization and other UED baselines.

Overview of "No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery"

In the field of reinforcement learning (RL), the paper "No Regrets: Investigating and Improving Regret Approximations for Curriculum Discovery," authored by Alex Rutherford et al. at the University of Oxford, addresses the pivotal issue of optimizing training data and environments to enhance downstream performance. The research investigates the robustness and efficacy of Unsupervised Environment Design (UED) methods, particularly in scenarios inspired by real-world robotics problems.

The core of curriculum learning in RL lies in creating diverse sets of training environments to foster the development of policies that generalize well across both in- and out-of-distribution tasks. UED methods, which automatically generate training environments, have been lauded for their theoretical robustness guarantees. However, this paper questions the practical performance of state-of-the-art UED approaches when applied to a new setting involving continuous single- and multi-robot navigation tasks, coined JaxNav.

Analysis and Findings

The authors rigorously assess existing UED methods and find them lacking. Notably, these methods either fail to surpass the simple baseline of Domain Randomization (DR) or necessitate considerable hyperparameter tuning to achieve comparable results. The analysis pinpoints the root cause: the scoring functions employed by these methods do not align well with intuitive measures of "learnability"—defined as the propensity for environments to foster gradual learning improvements.

Key insights from the analysis include:

1. Correlation of Score Functions and Learnability: The scoring functions, such as Positive Value Loss (PVL) and Maximum Monte Carlo (MaxMC), exhibit weak correlation with the actual learnability of tasks. The paper provides extensive empirical data showing that these functions fail to prioritize environments where the agent's performance varies significantly, a haLLMark of learnability.
2. True Regret vs. Heuristics: When true regret can be computed, it markedly outperforms DR. This finding underscores the inadequacy of current regret approximations, such as PVL and MaxMC.

The Proposed Solution: Sampling For Learnability (SFL)

In response to these findings, the authors introduce a novel and straightforward approach called Sampling For Learnability (SFL). The main premise of SFL is to directly train on levels that the agent can sometimes, but not always, solve. This strategy targets environments with a success rate that provides an optimal learning signal.

The algorithm operates by maintaining a buffer of highly learnable levels and periodically updating this buffer by rolling out the current policy on a uniformly sampled set of levels. Those levels where the policy shows inconsistent results (both successes and failures) are considered learnable and prioritized for further training.

Evaluation Protocol: Conditional Value at Risk (CVaR)

To evaluate the robustness of the trained policies, the authors introduce a new adversarial evaluation protocol inspired by the Conditional Value at Risk (CVaR) metric. This protocol assesses the performance of an agent by measuring its success rate in the worst $\alpha\%$ of a newly sampled set of environments. This comprehensive evaluation exceeds traditional methods that rely on a preselected hand-designed set of test environments, providing a more rigorous assessment of robustness.

Experimental Results

The empirical results presented are compelling:

1. Single-Agent JaxNav: SFL outperforms all baselines, including DR and various UED methods, across several challenging environments.
2. Multi-Agent JaxNav: SFL demonstrates significant improvements in environments involving multiple robots, showcasing its versatility.
3. Minigrid and XLand-Minigrid: In these standard UED domains, SFL outperforms traditional methods, particularly in worst-case scenarios as measured by CVaR.

Theoretical and Practical Implications

The findings of this paper have several far-reaching implications:

• Theoretical: The inadequacies of current UED scoring functions call for the development of more reliable regret approximations. The success of SFL suggests that direct measures of task learnability could serve as a better foundation for future UED methods.
• Practical: For real-world applications like robotic navigation, SFL provides a more efficient approach to curriculum learning, ensuring robust policies that can generalize well across diverse tasks and environments.

Future Directions

This research opens several avenues for future investigation:

1. Improving Scoring Functions: Continued refinement of scoring functions that better approximate true regret without computational intractability is crucial.
2. Adaptive Methods: Further exploration into adaptive methods that can dynamically adjust the difficulty and nature of training environments in response to the agent's learning progress.
3. Wider Applicability: Testing the efficacy of SFL in other domains beyond robotics to validate its generalizability and impact.

In conclusion, this paper presents a rigorous and insightful critique of current UED methodologies and introduces a promising new approach, SFL, that enhances the robustness and efficiency of curriculum discovery in reinforcement learning. The novel evaluation protocol, comprehensive empirical analysis, and practical implications make this paper a significant contribution to the field.