Rollout-Budget Allocation Across RL Environments

Determine how to allocate a reinforcement-learning rollout budget across heterogeneous environments according to rollout cost, reward reliability, and skill coverage.

Background

Unlike fixed-pool supervised fine-tuning, online reinforcement learning generates training data through interaction with environments. Different environments can require different rollout costs, provide rewards of varying reliability, and cover different skills.

The paper notes that WebRL filters tasks within WebArena, but this does not resolve the broader allocation problem across heterogeneous environments. The unresolved question concerns how an agent should distribute limited interaction resources while balancing these environment-specific differences.

References

For environment and task selection, where should an agent interact during RL? Unlike fixed-pool SFT, environments differ in rollout cost, reward reliability, and skill coverage. WebRL filters tasks generated from prior failures by feasibility and critic-estimated difficulty, but only within WebArena. How to allocate a rollout budget across heterogeneous environments remains largely unexplored~\citep{qi2025webrl}.

— Data Strategies for Large Language Models: A Unified Perspective on the Training Lifecycle  (2610.21681 - Xu et al., 15 Oct 2026) in Section 4, subsection “Summary and Discussion,” paragraph “Open questions”