Determine the effect of rollout oversampling and partial-rollout continuation

Determine whether the oversampling-based rollout collection rule, which accepts the first completed trajectories and cancels the remaining tail, introduces harmful bias against long and difficult terminal tasks, and establish whether preserving unfinished trajectories through partial-rollout continuation improves final task performance.

Background

The training system launches more terminal-task trials than needed, accepts the first trajectories to complete, and cancels the remainder to bound the duration of each reinforcement-learning step. Because completion time depends on task family, difficulty, and agent behavior, this procedure can preferentially retain faster tasks and exclude difficult long-horizon trajectories from optimization.

The paper identifies this as an unresolved trade-off. Although partial-rollout continuation is proposed as a possible way to preserve supervision from unfinished trajectories, the authors do not establish whether it improves final task performance. Resolving the issue would clarify the interaction between rollout throughput, task-distribution bias, and learning effectiveness in long-horizon terminal-agent reinforcement learning.

References

We retain oversampling for its wall-clock benefit, while treating its effect on long-task coverage as an open trade-off. Preserving unfinished trajectories through partial-rollout continuation is one direction discussed in Section~\ref{sec:limitations}; its benefit to final task performance remains to be established.

— T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks  (2609.11042 - Yang et al., 10 Sep 2026) in Section “Lessons learned and what did not work,” subsection “Rollout throughput changes the training distribution”