The Road Less Traveled: Enhancing Exploration in LLMs via Sequential Sampling (2510.15502v1)

Abstract: Reinforcement learning (RL) has been pivotal in enhancing the reasoning capabilities of LLMs, but it often suffers from limited exploration and entropy collapse, where models exploit a narrow set of solutions, leading to a loss of sampling diversity and subsequently preventing RL from further improving performance. This issue is exacerbated in parallel sampling methods, where multiple outputs are drawn from the same distribution, potentially causing the model to converge to similar solutions. We propose SESA, a novel SEquential SAmpling framework that mitigates this challenge by generating diverse solution sketches sequentially before expanding them into full reasoning paths. This approach ensures broader exploration by conditioning each new output on previous ones, promoting diversity throughout the process and preventing policy collapse. Our experiments on a synthetic task show that sequential sampling consistently outperforms traditional RL methods in terms of path diversity and recovery from collapse. Further evaluations on real-world tasks demonstrate that SESA improves both the exploration of valid strategies and the overall performance of LLMs. On three agent benchmarks, SESA lifts success rates by $+0.25$, $+0.42$, and $+0.07$ absolute over the base model (up to an additional $211\%$ relative improvement over baseline RL), underscoring its exploration advantage. This work introduces a structured approach to exploration, paving the way for more effective and diverse reasoning in RL-trained LLMs. Our code is released at https://github.com/MuLabPKU/sesa.

• The paper introduces SESA, a sequential sampling framework that conditions on previously generated outputs to prevent entropy collapse in RL-trained LLMs.
• It demonstrates that sequential sampling uncovers up to 80 unique solutions versus 11 with parallel methods, significantly enhancing exploration in various tasks.
• The method employs a two-stage approach to efficiently manage long output generation, preserving diversity while improving performance in synthetic, RL, and reasoning tasks.

Enhancing Exploration in LLMs via Sequential Sampling: An Analysis of SESA

Introduction and Motivation

Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving the reasoning capabilities of LLMs, particularly in domains requiring complex, multi-step reasoning. However, a persistent challenge in RLVR is the phenomenon of entropy collapse, where the model's policy converges to a narrow set of high-reward solutions, leading to a loss of output diversity and stagnation in further learning. This issue is exacerbated by the prevalent use of parallel sampling, where multiple outputs are generated independently from the same policy, resulting in highly similar completions and a rapid collapse of exploration.

The paper introduces SESA (SEquential SAmpling), a novel framework that addresses this exploration bottleneck by conditioning each new candidate solution on the previously generated ones, thereby actively promoting diversity and preventing policy collapse. The approach is validated across synthetic and real-world tasks, demonstrating substantial improvements in both solution diversity and downstream task performance.

Sequential Sampling vs. Parallel Sampling

Traditional RLVR methods employ parallel sampling, where $G$ completions are drawn i.i.d. from the current policy $\pi_\theta(\cdot|x)$. This approach is prone to rapid convergence to a single dominant solution, as all samples reinforce the same high-reward trajectory, leading to a "dead policy" state with minimal exploration.

SESA replaces this with sequential sampling, where each candidate $y^{(i)}$ is generated conditioned on the input $x$ and all previous candidates $y^{(1)},\ldots,y^{(i-1)}$. This conditioning can be implemented via prompt engineering, where the model is explicitly instructed to avoid duplicating prior solutions. The result is a set of candidates that are maximally diverse, as each new sample is actively discouraged from repeating previous outputs.

Figure 1: Training overview. Baseline parallel rollout (left) samples all solutions i.i.d. from the same distribution, while our sequential rollout (right) first generates diverse methods sequentially, then expands each into full solutions in parallel.

Empirical results on a synthetic path exploration task show that parallel sampling quickly plateaus in the number of unique solutions discovered, while sequential sampling continues to uncover new strategies, achieving up to 80 unique solutions compared to 11 for parallel sampling.

Two-Stage Sequential Sampling for Real-World Tasks

While fully sequential sampling is tractable for short outputs, real-world reasoning tasks (e.g., math, code generation) often require long completions, making sequential generation of all candidates computationally prohibitive and susceptible to context window limitations. SESA addresses this with a two-stage procedure:

1. Stage I: Sequential Method Drafting Generate $G$ concise "method sketches" sequentially, each conditioned on the input and all previous sketches. These sketches represent distinct high-level strategies or plans.
2. Stage II: Guided Solution Generation For each sketch, generate a full solution in parallel, conditioning only on the input and the corresponding sketch. This ensures that each solution is anchored to a unique plan, preserving both diversity and self-consistency.

This two-stage approach maintains the exploration benefits of sequential sampling while restoring the computational efficiency of parallel expansion for long outputs.

Policy Update and Training Objective

Despite the sequential generation of candidates, the policy update is performed using the standard RL objective, evaluating each candidate under the original input $x$ (not the full history). The loss is computed as in GRPO/DAPO, with advantages normalized across the $G$ candidates. This design ensures that the model does not overfit to the sequential context and remains compatible with standard RLVR evaluation protocols.

Empirical Results

Synthetic Path Exploration

Sequential sampling uncovers strategies that parallel sampling fails to discover and retains a larger proportion of correct outputs. In the synthetic task, SESA achieves a Coverage@32 of 77%, compared to 5% for parallel sampling, and is able to recover diversity even when initialized from a collapsed policy.

RL Agent Benchmarks

On three classic RL agent tasks (Sokoban, Countdown, FrozenLake), SESA outperforms both the RAGEN baseline and its entropy-regularized variant, achieving absolute success rate improvements of +0.25, +0.42, and +0.07, respectively, over the base model. Notably, the relative improvement in Sokoban (+277.8%) and Countdown (+280.0%) is substantial, indicating that SESA is particularly effective in environments with sparse or deceptive rewards.

Reasoning and Math Tasks

On symbolic Sudoku and AIME24 math reasoning tasks, SESA improves Pass@k metrics, especially at higher $k$, indicating that it preserves a broader set of valid solution strategies. In math reasoning, SESA achieves a 9% improvement in Pass@k over the DAPO baseline, with comparable Pass@1, demonstrating that diversity does not come at the expense of top-1 accuracy.

Figure 2: Performance comparison on Sudoku and AIME24. The results show the effectiveness of our two-stage sequential sampling method in improving solution diversity, particularly in higher Pass@k metrics.

Diversity Metrics and Policy Collapse

The Pass@16/Pass@1 ratio is used as a proxy for solution diversity. SESA maintains a high ratio throughout training, while parallel sampling methods see this ratio decline to 1, indicating a collapse into a single dominant solution.

Figure 3: Comparison of Pass@16/Pass@1 ratios for sequential and parallel sampling. Sequential sampling maintains higher diversity, while parallel sampling declines, indicating a collapse into a ``dead policy'' with reduced exploration.

SESA is also able to "revive" dead policies: when RL training with parallel sampling stalls, switching to sequential sampling restores diversity and enables further improvement.

Implementation Considerations

• Prompt Engineering: Sequential sampling requires careful prompt design to ensure that each candidate is aware of and avoids duplicating previous solutions. This can be implemented via explicit instructions or by listing prior outputs in the prompt.
• Context Window: For long outputs, the two-stage approach is essential to avoid exceeding the model's context length and to prevent instruction drift.
• Computational Cost: While sequential sampling introduces some overhead in candidate generation, the two-stage design mitigates this for long-form reasoning tasks.
• Compatibility: SESA is compatible with existing RLVR algorithms (e.g., GRPO, DAPO) and can be integrated with minimal changes to the policy update logic.

Theoretical and Practical Implications

The results demonstrate that exploration in RLVR for LLMs is fundamentally limited by the sampling paradigm. Parallel sampling structurally induces entropy collapse, while sequential sampling provides a principled mechanism for maintaining and expanding the reasoning frontier. This has direct implications for the design of RL algorithms for LLMs, suggesting that diversity-promoting sampling is essential for sustained improvement, especially in tasks with multiple valid solutions or sparse rewards.

From a practical perspective, SESA enables RL-trained LLMs to discover and retain a broader set of strategies, improving robustness and generalization. The ability to recover from collapsed policies is particularly valuable in long-running RL training regimes.

Future Directions

• Automated Diversity Constraints: Beyond prompt engineering, future work could explore learned diversity constraints or explicit penalization of duplicate solutions.
• Integration with Tree Search: Combining sequential sampling with tree-based exploration (e.g., MCTS) may further enhance exploration in structured reasoning tasks.
• Scaling to Larger Models and Tasks: Evaluating SESA on larger LLMs and more complex domains (e.g., code synthesis, theorem proving) will be critical for assessing its scalability and generality.
• Reward Model Interactions: Investigating how sequential sampling interacts with process-based reward models and dense credit assignment could yield further gains in RLVR.

Conclusion

SESA provides a robust, theoretically grounded, and empirically validated framework for enhancing exploration in RL-trained LLMs. By replacing parallel sampling with a sequential, history-aware candidate generation process, SESA overcomes the structural limitations of entropy collapse, enabling sustained improvement in both diversity and task performance. The two-stage design ensures practical applicability to real-world reasoning tasks, and the approach is compatible with existing RLVR pipelines. This work establishes sequential sampling as a critical component for future advances in LLM reasoning and RL-based alignment.