Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models

Abstract: Large Reasoning Models (LRMs) have shown remarkable performance on challenging questions, such as math and coding. However, to obtain a high quality solution, one may need to sample more than once. In principal, there are two sampling strategies that can be composed to form more complex processes: sequential sampling and parallel sampling. In this paper, we first compare these two approaches with rigor, and observe, aligned with previous works, that parallel sampling seems to outperform sequential sampling even though the latter should have more representation power. To understand the underline reasons, we make three hypothesis on the reason behind this behavior: (i) parallel sampling outperforms due to the aggregator operator; (ii) sequential sampling is harmed by needing to use longer contexts; (iii) sequential sampling leads to less exploration due to conditioning on previous answers. The empirical evidence on various model families and sizes (Qwen3, DeepSeek-R1 distilled models, Gemini 2.5) and question domains (math and coding) suggests that the aggregation and context length do not seem to be the main culprit behind the performance gap. In contrast, the lack of exploration seems to play a considerably larger role, and we argue that this is one main cause for the performance gap.

• The paper demonstrates that parallel sampling achieves higher accuracy than sequential methods on challenging math and code tasks.
• The paper identifies limited solution space exploration and pattern-copying by induction heads as key factors driving the performance gap.
• The paper shows that even advanced feedback and aggregation strategies cannot bridge the gap, emphasizing the need for diversity-promoting techniques.

Analysis of the Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models

Introduction

"Understanding Performance Gap Between Parallel and Sequential Sampling in Large Reasoning Models" (2604.05868) presents a rigorous empirical and mechanistic investigation of inference-time sampling strategies in state-of-the-art Large Reasoning Models (LRMs) on domains requiring complex multi-step reasoning, notably mathematical problem-solving and code generation. The paper scrutinizes why parallel sampling—where multiple candidate solutions are independently generated per prompt and then aggregated—consistently outperforms sequential sampling, where solutions are sampled in a dependent chain and each solution conditions on prior ones. The analysis is supported by evaluations on contemporary, highly capable LRM families, including Qwen3, DeepSeek-R1-Distill Qwen, and Gemini 2.5 series, and leverages the AIME2025 and LiveCodeBench v5 benchmarks. The central finding is that the performance gap is not primarily due to aggregation or context length, but is instead rooted in diminished solution space exploration under sequential regimes.

Formalization of Sampling Strategies

The methodology distinguishes two principal solution-sampling protocols:

• Parallel Sampling: $N$ solution candidates are sampled independently, each from the same prompt. Aggregation mechanisms (e.g., majority vote, best-of-$N$ by reward) then select the final output.
• Sequential Sampling: The sampling process is autoregressive—each round samples a new solution conditioned on the preceding solutions, optionally compressed or summarized, with possible feedback prompts for self-refinement or error-based debugging. The last solution in the chain is typically selected.

The paper extends this taxonomy by examining both Markov sequential sampling (conditioning only on the last solution) and fully autoregressive methods (conditioning on the full chain thus far).

Figure 1: Illustration of sampling approaches considered, contrasting independent parallel sampling with Markov and auto-regressive sequential sampling.

Empirical Evaluation and Performance Trends

Mathematical Reasoning

Experiments on AIME2025 (30 challenging, integer-answer math questions) demonstrate that parallel sampling robustly outperforms sequential sampling across all model scales and families, with the gap persisting as more solutions are sampled. Notably, performance scaling with the number of samples occurs for both paradigms, but sequential scaling saturates rapidly and achieves lower peak accuracy. Interestingly, the Markov variant outperforms fully autoregressive sequential sampling, reinforcing the hypothesis regarding accumulation of suboptimal dependencies.

Figure 2: Majority voting-based parallel sampling yields superior accuracy over both forms of sequential sampling in math benchmarks, even as the number of samples increases.

Code Generation

On LiveCodeBench v5 (167 programming tasks at varying difficulty), parallel sampling with best-of-$N$ aggregation consistently provides higher code correctness across model variants. Again, parallel sampling is especially advantageous on harder instances, with sequential sampling failing to scale as effectively in solution quality with more samples.

Figure 3: For code generation, best-of-N parallel sampling consistently surpasses sequential alternatives, prevalent across model sizes and difficulty strata.

Feedback and Self-Refinement

Multiple feedback prompts in sequential sampling—simple re-tries, targeted self-refinement, and error-based debugging using public and (oracle) private test feedback—were evaluated. The key finding is that, under practical feedback (no access to ground-truth failures), parallel sampling exhibits a performance advantage that cannot be closed by feedback sophistication alone.

(Figure 4; Figure 5)

Figure 4: Detailed comparison of different sequential feedback strategies shows re-answering often outperforms self-refinement for math tasks.

Figure 5: On Gemini 2.5, the performance dynamics for sequential sampling under "Please re-answer" and self-refinement feedback are generally similar and both are outperformed by parallel sampling.

Dissecting the Performance Gap

Aggregation is Not the Cause

Applying the same aggregation operator (majority vote for math, best-of-N for coding with oracle or pseudo-oracle reward) to sequential sampling narrows, but does not close, the gap with parallel sampling. Notably, with stronger ideal aggregation (oracle best-of-N), the gap sometimes widens, indicating that aggregation is not the dominant limiting factor.

(Figure 6; Figure 7; Figure 8)

Figure 6: Both sampling approaches with aggregation; sequential performance lags parallel even under identical aggregation rules.

Figure 7: Best-of-N aggregation and high-quality test-based reward cannot fully close the parallel/sequential gap in code generation.

Figure 8: Even with perfect-verification aggregation, parallel sampling achieves higher ultimate accuracy than sequential.

Context Length is Not the Primary Factor

Comparison of input context lengths between strategies reveals that, in practice, Markov sequential and parallel sampling have comparable context sizes. Fully autoregressive sequential sampling incurs the longest contexts but, crucially, the total context sizes (including full trace and solution chain) do not explain the systematic deficit in sequential performance. Insertion of large quantities of irrelevant context (up to 32k tokens) does not degrade aggregation performance, refuting the hypothesis that context size alone causes the gap.

Figure 9: Detailed plot shows input and overall context lengths, finding that context expansion in autoregressive sequential sampling does not correlate with improved performance over parallel.

Exploration: The Essential Deficit

Analysis of generated solutions reveals a conspicuous lack of diversity in sequential regimes. Models often converge to repeated, nearly identical solutions across sequential rounds, even in the presence of errors or explicit prompts to self-refine. Embedding-based cosine similarity metrics confirm that sequential solutions have higher inter-solution similarity than those from independent parallel sampling. This effect, described as "laziness," prevents escape from local solution modes and hinders exploration of the broader solution space.

Markov and autoregressive sequential variants suffer especially in more complex coding tasks, with models being unable to recover from early errors or invalid solution proposals.

Remediation Attempts: High-Quality Feedback

Providing oracle-level, high-quality running errors for both public and private test failures in the sequential chain improves performance, indicating that additional external guidance can force deeper exploration and partial recovery. Yet, even in this artificially favorable scenario, parallel sampling often matches or exceeds sequential performance on hard distributions.

Figure 10: Oracle-level error feedback in sequential sampling closes, but does not always eliminate, the gap with parallel sampling—especially for more capable models and easier tasks.

Mechanistic Insights: Induction Heads and Pattern Copying

The study visualizes Transformer attention maps (induction heads) in the Qwen3-14B model during sequential sampling. The attention clearly locks onto positions of previous solutions, confirming known pattern-copier and in-context learning mechanisms are active in LRMs.

(Figure 11; Figure 12)

Figure 11: Induction head activations overlay prior solutions onto new generations, irrespective of correctness, leading to recapitulation of prior outcomes in the absence of strong external feedback.

Figure 12: Increasing the number of previous solutions in the context proportionally multiplies induction head traces, underscoring the "anchoring" of new solution attempts to historical outputs.

This mechanistic evidence supports the empirical finding that sequential sampling encourages LRMs to remain in narrow solution neighborhoods, rather than exploring a diverse set of possible reasoning paths.

Implications and Future Directions

The central empirical and mechanistic result is that solution space exploration is inherently limited by sequential sampling paradigms due to a pattern-copying, local-fixation bias, even when context management and aggregation are carefully controlled. This realization has several notable implications:

• Test-Time Inference Strategy: Independent parallel sampling remains the most reliable, readily scalable test-time solution scaling strategy for LRMs in highly challenging reasoning tasks, given current architectures and training schemes.
• Training and Decoding Optimization: Strategies that explicitly encourage solution diversity—potentially through reinforcement learning or process-level dropout—are required to make sequential chains competitive, especially when external oracle guidance is unavailable in realistic deployment.
• Mechanistic Interpretability: Induction head phenomena should be considered when designing autoregressive multi-round LRM protocols, especially where exploration is critical. Interventions to weaken or diversify attention to past solutions may enhance search.
• Broader Reasoning Task Coverage: Similar analyses should be extended to agentic reasoning, structured multi-turn interaction, and hybrid parallel-sequential task decompositions in LRMs.

Conclusion

This work provides a comprehensive empirical and mechanistic account of why parallel sampling dominates sequential sampling in LRM test-time solution scaling for math and code reasoning tasks. The lack of exploration induced by over-conditioning on prior solutions is traced to pattern-copying behavior manifested by induction heads. Aggregation and context effects are found to be secondary. Future work should focus on exploration-promoting sampling and decoding strategies and mechanistic de-biasing to close the solution quality gap in multi-round inference with LRMs.