Sample More, Reflect Less: Self-Refine and Reflexion Lose to Repeated Sampling at Equal Token Cost, from 1.5B to 7B

Abstract: Methods that make a LLM plan, criticise and rewrite its own answer, reflect on mistakes, pick the best of several attempts, or debate with copies of itself nearly all make it generate far more text than a single chain of thought. Because generating more text raises accuracy by itself, a gain over one chain of thought does not show the method's idea is what helped. Wang et al. (2024) reported that a simple baseline, sampling the same question repeatedly and keeping the most common answer, often wins once budgets are comparable, but gave point estimates with no confidence intervals or significance tests. We rerun that comparison as a designed experiment: seven methods, open models of 1.5B, 3B and 7B parameters, two mathematics benchmarks, 150 questions each. We count every generated token, including those spent on critiques, reflections, debate turns and checking, and compare each method against repeated sampling at its own measured cost. All 36 comparisons are paired by question, with bootstrap intervals and multiplicity correction. No method is reliably better than repeated sampling at equal cost anywhere. Ten are reliably worse, all of them methods where the model inspects its own output, and all 18 self-inspection comparisons are negative. The two kinds of self-inspection part company as models grow. Choosing stops hurting: taking Best-of-N's eight samples and just counting the most common answer beats letting the model pick by 8.0 and 11.3 points at 1.5B, but only 2.0 and 1.3 at 7B, no longer distinguishable from zero. Rewriting does not recover: Self-Refine and a forced Reflexion stay 3.6 to 10.1 points below baseline at 7B. Reflexion as published never triggered its own retry on the smallest model. It judged itself correct every time and silently became a single chain of thought. We release code, prompts, all generations, and our verification scripts.

• The paper shows that no evaluated test-time reasoning method reliably outperforms cost-matched repeated sampling across GSM8K and MATH-500 for Qwen2.5 models from 1.5B to 7B parameters.
• The paper finds that Self-Refine, forced Reflexion, and Best-of-N self-verification often perform worse than independent sampling, with losses reaching 14.1 percentage points after accounting for generated-token cost.
• The paper demonstrates that majority voting over the same eight candidate solutions beats untrained self-verification by up to 17.3 percentage points, indicating that verifier training may matter more than prompting a model to critique itself.

The paper evaluates whether test-time reasoning methods provide benefits beyond the accuracy obtained by allocating the same inference budget to repeated sampling. Its central methodological claim is that comparisons against a single chain of thought are insufficient whenever the proposed method generates substantially more tokens. Self-Refine, Reflexion, Best-of-$N$ self-verification, and multi-agent debate may appear stronger than one-shot reasoning simply because they consume more computation. The appropriate comparator is therefore a self-consistency baseline evaluated at the same realized token cost. The study’s principal conclusion is that, on automatically graded mathematical reasoning tasks and open-weight models from $1.5$B to $7$B parameters, no evaluated method reliably improves on repeated sampling at equal generated-token cost, while methods involving untrained self-assessment or self-revision are frequently worse (2607.28576).

Study design and evaluation protocol

The work is framed as a statistically strengthened replication and extension of budget-aware evaluations such as “Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies” (Wang et al., 2024). It evaluates Qwen2.5-Instruct models at $1.5$B, $3$B, and $7$B parameters on 150 fixed questions from GSM8K and 150 fixed questions from MATH-500. Inference uses Q8 quantisation and CPU-based llama.cpp execution. The models generate at most 1,024 completion tokens per call.

The methods include greedy chain-of-thought, Plan-and-Solve, Self-Refine, Reflexion, a forced Reflexion variant, Best-of-$N$ with self-verification, and three-agent multi-agent debate. The reference method is self-consistency: independent temperature-$0.7$ chains are generated and aggregated by majority vote. Rather than imposing a common query cap, the authors measure each method’s actual completion-token expenditure and compare its accuracy with the self-consistency curve at the corresponding cost. All critique, reflection, verification, debate, and intermediate messages count toward the budget.

This design is important because the baseline is not a single operating point. Self-consistency produces an accuracy-cost frontier. For example, on Qwen2.5-1.5B, self-consistency increases GSM8K accuracy from approximately $70.1\%$ at one sample to $80.9\%$ at 16 samples, while MATH-500 accuracy rises from $1.5$0 to $1.5$1. At $1.5$2B, the corresponding curves rise from $1.5$3 to $1.5$4 on GSM8K and from $1.5$5 to $1.5$6 on MATH-500. A proposed reasoning procedure must therefore outperform the relevant point on this curve, not merely exceed a cheap one-shot baseline.

To make cost matching computationally tractable, the authors generate a pool of 16 independent samples per question and estimate self-consistency performance at several sample counts by subsampling this pool. They average 200 subsampled runs at each value of $1.5$7. The resulting estimates are highly stable: changing the subsampling seed shifts method-versus-baseline differences by at most 0.36 percentage points, with a mean shift of 0.12 points.

The statistical analysis is paired at the question level. Differences are estimated using 10,000 paired bootstrap resamples, and Holm–Bonferroni correction is applied to account for multiple method comparisons within each model-dataset setting. The design has limited power: typical 95% intervals span approximately $1.5$8 to $1.5$9 percentage points, so modest effects cannot be distinguished reliably from zero. The authors appropriately interpret nonsignificant results as compatibility with a range of effects rather than evidence of exact equivalence.

Main cost-matched results

Across 36 method-setting comparisons, no method is significantly better than cost-matched self-consistency. Ten comparisons are significantly worse after within-setting Holm correction, and 26 are statistically indistinguishable from the baseline. Thirty of the 36 point estimates are negative. When the correction is applied across all 36 comparisons simultaneously, only two differences remain significant, making the strongest claims dependent on the within-setting multiplicity convention.

The negative results are concentrated in methods that inspect, revise, or select among model-generated work. Self-Refine, forced Reflexion, and Best-of-$7$0 with self-verification are below the cost-matched baseline in all 18 comparisons reported for those methods. The strongest losses include:

Model and benchmark	Method	Difference from matched sampling
Qwen2.5-3B, MATH-500	Best-of-$7$1 self-verification	$7$2 pp
Qwen2.5-7B, MATH-500	Forced Reflexion	$7$3 pp
Qwen2.5-1.5B, MATH-500	Forced Reflexion	$7$4 pp
Qwen2.5-1.5B, MATH-500	Best-of-$7$5 self-verification	$7$6 pp
Qwen2.5-1.5B, GSM8K	Best-of-$7$7 self-verification	$7$8 pp
Qwen2.5-7B, MATH-500	Self-Refine	$7$9 pp

Several of these intervals exclude zero after correction. In particular, Best-of-$1.5$0 self-verification is significantly worse on Qwen2.5-1.5B/GSM8K, Qwen2.5-1.5B/MATH-500, and Qwen2.5-3B/MATH-500. Forced Reflexion is significantly worse in multiple settings, including a $1.5$1 percentage-point difference on Qwen2.5-7B/MATH-500. Self-Refine remains significantly below the baseline at $1.5$2B by $1.5$3 points on GSM8K and $1.5$4 points on MATH-500.

The results do not support a simple distinction between elaborate and simple prompting. Chain-of-thought and Plan-and-Solve generally lie near the matched sampling frontier, as does debate in most settings. The more specific pattern is that tokens spent on self-assessment and iterative rewriting are less productive than tokens spent on additional independent attempts. This is a stronger and more operational claim than the assertion that “simple methods win”: the relevant variable is how additional computation is converted into a final decision.

The paper also checks whether temperature confounds the comparison. Greedy chain-of-thought is compared with a single temperature-$1.5$5 sample at nearly identical cost. The differences are generally small and statistically uncertain, although greedy chain-of-thought is 5.7 points better on Qwen2.5-3B/MATH-500. This control does not eliminate every possible decoding interaction, but it indicates that the broad negative results cannot be explained solely by comparing deterministic methods with stochastic sampling.

The controlled selection experiment

The most compelling experiment isolates the final selection rule while holding the candidate solutions fixed. Best-of-$1.5$6 generates eight solutions and asks the model to identify the most likely correct one. The authors then apply ordinary majority voting to those same eight solutions. Thus, the model, problem, candidate set, generated tokens, and candidate ordering are held constant; only model-based judging versus answer counting changes.

Counting outperforms self-verification in all six model-benchmark settings:

Model and benchmark	Self-verification	Majority counting	Difference
Qwen2.5-1.5B, GSM8K	$1.5$7	$1.5$8	$1.5$9 pp
Qwen2.5-1.5B, MATH-500	$3$0	$3$1	$3$2 pp
Qwen2.5-3B, GSM8K	$3$3	$3$4	$3$5 pp
Qwen2.5-3B, MATH-500	$3$6	$3$7	$3$8 pp
Qwen2.5-7B, GSM8K	$3$9	$7$0	$7$1 pp
Qwen2.5-7B, MATH-500	$7$2	$7$3	$7$4 pp

The authors’ strongest causal interpretation follows from this matched-candidate comparison: the principal loss is not caused by generating multiple solutions, but by asking the same untrained model to judge them. The verifier agrees with the majority answer on 63–95% of questions, but its errors are concentrated on disagreements. On smaller models, it tends to override the majority with an incorrect candidate. The paper reports substantial first-position bias, particularly for the $7$5B model, which selects the first candidate approximately three-quarters of the time.

The dependence on model scale is also informative. The penalty decreases from 8.0 and 11.3 points at $7$6B to 2.0 and 1.3 points at $7$7B. Below $7$8B, the judging-versus-counting difference is significant in all four settings. At $7$9B, neither difference is statistically distinguishable from zero. The authors do not claim that self-verification surpasses majority voting at $N$0B; rather, the data support parity within the available sample size. This result is consistent with work showing that trained verifiers can outperform untrained majority voting, including “Incentivizing LLMs to Self-Verify Their Answers” (Zhang et al., 2 Jun 2025). The comparison suggests that verifier training, rather than the mere act of prompting a generator to verify, may be the relevant intervention.

Reflexion and the problem of silent degeneration

Reflexion provides a separate methodological lesson because its control flow is adaptive. The published procedure asks the model whether its answer is correct and retries only when the model responds that it is not. On Qwen2.5-1.5B, the self-assessment declared the answer correct on every GSM8K and MATH-500 question. Reflexion therefore performed no reflection or retry and collapsed into a single chain-of-thought call.

This behavior materially changes the interpretation of its score. The method appears competitive on the smallest model because it is inexpensive, not because reflection improves reasoning. On Qwen2.5-3B, the same prompts trigger retries on approximately 23% of GSM8K questions and 69% of MATH-500 questions, producing much higher costs. The authors consequently evaluate a forced variant that always executes three reflection-and-retry rounds. That variant is consistently below cost-matched sampling, including losses of $N$1 points on Qwen2.5-3B/GSM8K and $N$2 points on Qwen2.5-7B/MATH-500.

The implication is methodological rather than specific to Reflexion: final accuracy is insufficient for adaptive test-time methods. Evaluation must instrument whether the adaptive branch actually executes, how frequently it executes, and how much budget it consumes. Otherwise, a procedure can receive credit for a mechanism that is never activated.

Robustness, implementation errors, and auditability

The study includes unusually extensive debugging and reproducibility analysis. Two errors were discovered during the project. First, an initial server context limit caused Best-of-$N$3 requests with long MATH-500 candidate sets to fail selectively. The failures disproportionately removed hard questions: on Qwen2.5-1.5B/MATH-500, chain-of-thought scored $N$4 on the surviving 78 questions but only $N$5 on the complete 150-question sample. The affected cells were rerun with a larger context allocation, preventing biased evaluation of Best-of-$N$6.

Second, an analysis routine mistakenly scored Best-of-$N$7 by majority vote over its candidate list rather than by the verifier’s selected candidate. This bug made the method appear approximately equivalent to self-consistency. After correction, Best-of-$N$8 became worse than the baseline in every setting and supplied the paper’s clearest negative result. The authors independently re-derived the reported statistics using a second analysis program and obtained agreement on 96 checks.

These failures strengthen the paper’s argument about evaluation design. The claimed conclusion depends not only on model outputs but on preserving the semantics of each method throughout data storage, grading, aggregation, and cost matching. The release of prompts, raw generations, harnesses, and analysis scripts makes the results auditable rather than dependent on summary statistics.

The robustness analysis also reveals sensitivity to answer truncation. On MATH-500, unparseable responses reach 16% in some settings because the 1,024-token cap prevents the final boxed answer from being emitted. Sampling-based methods are less affected because only some candidates need to finish successfully, whereas single-shot and iterative methods can lose a question when their final generation is truncated. Restricting evaluation to questions for which every method produced a parseable answer changes some estimates by as much as 5.2 points and alters significance for several comparisons. Because this restricted subset excludes harder questions non-randomly, neither analysis should be treated as universally preferable. The primary results therefore include a task- and cap-dependent robustness concern.

Limitations and open questions

The evidence is bounded to exact-answer mathematical reasoning on GSM8K and MATH-500, three Qwen2.5 model sizes, Q8 quantisation, and one implementation of each method. The authors explicitly do not establish that self-critique is ineffective for open-ended generation, where quality is multidimensional and majority voting over exact answers is unavailable. On such tasks, the baseline tested here may not exist in a directly applicable form.

The statistical intervals quantify variation across questions, not variation across random seeds. Sampling, Best-of-$N$9, and debate are each executed once per question-method configuration, so seed-level uncertainty is not estimated. This omission likely makes the reported significance evidence somewhat overconfident for stochastic methods. The strongest claims should consequently be read as conditional on the realized sampling runs.

The cost metric counts generated tokens but excludes input tokens. This accounting is favorable to methods that repeatedly reread critiques, candidate solutions, or debate messages. A partial reconstruction for Best-of-$0.7$0 shows that it consumes 1.25–1.42 input tokens per output token, compared with 0.22–0.37 for sampling. Under total-token accounting, Best-of-$0.7$1 becomes even worse in the smaller models, with differences reaching $0.7$2 points. Input costs were not reconstructed for Self-Refine, Reflexion, or debate because their intermediate contexts were not fully stored.

Latency may also alter the practical comparison. Independent sampling is highly parallelizable, whereas refinement and reflection are sequential. Thus, a latency-constrained deployment with sufficient parallel hardware could penalize sequential methods more strongly than the completion-token metric does. Conversely, hardware constraints, batching, and model-serving economics can make token count an incomplete proxy for monetary cost.

Finally, the $0.7$3B results do not establish behavior at larger scales. The judging penalty approaches zero but does not reverse, and the number of verifier-majority disagreements becomes small: eight on GSM8K and 24 on MATH-500 at $0.7$4B. The specific open question is whether a larger or trained verifier raises its accuracy on disagreement cases above 50%, the threshold required to outperform majority voting on otherwise identical candidates. The present results identify that quantity but do not estimate it reliably beyond the tested scale.

Conclusion

The paper demonstrates that test-time reasoning methods must be evaluated against repeated sampling at matched inference cost. Across the reported mathematical reasoning experiments, no method significantly beats this baseline, while Self-Refine, forced Reflexion, and untrained Best-of-$0.7$5 self-verification frequently lose by substantial margins. The controlled eight-sample experiment isolates the decisive factor: majority counting is better than asking the same model to judge its own candidates, by up to 17.3 percentage points on the tested settings.

The study does not show that reflection or verification is universally ineffective. It establishes a narrower result: for small and mid-sized open models solving exact-answer mathematics, untrained self-assessment is a poor use of additional tokens relative to independent sampling. It also shows that adaptive procedures can silently fail to execute and that implementation-level auditing is essential for credible test-time compute evaluations (2607.28576).