CorrectBench: Self-Correcting LLM Benchmark

• CorrectBench is a benchmark that measures how large language models self-correct their responses in reasoning-intensive tasks like commonsense, mathematical, and code generation challenges.
• It categorizes self-correction into intrinsic, external, and fine-tuned strategies, analyzing trade-offs between accuracy improvements and computational overhead.
• Empirical evaluations reveal that iterative self-correction can boost accuracy significantly, yet often at the cost of increased inference time and resource consumption.

CorrectBench is a specialized benchmark created to rigorously evaluate the capacity of LLMs to self-correct their outputs in reasoning-intensive scenarios. Unlike conventional evaluations focused solely on generation, CorrectBench systematically measures the accuracy and efficiency impacts of iterative self-correction strategies across three core domains: commonsense reasoning, mathematical reasoning, and code generation. It differentiates among self-correction approaches—categorizing them into intrinsic, external, and fine-tuned strategies—while highlighting the nuanced trade-offs between accuracy gains and computational overhead. CorrectBench thus serves as a diagnostic and comparative testbed for both LLM development and deployment, with empirical analysis of iterative correction, hybrid frameworks, and comprehensive baselines including chain-of-thought reasoning.

1. Motivation and Scope

The central motivation behind CorrectBench is to address whether LLMs can reliably refine their own outputs and, if so, under which circumstances and strategies such improvements manifest. The benchmark covers three principal reasoning-intensive tasks:

• Commonsense reasoning (e.g., HotpotQA, CommonsenseQA, GPQA)
• Mathematical reasoning (e.g., GSM8K, AQUA, MATH)
• Code generation (e.g., HumanEval)

CorrectBench assesses not just an LLM’s initial response, but how performance evolves as the model is tasked with correcting or improving its own output through iterative, self-reflective prompting. The comprehensive framework includes standard instruction-following LLMs and those specifically enhanced for reasoning.

2. Self-Correction Approaches

CorrectBench formalizes and distinguishes self-correction in LLMs along three principal axes:

• Intrinsic Correction (S1): Self-refinement performed internally by the LLM, without requiring additional resources. Methods in this category (e.g., RCI, Self-Refine, CoVe) prompt the LLM to detect and fix its own errors leveraging in-context reasoning.

Strengths: Requires no external tools or data; easy to deploy.

Weaknesses: Efficacy highly model-dependent and may falter for complex, subtle errors.

• External Correction (S2): Employs resources outside the model, such as tool-assisted verification or retrieval-augmented mechanisms (e.g., Reflexion-v2, RARR, RATT, CRITIC). Outputs are refined by flagging discrepancies through external processes, then using the feedback to guide improvement.

Strengths: Can yield higher accuracy by consulting external or “authoritative” sources; beneficial for structured or fact-based corrections.

Weaknesses: Introduces computation and infrastructure dependencies; may constrain open-ended reasoning.

• Fine-Tuned Correction (S3): Involves specialized continued pretraining or supervised fine-tuning from correction-oriented datasets (e.g., DCoT, SCORE, SuperCorrect). The LLM leverages learned correction patterns directly in generation.

Strengths: Potential for substantial gains in domain-specific settings.

Weaknesses: Requires curated correction data; improvements may not generalize across diverse task types.

An important design in CorrectBench is the mixture framework, wherein the output of one correction approach can seed another, enabling hybrid strategies for potentially synergistic improvements.

3. Evaluation Methodology and Metrics

The evaluation flow in CorrectBench consists of two primary regimes: baseline single-pass prompting and iterative self-correction. In the latter, after generating an initial answer, the model receives the previous response in its prompt and is then tasked to refine its output, iteratively.

Empirical comparisons are provided for diverse LLMs and correction strategies across the three task domains. Accuracy is quantified as: $$\text{ACC} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(\text{predicted}_i = \text{true}_i)$$ where $\mathbb{I}$ is the indicator function over $N$ examples.

Results are reported both for accuracy improvements (absolute and relative to base prompting) and computational costs (inference time, number of correction steps). This permits analysis of the accuracy–efficiency trade-off.

4. Empirical Findings and Comparative Baselines

CorrectBench establishes that:

• Self-correction often produces substantial accuracy gains, especially on difficult, multi-step reasoning problems such as those in GPQA and MATH.
• The margin of improvement can exceed 20% over base prompting for some intrinsic methods on hard datasets.
• External and hybrid strategies may yield further, albeit incremental, gains when leveraged in combination.
• Reasoning-enhanced LLMs (e.g., DeepSeek-R1) are less susceptible to further gains from additional correction steps, with diminishing returns despite increased time costs.
• A standard chain-of-thought (CoT) prompting baseline remains notably competitive, frequently providing a favorable balance between accuracy and compute overhead.

This suggests that, beyond a certain sophistication level, more elaborate correction pipelines do not always translate to proportional performance improvements.

5. Trade-Offs and Efficiency Challenges

A core issue identified by CorrectBench is the trade-off between accuracy improvements and operational efficiency. Increased accuracy via multiple self-correction iterations or hybrid pipelines comes at the cost of longer inference time and greater computational resources. In time-sensitive or resource-constrained environments, this could negate the practical value of such strategies, especially as CoT baselines deliver similar gains at lower cost.

Additionally, for models explicitly trained for reasoning, further correction may yield only marginal improvements. This indicates an upper bound determined by the underlying model architecture and pretraining.

6. Implications and Future Directions

CorrectBench demonstrates the practical promise of self-correction for enhancing the reliability and accuracy of LLMs in high-stakes, reasoning-centric applications. However, the computational inefficiencies observed indicate a need for:

• Adaptive mechanisms that determine dynamically when (or whether) self-correction is warranted based on model uncertainty or detected error patterns.
• Exploring lightweight self-correction modules or judicious integration of external verification only when significant uncertainty or detected inconsistency arises.
• Improved strategies for hybridization to maximize synergistic benefit while minimizing redundant computation.

A plausible implication is that the path forward may involve context- and confidence-aware self-correction triggers, selective use of external tools, and continued development of specialized correction-augmented pretraining.

7. Practical Deployment and Benchmark Significance

CorrectBench provides a reproducible, structured testbed for the design and deployment of robust self-correcting LLM pipelines. Its multi-faceted structure and reporting facilitate transparent comparison across strategies, models, and reasoning tasks. For practitioners, it provides actionable insights into which correction methods are likely to be cost-effective or necessary for specific deployment constraints.

CorrectBench’s empirical findings—that self-correction boosts accuracy, especially for complex reasoning, but at the expense of efficiency—inform both research priorities and the design of balanced, adaptable LLM systems.