ReasonBENCH: Unified Reasoning Benchmarks

• ReasonBENCH is a unified framework that integrates diverse benchmarks, such as ActionReasoningBench and L0-Reasoning Bench, to evaluate reasoning across domains.
• It employs multi-run protocols and variance metrics like confidence intervals, coefficient of variation, and MAD to overcome saturation and assess model stability.
• The methodology emphasizes stepwise, explainable output and adversarial test construction to probe both surface performance and deep reasoning capabilities.

ReasonBENCH refers both to concrete reasoning benchmarks and to a broader conceptual aspiration for unified, rigorous evaluation of reasoning capabilities across LLMs, multimodal models, and generation systems. The term has been used for diverse benchmarks including ActionReasoningBench (for actions and change), L0-Reasoning Bench (for procedural step-by-step correctness), and, by proposal, as a blueprint for a next-generation comprehensive reasoning benchmark suite. Its emergence is rooted in persistent limitations and the saturation of prior benchmarks, the challenge of measuring actual reasoning (as opposed to surface pattern matching or rote recall), and the need for reproducible, granular, and variance-aware assessment across a wide range of domains and modalities.

1. Motivation: The Need for Unified and Unbiased Reasoning Evaluation

Reasoning benchmarks are curated evaluation suites designed to probe models' capabilities for complex inferences—commonsense, mathematical, spatial, logical, compositional, and, in some cases, procedural or visual reasoning. The rapid scaling of LLMs and large reasoning models (LRMs) has led to saturation on many existing datasets: state-of-the-art models achieve near-ceiling performance (≥0.8 capacity as defined by $\mathrm{capacity}(B) = \max_{m \in M} A(m, B)$), thus eroding the ability of these tests to differentiate models or drive research advances (Deveci et al., 3 Nov 2025).

Saturation is compounded by training data contamination and the lack of diversity in benchmark structure, format (e.g., mainly multiple-choice), and reasoning categories. This has led to calls for successor benchmarks—sometimes denoted as "ReasonBENCH"—explicitly designed to overcome these limitations through formal reasoning type categorization, adversarial test construction, metadata-rich instances, and requirement of explainable reasoning outputs (Deveci et al., 3 Nov 2025).

2. Key Benchmarks and ReasonBENCH Implementations

ActionReasoningBench ("ReasonBENCH" for Actions and Change)

ActionReasoningBench (Handa et al., 6 Jun 2024) systematically evaluates model performance in reasoning about actions and change (RAC), particularly focusing on the classical AI "frame problem" and on ramification constraints (indirect effects). It covers 13 deterministic IPC domains and probes six core reasoning dimensions:

• Fluent Tracking
• State Tracking
• Action Executability
• Effects of Actions
• Numerical RAC
• Composite Questions

The benchmark measures model accuracy across these categories, with task protocols including plan traces of varying lengths, binary and free-form questions, and systematic inclusion/exclusion of ramification constraints for indirect effect inference. Top models reach only moderate performance in fluent and executability tasks and struggle profoundly with longer plans, numerical reasoning, and ramifications, highlighting a fundamental gap in current LLM reasoning for dynamic domains.

L0-Reasoning Bench: Procedural (Level-0) Reasoning

L0-Reasoning Bench (Sun et al., 28 Mar 2025), also referred to as ReasonBENCH in this context, isolates "level-0 reasoning"—the ability to apply simple rules step by step, without error, across long computational traces (e.g., simple Python function traces). This focus on procedural correctness enables fine-grained detection of failure modes, error accumulation, and robustness of state tracking. The benchmark reveals that larger models and those trained with explicit stepwise thought tokens maintain procedure accuracy better, but all models degrade as sequence length increases. Techniques such as majority voting, in-context exemplars, and chain-of-thought are systematically ablated for their effects on reliability.

ReasonBENCH: Stability and Reproducibility of Reasoning

The most recent reference implementation, ReasonBENCH (Potamitis et al., 8 Dec 2025), targets a key deficit of previous evaluations: reporting only single-run accuracy and ignoring the intrinsic variability induced by stochastic decoding. ReasonBENCH introduces:

• Quality–Stability Protocol: For each (model, method, task) triple, multiple independent runs (default $N=10$) are performed; solve-rate mean, confidence interval ($\mathrm{CI}$), coefficient of variation ($\mathrm{CV}=\sigma/\bar{X}$), and mean absolute deviation ($\mathrm{MAD}$) are reported.
• Modular Evaluation Library: Standardized abstractions (Model, Agent, Environment, State, Method) enabling plug-and-play evaluation of any model, reasoning strategy, or benchmark task.
• Task Suite: Encompasses multi-step mathematical reasoning, code synthesis (HumanEval), multi-hop QA, scientific tasks, and creative generation.
• Findings: Wide instability across prevalent reasoning strategies; high-performing architectures may nonetheless suffer large confidence intervals or break down under prompt variants and cost constraints. Model scaling and prompt clarification improve stability, but not universally.

This paradigm formalizes stability as a central metric—placing explicit emphasis on variance-aware reporting, reproducible evaluation protocols, and cost–quality tradeoff analysis across tasks.

3. Taxonomy, Structure, and Best Practices in ReasonBENCH Design

Modern ReasonBENCH implementations and theoretical proposals categorize reasoning evaluation along dimension axes including:

• Reasoning Type: Commonsense, mathematical, procedural, compositional, visual-graphic, multimodal, tool-use, code synthesis.
• Task Format: Multiple-choice, open-ended QA, stepwise solution traces, multimodal (image/video) reasoning, plan execution, and state simulation.
• Granularity of Evaluation: Final answer correctness, step-wise correctness (e.g., L0-Bench, MMReason), robustness to prompt perturbation (RiddleBench), and process consistency across stochastic runs.

Recommended design principles are (Deveci et al., 3 Nov 2025, Potamitis et al., 8 Dec 2025):

• Include rich, type-labeled metadata per instance.
• Require models to produce explainable, stepwise outputs that can be compared to reference chains.
• Use adversarially or dynamically updated test construction so benchmarks remain unsolved and discriminative as models scale.
• Implement both aggregate and per-dimension scoring (accuracy, consistency, calibration, step-wise correctness, contamination risk).
• Quantify, and report, variance-aware metrics beyond mean performance (e.g., CIs, MADs, cost variance).
• Ensure evaluation protocols are multi-run, reproducible, and resistant to superficial or single-path "shortcuts."

4. Empirical Results and Model Performance

Analysis across ReasonBENCH variants consistently reveals the following trends:

• No Universal Winner: Efficiency-oriented methods, as in EffiReason-Bench, show complex trade-offs; no single paradigm or scaling regime dominates (Huang et al., 13 Nov 2025).
• Instability in Reasoning: High average performance may coincide with large variance and cost unpredictability, undermining reliability (Potamitis et al., 8 Dec 2025).
• Domain Sensitivity: Models that perform well on short, shallow, or "surface" benchmarks (e.g., MMLU, MMMU) may fail dramatically on graduate-level, stepwise, or open-ended tasks (R-Bench (Guo et al., 4 May 2025), MMReason (Yao et al., 30 Jun 2025)).
• Stepwise and Intermediate Checks: Models routinely achieve high final-answer accuracy via short-cutting, but fail to produce valid intermediate chains, especially under chain-of-thought or dynamic reasoning (MMReason, L0-Bench).

5. Recommendations, Limitations, and Future Directions

The following unified practices are emerging across ReasonBENCH and its kin:

• Evaluate stability and reproducibility as first-class metrics (variance, CIs, MAD) (Potamitis et al., 8 Dec 2025).
• Probe genuine reasoning by requiring stepwise, reference-aligned chains, not just correct answers (Sun et al., 28 Mar 2025, Yao et al., 30 Jun 2025).
• Employ multi-run protocols with cache-saving and standardized logging for efficiency and consistency.
• Address contamination and shortcut exploitation by integrating multi-model voting, adversarial filtering, and prompt diversification (Deveci et al., 3 Nov 2025, Yao et al., 30 Jun 2025).
• Extend benchmarks to new axes: partial observability, non-determinism, multi-agent/interactive reasoning, and joint plan/action synthesis (Handa et al., 6 Jun 2024).
• Anchor evaluation in formally defined constructs (e.g., fluents, computation graphs, mental models), and leverage dynamic benchmark generation to address the saturation problem.

6. Comparative Perspective and Benchmark Interoperability

ReasonBENCH-type suites are increasingly compared against traditional reasoning benchmarks for discriminative power, robustness to model advances, and extensibility:

Benchmark	Reasoning Classes	Stepwise Eval	Variance/Robustness	Multimodality	Reference Source
ActionReasoningBench	Dynamic/action, indirect effect	Yes	No	No	(Handa et al., 6 Jun 2024)
L0-Bench	Procedural, level-0 reasoning	Yes	No	No	(Sun et al., 28 Mar 2025)
ReasonBENCH	Multi-domain, stability	Yes	Yes	No	(Potamitis et al., 8 Dec 2025)
EffiReason-Bench	Efficient chain-of-thought	Yes	No	No	(Huang et al., 13 Nov 2025)
MMReason	Multimodal, multistep	Yes	No	Yes	(Yao et al., 30 Jun 2025)
R-Bench	Graduate-level, multi-disciplinary	No	No	Yes	(Guo et al., 4 May 2025)

These efforts collectively lay the groundwork for unified, continuous, and robust measurement of reasoning—across modalities, domains, and methodological paradigms—reifying the original ReasonBENCH vision. They also set explicit baselines and establish reproducibility standards likely to persist in the next generation of model evaluation protocols.