ReasonBench: A Benchmark Family

• ReasonBench is a family of benchmarks that assess non-trivial inference across vision-language, text-to-image, and LLM reasoning tasks.
• It employs domain-specific methodologies—like structured graphic tasks, two-stage evaluations, and stochastic decoding—to capture intermediate reasoning processes.
• The benchmarks drive model improvements by leveraging targeted strategies such as diagrammatic reasoning chains, reward designs, and prompt optimizations for enhanced stability.

Searching arXiv for papers using “ReasonBench” and closely related benchmark names. I’m checking the current arXiv record for “ReasonBench” and its major variants to ground the article in the latest benchmark papers. ReasonBench is not a single uniformly defined benchmark in recent arXiv literature, but a family of reasoning-centered evaluation programs that share a common objective: to measure whether a model can carry out non-trivial inference rather than merely match surface patterns. The name appears in at least three prominent forms: ReasonBench for complex graphic reasoning in vision–LLMs, T2I-ReasonBench for reasoning-informed text-to-image generation, and ReasonBENCH for the instability of LLM reasoning under stochastic decoding. Across these usages, the term has come to denote benchmarks that emphasize compositional structure, intermediate constraints, or variance-aware evaluation over answer-only scoring (Zhang et al., 1 Aug 2025, Sun et al., 24 Aug 2025, Potamitis et al., 8 Dec 2025).

1. Terminological scope and benchmark family

In current usage, “ReasonBench” functions as a benchmark label across multiple modalities rather than as a single canonical dataset. One line uses the exact name ReasonBench for structured graphic reasoning in real-world intelligence-test style problems; another defines T2I-ReasonBench for reasoning-informed image generation; a third introduces ReasonBENCH as a multi-run benchmark for the reproducibility and cost stability of LLM reasoning. The shared naming is substantive rather than accidental: each benchmark makes reasoning itself the evaluation target, whether through graphic abstraction, implicit prompt interpretation, or statistical instability across repeated runs (Zhang et al., 1 Aug 2025, Sun et al., 24 Aug 2025, Potamitis et al., 8 Dec 2025).

Benchmark name	Primary target	Distinctive focus
ReasonBench (Zhang et al., 1 Aug 2025)	Vision–LLMs	1,613 structured graphic reasoning questions from real-world intelligence tests
T2I-ReasonBench (Sun et al., 24 Aug 2025)	Text-to-image generation	800 prompts across four reasoning dimensions with two-stage evaluation
ReasonBENCH (Potamitis et al., 8 Dec 2025)	LLM reasoning systems	10-run protocol for quality and cost instability

This multiplicity matters because later papers explicitly treat “ReasonBench” as a design pattern or reference point when introducing new reasoning benchmarks or reasoning-aware training methods. In that sense, the term has acquired both a specific and a generic meaning: it names several concrete artifacts while also indexing a broader methodological movement toward benchmark designs that expose hidden reasoning failure modes.

2. ReasonBench for structured graphic reasoning

The benchmark titled ReasonBench in “Oedipus and the Sphinx: Benchmarking and Improving Visual LLMs for Complex Graphic Reasoning” is the first evaluation benchmark focused on structured graphic reasoning tasks. It contains 1,613 questions drawn from real-world intelligence tests, covers 11 cognitive dimensions and 29 task types, and reports a human baseline of 69.76%. The questions come from Chinese Civil Service Aptitude Tests, Mensa IQ tests, and Raven’s Progressive Matrices, and are evaluated in both integrated and separated visual input formats using multiple-choice answers and Pass@1 accuracy. Across 11 mainstream VLMs, the best reported integrated-format result is 27.22% from Gemini-2.0, which is only slightly above random-choice behavior for a four-option setting and far below human performance (Zhang et al., 1 Aug 2025).

The benchmark’s internal taxonomy spans positional, stylistic, attribute, quantitative, spatial, alphanumeric, and other IQ-style graphic transformations. The task inventory includes translation, rotation, symmetry, open/closed states, counting lines and points, cube and polyhedron reasoning, three-view and sectional-view reasoning, black–white block patterns, Mensa items, and Raven-style matrix completion. Its central claim is that prior visual reasoning benchmarks emphasize simpler graphics, whereas these tasks require multiple interacting elements and abstract rule composition.

The same paper also proposes a dual optimization strategy. Diagrammatic Reasoning Chain (DiaCoT) decomposes multi-element diagrams into layered descriptions and intermediate rules, improving interpretability and substantially raising performance on a 200-question validation set. ReasonTune fine-tunes a smaller model on benchmark-style data. For Qwen-7B, the combination of DiaCoT and ReasonTune yields a 33.5 percentage point improvement on that validation set, showing that prompt-structured decomposition and task-specific adaptation can materially improve complex graphic reasoning without altering the underlying benchmark definition (Zhang et al., 1 Aug 2025).

3. T2I-ReasonBench and reasoning-informed image generation

T2I-ReasonBench defines reasoning for text-to-image generation as the ability to infer latent visual content that is not explicitly enumerated in the prompt. It consists of 800 prompts across four dimensions—Idiom Interpretation, Textual Image Design, Entity-Reasoning, and Scientific-Reasoning—with 200 prompts in each category. Its evaluation protocol is explicitly two-stage: DeepSeek-R1 first generates prompt-specific question–criterion pairs, and Qwen2.5-VL then scores generated images for Reasoning Accuracy and Image Quality. In the original benchmarking study, GPT-Image-1 achieves the strongest overall results, with 0.787 reasoning accuracy and 0.958 image quality, while most open models remain far lower, especially on idiomatic and scientifically constrained prompts (Sun et al., 24 Aug 2025).

Because T2I-ReasonBench concentrates on implicit meaning rather than literal prompt following, it quickly became a standard evaluation target for reasoning-aware T2I methods. In SRUM, the benchmark is used to test whether a unified multimodal model can transfer internal understanding ability to image generation through self-rewarding. On Bagel, the overall T2I-ReasonBench score improves from 43.82 to 46.75, with category scores of 52.85 for Entity, 40.51 for Idiom, 47.83 for Scientific, and 45.83 for Textual, supporting the claim that a global-local reward design can improve reasoning-heavy generation (Jin et al., 14 Oct 2025).

A second line, StruVis, treats T2I-ReasonBench as a benchmark for “reasoning-based text-to-image generation.” It uses text-based structured visual representations rather than intermediate image generation as reasoning states, and reports a 4.61% gain on T2I-ReasonBench for Qwen3-VL-8B, improving accuracy from 68.97 to 73.57 over the best interleaved baseline. Category gains are largest in Idiom and Entity, indicating that explicit object–relation–layout planning is particularly helpful where prompt semantics are indirect or structurally constrained (Lyu et al., 6 Mar 2026).

A third line, Generation Navigator, reframes T2I as a multi-turn state-conditioned control problem and uses T2I-ReasonBench as its primary reasoning benchmark. With PRE-GRPO, it reaches 79.06% overall reasoning accuracy, with 74.96 on Idiom, 89.71 on Textual, 73.90 on Entity, and 77.66 on Scientific. This positions T2I-ReasonBench not merely as a static test set, but as a training-time and trajectory-level optimization target for agentic T2I systems (Liu et al., 18 May 2026).

4. ReasonBENCH and variance-aware evaluation of LLM reasoning

ReasonBENCH: Benchmarking the (In)Stability of LLM Reasoning shifts the meaning of the name from capability measurement to reliability measurement. Its object is not a single task family but the variability of reasoning outcomes under stochastic decoding. The benchmark provides a modular evaluation library, a public leaderboard, and a protocol that runs 10 independent trials for each model–method–task combination. It standardizes 11 reasoning strategies across 5+ tasks and reports not only mean quality but also confidence intervals, coefficient of variation, median absolute deviation, and cost statistics, thereby treating instability as a first-class property of reasoning systems rather than a nuisance term (Potamitis et al., 8 Dec 2025).

Its empirical claim is that high average performance can mask severe instability. Across methods and tasks, confidence intervals can differ by as much as 4× even for strategies with similar mean accuracy, and top-performing methods often incur higher and less stable costs. The tasks span mathematical reasoning, code generation, multi-hop question answering, scientific reasoning, and constrained creative generation, including Game of 24, MathArena, HumanEval, HotpotQA, Humanity’s Last Exam, SciBench, and Shakespearean sonnet writing. In framework-level comparisons under GPT-4.1, FoA achieves 36.0 ± 1.4 with low variance, whereas methods such as GoT and some ToT variants exhibit substantially weaker stability profiles despite similar or higher computational cost.

ReasonBENCH also shows that prompt and parser quality materially alter both average performance and dispersion. After prompt refinements, several methods improve dramatically—for example, IO rises from 3.0 ± 0.8 to 31.3 ± 0.7, CoT from 8.0 ± 1.6 to 39.8 ± 1.4, and FoA from 36.0 ± 1.4 to 54.6 ± 1.3—which makes prompt and parsing stability part of the benchmarked phenomenon rather than a mere implementation detail. In this usage, ReasonBENCH extends the scope of reasoning evaluation from correctness to reproducibility, calibration, and cost consistency (Potamitis et al., 8 Dec 2025).

5. Expansion into multimodal and domain-specific descendants

The vocabulary and methodology of ReasonBench have already propagated into a broader ecosystem of reasoning benchmarks. In video generation, V-ReasonBench defines 326 reasoning instances and 13 tasks across structured problem-solving, spatial cognition, pattern-based inference, and physical dynamics, and evaluates models with last-frame pass@5; among six video generators, Sora-2 reaches an average of 43.86, while Hailuo-02 reaches 37.52 (Luo et al., 20 Nov 2025). In multimodal long-chain reasoning, MMReason provides 1,384 open-ended multimodal questions after multi-model filtering, and reports that GPT-4o attains only 25.7% final-answer accuracy and 42.1% intermediate-step reasoning score, underscoring the distance between answer correctness and process quality (Yao et al., 30 Jun 2025). In graduate-level general reasoning, R-Bench spans 1,094 text questions across 108 subjects and 665 multimodal questions across 83 subjects, with OpenAI o1 reaching 69.0% on the text benchmark and 53.2% on the multimodal benchmark (Guo et al., 4 May 2025).

Process-aware and domain-specific variants further extend the template. ReasoningMath-Plus contains 150 structural mathematics problems, adds HCRS and a PRM, and shows that answer-only accuracy can overestimate reasoning robustness (Zheng et al., 31 Jan 2026). LongReasonArena scales algorithmic reasoning difficulty by execution length, with 262, 306, and 288 problems across three levels and DeepSeek-R1 reaching only 7.5% at the hardest level (Ding et al., 26 Aug 2025). RE2-Bench builds 1,101 code-reasoning problems, including 195 from mature real-world projects, and organizes them into well-separated Easy and Hard subsets using nine complexity metrics (Liu et al., 16 Dec 2025).

Other descendants adapt the same reasoning-centered logic to applied domains. TFRBench spans ten datasets across five domains and evaluates both forecasting accuracy and reasoning quality, reporting that prompting LLMs with generated traces can raise success rates from roughly 40.2% to 56.6% (Ahamed et al., 7 Apr 2026). ReasonTabQA contributes 1,932 tables and 5,523 questions for industrial table reasoning, with explicit reasoning chains and the RLVR method TabCodeRL (Pan et al., 12 Jan 2026). A transparent law reasoning benchmark defines a tree-organized structure over 453 legal cases with 2,627 facta probanda, 14,578 evidence pieces, and 16,414 experiences, making legal reasoning inspectable rather than verdict-only (Shen et al., 2 Mar 2025). Taken together, these works show that “ReasonBench” has become shorthand for a class of benchmarks that encode reasoning as structure, trajectory, or process rather than as a single final label.

6. Methodological themes, limitations, and significance

Across its variants, ReasonBench is marked by a recurring methodological shift: reasoning is increasingly evaluated through intermediate structure. T2I-ReasonBench uses prompt-dependent question–criterion sets rather than a single similarity score; MMReason annotates step-by-step solutions and uses ternary step scoring; ReasoningMath-Plus introduces hazard-aware process scoring; the law benchmark explicitly separates factum probandum, evidence, and experience in a tree-organized representation (Sun et al., 24 Aug 2025, Yao et al., 30 Jun 2025, Zheng et al., 31 Jan 2026, Shen et al., 2 Mar 2025). This suggests a common principle: benchmark designers increasingly treat reasoning as an observable sequence of commitments or constraints, not merely as latent competence inferred from final answers.

A second shared theme is difficulty calibration. Graphic ReasonBench uses real-world intelligence tests and a human baseline of 69.76%; R-Bench filters items using o1 reasoning-token thresholds above 2,000; LongReasonArena indexes difficulty by executed lines from $10^2$ up to $10^6$; RE2-Bench separates Easy and Hard problems by majority voting over nine interpretable complexity metrics (Zhang et al., 1 Aug 2025, Guo et al., 4 May 2025, Ding et al., 26 Aug 2025, Liu et al., 16 Dec 2025). The broader implication is that future reasoning evaluation will likely depend less on static benchmark size and more on principled control of structural hardness, ambiguity, and shortcut resistance.

A third theme is the reliance on LLM/VLM judges and benchmark-internal evaluators. T2I-ReasonBench depends on DeepSeek-R1 and Qwen2.5-VL, MMReason uses GPT-4o for answer and step grading, and TFRBench uses Gemini-3-Pro as an LLM judge (Sun et al., 24 Aug 2025, Yao et al., 30 Jun 2025, Ahamed et al., 7 Apr 2026). ReasonBENCH, by contrast, emphasizes repeated execution and statistical reporting to reduce overconfidence in single-run numbers (Potamitis et al., 8 Dec 2025). A plausible implication is that the future of ReasonBench-style evaluation will involve both richer structured supervision and more explicit uncertainty accounting, because richer judges alone do not eliminate evaluator bias, variance, or benchmark saturation.

In that broader sense, ReasonBench names both a set of individual datasets and a research program. Its central contribution is not any one question bank or metric, but the redefinition of reasoning evaluation as a problem of structure, process, calibration, and transparency across language, vision, generation, code, forecasting, law, and multimodal reasoning.