GSM8K-Hi: Enhanced Math Reasoning Benchmark
- GSM8K-Hi redefines grade-school math evaluation by integrating synthetic executable supervision, meta-reasoning, and visual grounding to capture nuanced reasoning steps.
- It reveals that high answer accuracy can mask shallow process understanding, as shown by GPT-4’s MR-Score of 0.495 despite near-saturated text performance.
- TinyGSM demonstrates that careful verifier-based candidate selection can enable a 2.6B model to outperform larger models on GSM8K-family tasks.
GSM8K-Hi is not explicitly defined as a standardized benchmark name in the cited literature. A plausible interpretation is that it denotes a GSM8K-family regime aimed at higher-fidelity, more discriminative, or higher-performing evaluation and modeling for grade-school mathematical reasoning. Under that interpretation, the relevant research converges on three complementary directions: improving system accuracy on standard GSM8K through synthetic executable supervision and verifier-based selection, reformulating GSM8K into a meta-reasoning task that evaluates reasoning traces rather than only final answers, and preserving the same GSM8K problem distribution while shifting the input modality to visually grounded multi-image scenes (Liu et al., 2023, Zeng et al., 2023, Yuan et al., 29 Sep 2025).
1. Scope and relation to the GSM8K family
GSM8K is a benchmark of about 8.8K grade-school math word problems, with roughly a 7k/1k train-test split, and problems requiring 2 to 11 reasoning steps (Liu et al., 2023). It has become a standard benchmark for mathematical reasoning in natural language, and recent work repeatedly treats it as close to saturated for many strong models (Zeng et al., 2023, Yuan et al., 29 Sep 2025).
Within that context, GSM8K-Hi is best understood as an inferred label for work that tries to recover discriminative power once standard answer-accuracy evaluation becomes compressed. The literature suggests at least three routes. One route is system-centric: push accuracy higher on GSM8K itself, even with small models, through targeted supervision and search-based selection. A second route is task-centric: replace answer generation with evaluator-style judgment of candidate reasoning traces. A third route is modality-centric: keep the underlying GSM8K items but render them as visual multi-image stories so that perception, grounding, and cross-scene integration become part of the task.
This suggests that “high” in GSM8K-Hi need not mean only harder arithmetic. It can also mean higher diagnostic resolution, higher-fidelity process evaluation, or higher robustness under modality shift.
2. Why standard GSM8K is no longer sufficient
A central claim across the literature is that standard GSM8K-style evaluation is too result-oriented. In ordinary GSM8K evaluation, a model is given a math word problem and is scored mainly on whether it produces the correct final answer. Even when chain-of-thought is elicited, the benchmark metric is still usually final-answer correctness. The argument against relying exclusively on that metric is twofold: high answer accuracy may conceal shallow process understanding, and benchmark saturation may compress real capability differences (Zeng et al., 2023).
The empirical evidence is direct. On standard GSM8K, GPT-4, Claude2, and GPT-3.5 score 92.0%, 88.0%, and 80.8% respectively, but on MR-GSM8K their MR-Scores are 0.495, 0.191, and 0.097, revealing much larger separations than answer accuracy alone suggests (Zeng et al., 2023). A similar pattern appears under modality shift: Gemini-2.5-Pro achieves 95.22% on text-only GSM8K but only 46.93% on GSM8K-V, while human accuracy on GSM8K-V is 91.15% (Yuan et al., 29 Sep 2025).
| Regime | Primary task | Main signal |
|---|---|---|
| GSM8K | Generate final numeric answer | High text accuracy can compress differences |
| MR-GSM8K | Judge a candidate solution trace | Reveals large gaps in reasoning evaluation |
| GSM8K-V | Solve GSM8K visually across scenes | Reveals failures in grounding and perception |
A common misconception is that a higher GSM8K answer score is a sufficient proxy for stronger reasoning in general. The cited results do not support that view. They instead indicate that answer generation, evaluator-style reasoning, and grounded multimodal reasoning are overlapping but non-interchangeable capability measures.
3. Meta-reasoning as a higher-fidelity reformulation
MR-GSM8K introduces a teacher-like evaluation paradigm in which the model no longer merely solves a GSM8K problem. Instead, it receives a question paired with a candidate solution trace and must reason about that reasoning. The required output has three parts: predict whether the provided solution is correct or incorrect; if incorrect, identify the first error step; and if incorrect, explain the reason for that first error (Zeng et al., 2023).
The benchmark is built from GSM8K in three forms: original GSM8K problems with generated step-by-step solutions, Program-of-Thought variants requiring code-style reasoning, and reverse or backward reasoning variants inspired by MetaMath-style backward transformations. Candidate solution traces are generated with MetaMath-7B using temperature . The final benchmark contains 3000 question-solution pairs: 1418 Original, 222 POT, and 1360 Reverse. At the correctness level it is relatively balanced, with 1427 correct and 1573 incorrect solutions (Zeng et al., 2023).
The evaluation is process-sensitive. Solutions are segmented into steps by newline boundaries. For annotation, each step is treated as positive or neutral if the correct final answer remains attainable, and negative if the reasoning has diverged from a valid path; the first error step is the initial negative step. Error reason is not categorized into a fixed ontology such as arithmetic or unit errors. Instead, it is a free-form explanation judged by semantic agreement with human annotation, either manually or by GPT-4 (Zeng et al., 2023).
The overall metric is the composite MR-Score:
$MR\mbox{-}Score = 0.2 \cdot \max(0, MCC) + 0.3 \cdot ACC_{\text{step}} + 0.5 \cdot ACC_{\text{reason}}.$
Here, correctness classification uses Matthews Correlation Coefficient, first-error-step accuracy is evaluated over incorrect solutions, and error-reason accuracy counts cases in which the model both identifies the correct first error step and provides the correct reason (Zeng et al., 2023).
The reported results make the benchmark’s diagnostic intent explicit. GPT-4 reaches , , , and $MR\mbox{-}Score = 0.495$. Claude2 reaches 0.191 MR-Score, and GPT-3.5 reaches 0.097. Open-source math-specialized 70B models collapse much more severely: WizardMath-70B scores 0.001, Mammoth-70B 0.012, and MetaMath-70B 0.013, despite reported GSM8K accuracies of 81.6%, 76.7%, and 82.3% respectively (Zeng et al., 2023).
These results motivate a specific interpretation of GSM8K-Hi. If the objective is to make GSM8K-family evaluation more discriminative, MR-GSM8K indicates that changing the task itself can separate models more strongly than increasing answer-generation difficulty alone. It is therefore not simply a harder GSM8K in the ordinary sense; it is a meta-evaluative GSM8K.
4. Visual grounding as a modality-controlled extension
GSM8K-V takes a different route. Instead of changing the output task, it preserves the underlying GSM8K problems while converting each sample into a purely visual multi-image mathematical reasoning problem. The benchmark is motivated by three gaps in prior visual math evaluation: dominance of geometry-style tasks, lack of broad math word problems in visualized real-world contexts, and limited testing of reasoning across multiple images with semantic dependencies distributed over scenes (Yuan et al., 29 Sep 2025).
Its construction pipeline is structured. Each GSM8K sample is first decomposed into triples
then assigned to one of 13 mathematical information classes and 6 higher-level benchmark categories. Facts are allocated to scenes under three explicit principles: contextual grouping, final isolation, and atomic fidelity. Each scene contains only 1–2 pieces of information, and the final scene contains the question with no extra numerical values or objects. Controlled interference is deliberately added through visually salient but irrelevant objects and contextually similar but mathematically irrelevant distractors (Yuan et al., 29 Sep 2025).
The final benchmark contains 1,319 samples and 5,343 total images, for an average of 4.05 images per problem, with a minimum of 2 and a maximum of 11. Images are rendered with GPT-Image-1 at resolution, and every item undergoes dual human cross-check under the “3C” principles: Consistency, Completeness, and Compliance. The annotator pool consists of seven annotators, all with mathematics and computer science backgrounds (Yuan et al., 29 Sep 2025).
The core empirical result is the modality gap. Among closed-source VLMs, Gemini-2.5-Pro is best at 46.93% on GSM8K-V and 95.22% on GSM8K; GPT-5 gets 30.10% visual and 95.00% text; GPT-4o gets 29.57% visual and 94.92% text. Among open-source models, Llama-4-17B-128E-Instruct is strongest at 31.69% visual and 95.30% text. Human accuracy is 91.15% on GSM8K-V (Yuan et al., 29 Sep 2025).
The paper’s modality-validity analysis is particularly relevant to any GSM8K-Hi conception that values grounded reasoning. For GPT-4o, text-only accuracy is 94.92%, image-only 29.57%, OCR 11.98%, and caption 55.11%. The caption gains show that once scenes are converted back into structured text, much of the difficulty disappears, but not all. This supports the claim that the benchmark probes visual grounding, cross-scene integration, and semantic binding rather than only arithmetic (Yuan et al., 29 Sep 2025).
A second misconception is that a GSM8K-family benchmark becomes more informative only by making the symbolic mathematics more advanced. GSM8K-V contradicts that assumption: it preserves the grade-school distribution yet becomes substantially harder because the reasoning must survive visual grounding, distractors, and multi-image composition.
5. High-performance system design through TinyGSM
TinyGSM addresses a different question: how far small LLMs can be pushed on GSM8K when supervision and inference are designed carefully. The paper introduces a synthetic dataset of 12.3M grade-school math problems paired with Python solutions, totaling about 1.8B tokens, generated fully by GPT-3.5-turbo from the GSM8K training set and GSM-IC variants. About one third of TinyGSM consists of GSM-IC-style augmentation with irrelevant context (Liu et al., 2023).
The system has two components. A Phi-family generation model is fine-tuned on TinyGSM to produce Python programs that solve the question step by step. A separate verifier model, initialized from the same base family, is trained on GSM8K training questions and sampled candidate solutions labeled by whether executing the generated Python yields the gold answer. At inference, the generator samples many candidates, the verifier scores them, and the highest-scoring program is executed to obtain the final answer (Liu et al., 2023):
The headline result is that a 1.3B generator plus a 1.3B verifier achieves 81.5% on GSM8K, denoted verify48@1 in code format. The corresponding 1.3B generator alone achieves 68.2% pass@1, but 94% pass@32. The verifier therefore recovers latent capability that is already present in the proposal distribution but not reliably surfaced by the first sample (Liu et al., 2023).
The scale-efficiency result is central. The paper states that 34B was the smallest size previously reported to break 80% on GSM8K, whereas the TinyGSM system crosses that threshold with 2.6B total parameters split across generator and verifier. It also surpasses the reported GPT-3.5-0613 GSM8K score of 77.4%, despite using GPT-3.5-generated synthetic supervision (Liu et al., 2023).
Several technical lessons follow. First, synthetic data quality matters: the 1.3B Phi-1.5 base model improves from 44.6% to 68.2% after TinyGSM fine-tuning. Second, executable program supervision is effective for arithmetic-heavy GSM8K-style reasoning. Third, verifier scaling is unusually valuable: the cross-scale experiments indicate that scaling the verifier is more effective than scaling the generator. Fourth, verifier training benefits from diversity in sampled generations, including multiple checkpoints and temperatures, rather than only polished near-greedy outputs (Liu et al., 2023).
For GSM8K-Hi as an inferred high-performance regime, TinyGSM supplies the system-design counterpart to the benchmark-design innovations of MR-GSM8K and GSM8K-V. It shows that selection and supervision quality can matter as much as, or more than, sheer model size.
6. Measurement implications, failure modes, and limitations
Taken together, the cited work suggests that GSM8K-Hi should not be equated with a single benchmark or a single scalar metric. The relevant measurements span exact-answer accuracy, composite process-evaluation scores, modality-controlled stress tests, and verifier-reranked executable reasoning. This suggests that “higher” performance in the GSM8K family is multidimensional rather than reducible to pass@1 on text-only answer generation.
The failure analyses are correspondingly heterogeneous. MR-GSM8K exposes sycophancy or over-acceptance of provided solutions, over-rejection by stronger evaluators, inability to localize the first error, superficial mastery of training solution paths, lack of ontological or conceptual grounding, and benchmark-overfitting behavior such as reverting to the standard GSM8K answer format instead of following evaluator instructions (Zeng et al., 2023). GSM8K-V identifies perception-calculation errors and instrument-reading errors, with examples involving miscounted objects, incorrect quantity-entity binding, misread pie charts, misread gauges, and confusion about cross-scene temporal order (Yuan et al., 29 Sep 2025). TinyGSM, by contrast, shows that high answer accuracy can be obtained even with small models, but also acknowledges noisy synthetic labels, imperfect verifier token-level interpretability, remaining contamination concerns, and the fact that code format is not universally best, especially for some equation-solving questions (Liu et al., 2023).
Several interpretive cautions follow. MR-GSM8K is derived from GSM8K and built only from incorrect solutions generated by MetaMath-7B, so it addresses saturation by task reformulation rather than by moving to a wholly external source distribution (Zeng et al., 2023). GSM8K-V uses synthetic comic-style images produced by a fixed rendering pipeline, so construction artifacts may remain despite human verification, even though the human score of 91.15% indicates preserved solvability (Yuan et al., 29 Sep 2025). TinyGSM’s student surpasses its GPT-3.5 teacher only because it is augmented with a verifier trained on real GSM8K labels; it is therefore not pure distillation from synthetic data alone (Liu et al., 2023).
A third misconception is that gains in one GSM8K-family regime automatically transfer to another. The evidence argues against that simplification. The fine-tuned Llama-2 diagnostic model in MR-GSM8K exceeds GPT-3.5’s MR-Score, 0.105 versus 0.097, while still trailing GPT-3.5 on standard GSM8K, 74.3% versus 80.8% (Zeng et al., 2023). Likewise, GSM8K-V shows that near-saturated text performance does not imply strong visually grounded reasoning (Yuan et al., 29 Sep 2025).
In that sense, GSM8K-Hi is best treated not as a single benchmark name but as an emerging research orientation around stronger GSM8K-family evaluation and stronger GSM8K-family systems. The literature indicates that this orientation includes at least three non-equivalent objectives: higher answer accuracy on the original benchmark, higher-fidelity evaluation of reasoning processes, and higher robustness when the same mathematical content is embedded in grounded multimodal contexts.
