---
title: 'GSM-MC: Proactive Critical Thinking Benchmark'
url: https://www.emergentmind.com/topics/gsm-mc
type: topic
---

# GSM-MC: Proactive Critical Thinking Benchmark

GSM-MC is the paper’s first new benchmark for **proactive critical thinking** in math word problems. It is introduced as a version of GSM8K in which one key variable is deliberately removed, so the initial problem is unanswerable unless the model notices the incompleteness, asks for the missing information, and then uses the reply to complete the solution. In this formulation, the benchmark shifts evaluation away from whether a model can simply solve a math problem and toward whether it can collaborate when the problem statement is incomplete. The benchmark is presented together with GSM-MCE, a harder extension that injects irrelevant details, in order to distinguish core proactive questioning from proactive questioning under distraction [2507.23407].

## 1. Conceptual definition

GSM-MC is designed to evaluate a behavior the paper calls **proactive critical thinking**. Prior work is described as focusing mainly on **passive critical thinking**, where models reject problematic queries without taking constructive steps to resolve them. GSM-MC operationalizes a different requirement: the model should recognize that the problem cannot be answered as written, identify what is missing, and ask a clarifying question rather than guessing or refusing blindly [2507.23407].

This design makes GSM-MC distinct from conventional mathematical reasoning benchmarks. Compared with GSM8K, it is not a standard solvable math set. Each item begins as a GSM8K question, but one key variable is removed so that the initial prompt becomes incomplete. The expected behavior therefore has two stages: first, detection of incompleteness; second, recovery of solvability through interaction. The benchmark’s emphasis is not merely on final-answer production, but on the intermediate act of eliciting the necessary information from the user.

A common misconception is to treat GSM-MC as a corrupted or adversarial version of GSM8K whose purpose is to test refusal behavior. The paper explicitly frames it differently. The benchmark is intended to measure whether a model can **actively help fix the query**. In that sense, GSM-MC evaluates a collaborative competence rather than a purely defensive one.

## 2. Relation to GSM8K and GSM-MCE

The benchmark family is structured around three different task regimes. GSM-MC is the cleaner baseline for proactive questioning, whereas GSM-MCE adds distractors to test robustness against irrelevant context [2507.23407].

| Benchmark | Construction | Evaluation focus |
|---|---|---|
| GSM8K | Standard solvable math set | Traditional reasoning tasks |
| GSM-MC | One key variable removed from a GSM8K problem | Core proactive questioning |
| GSM-MCE | Missing-variable problems plus irrelevant details | Proactive questioning under distraction |

The paper states that GSM-MC contains **1,368 high-quality questions**. GSM-MCE is described as the harder extension built from the same missing-variable setup, but with **irrelevant, distracting information** added. This distinction matters because it separates failure to recognize incompleteness from failure to stay focused on the truly relevant context.

A plausible implication is that GSM-MC measures the minimal interactional capability required for repair-oriented reasoning, while GSM-MCE probes whether that capability survives under context contamination. The paper itself phrases this as GSM-MC measuring core proactive questioning and GSM-MCE measuring the same skill under distraction.

## 3. Task protocol and formal interaction pattern

The task setup is explicitly interactive. The authors first use **DeepSeek-V3** to identify key information in each GSM8K problem, marking variables with brackets in a variable-recognition step. They then **randomly remove one key variable** and ask the model to rephrase the result into a fluent but unanswerable question. At evaluation time, the model receives an instruction of the form: if the question is answerable, give the final answer; otherwise, ask the user for the necessary information as a question [2507.23407].

For unanswerable questions, a separate **user agent** is introduced. The user agent receives the original, unmodified question \( \hat{x} \) and answers the model’s clarification query \( q \) with the missing information \( a \). The model then computes the final response \( y = \pi(x, q, a) \). The paper formalizes the overall interaction as follows:

- if answerable: the model directly outputs \( y = \pi(x) \),
- if unanswerable: it first generates a clarifying question \( q = \pi(x) \),
- after the user reply \( a \), it outputs \( y = \pi(x, q, a) \).

This formulation is central to GSM-MC. The benchmark does not score a single forward pass over a static prompt; instead, it evaluates a two-turn repair process in which the model must decide whether additional information is needed and, if so, request exactly that information. The procedural emphasis is therefore on answerability recognition, clarification generation, and post-clarification solution.

## 4. Data construction and filtering pipeline

The paper emphasizes that benchmark quality depends on a strict construction pipeline rather than simple automatic perturbation. The pipeline has four stages: **(1) Variable Recognition, (2) Unanswerable Question Creation, (3) Irrelevant Information Injection, and (4) Sampling-Based Filtering**. Although GSM-MC is the cleaner baseline, the full pipeline governs the generation process for the benchmark suite [2507.23407].

For each candidate item, the authors sample **16 solutions** from DeepSeek-V3 and filter out three problematic categories: questions that remain answerable, questions with overly obvious missing information, and questions that are unclarifiable even after many attempts. The paper states that this filtering is crucial for quality and that removing it significantly hurts training and evaluation.

This filtering criterion clarifies what GSM-MC is intended to exclude. It is not meant to consist of trivially broken prompts, nor of items whose missing element is so conspicuous that proactive questioning becomes mechanical, nor of cases that cannot be repaired through a reasonable clarification turn. A plausible implication is that the benchmark aims to approximate a narrow but important band of realistic incompleteness: problems that are genuinely under-specified, yet recoverable through targeted interaction.

## 5. Empirical behavior of off-the-shelf and trained models

The reported results show a large gap between conventional reasoning competence and proactive-questioning competence. On GSM-MC, off-the-shelf models often collapse. The paper reports that **Qwen3-1.7B without thinking** gets **0.95% ACC** and **1.75% REQ**, while **Llama-3.2-3B-Instruct** gets **0.15% ACC** and **0.44% REQ**. Even stronger models such as **Qwen3-8B** show a large drop relative to GSM8K [2507.23407].

The most prominent result is the trained **Qwen3-1.7B with thinking mode**. The vanilla model scores **0.15%** on GSM-MC, but after the paper’s training method it reaches **73.98% ACC** with **97.00% REQ**. The same trained setting also lifts GSM-MCE to **41.09% ACC** and **92.95% REQ**. Across models, the trained systems improve by roughly **70% on GSM-MC** and about **40% on GSM-MCE** for smaller models, while maintaining strong performance on ordinary GSM8K.

These results support the paper’s claim that proactive questioning is not an automatic byproduct of extensive post-training or inference-time scaling. The models are described as excelling in traditional reasoning tasks, yet still struggling with proactive critical thinking, especially smaller ones. GSM-MC therefore exposes a failure mode that is not visible on standard solvable benchmarks.

## 6. Training methodology, significance, and terminological ambiguity

Reinforcement learning is central to the reported gains. The paper uses an enhanced **GRPO**-style RL method that optimizes final-answer correctness while adding heuristic rewards for asking when a question is unanswerable and penalties for unnecessary requests on answerable ones. The heuristic reward shaping is specified as \(-0.5\) for unnecessary requests on answerable questions and \(+0.5\) for requesting clarification on unanswerable questions. The RL objective uses a GRPO objective with an advantage term and a KL regularizer, and user turns are excluded from the loss. For supervised fine-tuning, the paper minimizes a cross-entropy loss over the clarification and final-answer turns, namely the two terms \( - \log p(q \mid x, \pi) \) and \( - \log p(y \mid x, q, a, \pi) \) [2507.23407].

The authors state that RL alone gives major jumps, and that combining **SFT + RL** works best for the smaller Qwen3-1.7B model. They also argue that heuristic answerability guidance improves both data efficiency and reward learning. In that framing, GSM-MC is not only a benchmark but also a training target for models that can collaborate more effectively with users in problem-solving.

The broader significance is that GSM-MC moves evaluation from “can the model solve?” to “can the model collaborate when the problem statement is incomplete?” The paper identifies settings such as medicine, tutoring, and technical support as domains where users often omit critical details. This suggests that the benchmark is intended as a proxy for interaction under under-specification rather than as a narrow mathematics-only diagnostic.

The label **GSM-MC** is also terminologically ambiguous outside this paper. In unrelated arXiv literature, the same string is not standardized: in "Global Systems Performance Analysis For Mobile Communications (GSM) using Cellular Network CODECS" [2107.00126], it can be understood as a GSM cellular-network performance analysis setup using codecs; in "Facilitating Machine to Machine (M2M) Communication using GSM Network" [1302.5384], the exact acronym does not define the proactive-questioning benchmark and instead refers to GSM-based signaling scalability for SMS-oriented M2M devices. This suggests that, in current research usage, **GSM-MC** should be interpreted by local paper context rather than by acronym alone. In the 2025 benchmark paper, its meaning is specific: a proactive-critical-thinking benchmark for incomplete mathematical word problems.

Source: https://www.emergentmind.com/topics/gsm-mc