StepSense: Metric for Multimodal Reasoning

• StepSense is a metric for reasoning validity in multimodal models, measuring the correctness of each intermediate step in a chain of thought using human annotations.
• It quantifies reasoning quality by averaging binary correctness labels, either as a macro-average per instance or a micro-average over all steps.
• StepSense serves as a diagnostic tool to expose hallucinated inferences and failure modes, distinguishing step-level accuracy from final answer correctness.

StepSense is a metric introduced in the MagiC benchmark to evaluate grounded multimodal cognition in large vision-LLMs by quantifying the correctness of intermediate reasoning steps rather than only the final answer (Wu et al., 9 Jul 2025). In MagiC, it is used to assess whether a model generates valid, coherent, and factually grounded chains of thought for question-and-image instances, and it is explicitly separated from metrics that measure final-answer correctness or visual grounding fidelity (Wu et al., 9 Jul 2025).

1. Position within MagiC

MagiC is a benchmark designed to evaluate grounded multimodal cognition, assessing not only answer accuracy but also the quality of step-by-step reasoning and its alignment with relevant visual evidence (Wu et al., 9 Jul 2025). The benchmark includes approximately 5,500 weakly supervised QA examples generated from strong model outputs and 900 human-curated examples with fine-grained annotations, including answers, rationales, and bounding box groundings. It evaluates 15 vision-LLMs ranging from 7B to 70B parameters across four dimensions: final answer correctness, reasoning validity, grounding fidelity, and self-correction ability (Wu et al., 9 Jul 2025).

Within this four-dimensional evaluation, StepSense is the metric for reasoning validity. It is intended to probe whether a model “thinks” in logically sound sub-steps, traces each sub-conclusion back to the image in a correct way, and avoids non-grounded or hallucinated inferences within its chain of thought (Wu et al., 9 Jul 2025). This suggests that StepSense is not an end-task score for visual question answering in the narrow sense; rather, it operationalizes a finer-grained notion of multimodal reasoning quality at the level of intermediate textual steps.

2. Formal definition

Let $D$ be the set of test questions, with $|D| \approx 689$ in the human-annotated split. For each question $q \in D$, the model produces a chain of $n_q$ reasoning steps,

$$S_{\mathrm{LM}}^{(q)} = [s_1^{(q)}, s_2^{(q)}, \ldots, s_{n_q}^{(q)}].$$

Human annotators assign a binary correctness label $c_i^{(q)} \in \{0,1\}$ to each step $s_i^{(q)}$, where $1$ denotes “factually and logically correct” and $0$ denotes otherwise (Wu et al., 9 Jul 2025).

StepSense can be expressed as a macro-average over questions:

$$\mathrm{StepSense}_{\mathrm{macro}} = \frac{1}{|D|}\sum_{q\in D}\left(\frac{1}{n_q}\sum_{i=1}^{n_q} c_i^{(q)}\right),$$

or as an equivalent micro-average over all steps:

$|D| \approx 689$0

In this notation, $|D| \approx 689$1 is the test-set of question-and-image instances in the human-annotated split; $|D| \approx 689$2 is the number of instances, reported as 689 in MagiC; $|D| \approx 689$3 is the number of reasoning steps generated by the model for question $|D| \approx 689$4; $|D| \approx 689$5 is the $|D| \approx 689$6-th reasoning step in the chain for question $|D| \approx 689$7; and $|D| \approx 689$8 is the human-provided correctness label for that step (Wu et al., 9 Jul 2025). StepSense$|D| \approx 689$9 is therefore the average per-instance fraction of correct steps, whereas StepSense$q \in D$0 is the fraction of all steps across all instances that are correct.

Because the authors report a single StepSense number, such as 61.38%, they use the micro-average form (Wu et al., 9 Jul 2025).

3. Annotation protocol and practical computation

The practical computation of StepSense consists of three stages. First, for each test pair of image and question, the LVLM is prompted to output a sequence of steps $q \in D$1 (Wu et al., 9 Jul 2025). Second, four expert annotators label each step $q \in D$2 as correct or incorrect, assigning $q \in D$3 or $q \in D$4 respectively; the reported inter-annotator agreement is Cohen’s $q \in D$5 (Wu et al., 9 Jul 2025). Third, the labels are aggregated by counting all $q \in D$6 across the test set and dividing by the total number of steps, yielding $q \in D$7 (Wu et al., 9 Jul 2025).

A central property of the metric is that no additional automated scoring or LLM-judging is used for StepSense. Final-answer accuracy in the paper is evaluated with an LLM judge, but StepSense is purely human-annotated (Wu et al., 9 Jul 2025). This distinction matters methodologically: StepSense is explicitly grounded in expert evaluation of each intermediate step rather than in automatic comparison of the final response alone.

The construction also separates step-level correctness from answer-level correctness. A chain may contain incorrect intermediate inferences even when the final answer is right, and a final answer may be wrong despite some correct intermediate steps. StepSense is designed to make those cases visible.

4. Motivation and conceptual scope

The stated motivation for StepSense is that many existing VQA benchmarks judge only the end answer (Wu et al., 9 Jul 2025). By contrast, StepSense quantifies the model’s ability to generate valid, coherent, and factually grounded intermediate reasoning steps—independently of whether the final answer is right or wrong.

Its intended target is therefore not merely verbal plausibility. A step is assessed for factual and logical correctness, and the metric is meant to capture whether the model traces its sub-conclusions back to the image in a correct way while avoiding non-grounded or hallucinated inferences (Wu et al., 9 Jul 2025). In this sense, StepSense addresses a common misconception in multimodal evaluation: high answer accuracy does not by itself establish grounded visual reasoning.

A plausible implication is that StepSense functions as a diagnostic instrument for chain-of-thought quality under multimodal supervision. Because it scores individual steps, it exposes failure modes that can remain latent when evaluation is restricted to the terminal answer string.

5. Relation to MagiScore and the four-dimensional evaluation

StepSense and MagiScore are complementary rather than interchangeable (Wu et al., 9 Jul 2025). MagiScore, often called “region-focus F1” in the paper, measures grounding fidelity: how well the model’s attention to bounding boxes overlaps the human-specified relevant regions. It is computed via precision, recall, and F1 on predicted region indices. StepSense, by contrast, measures reasoning validity: the semantic correctness of textual chain-of-thought steps (Wu et al., 9 Jul 2025).

The contrast can be summarized as follows.

Metric	What it measures	Operationalization
StepSense	Reasoning validity	Human labels on step correctness
MagiScore	Grounding fidelity	Precision/recall/F1 on predicted region indices

The paper characterizes the distinction with two questions: MagiScore asks, Are you looking at the right places?; StepSense asks, Are your intermediate inferences correct? (Wu et al., 9 Jul 2025). This suggests that MagiC treats visual attention and textual reasoning as separate axes of grounded multimodal cognition. A model may attend to appropriate regions yet still generate unsound inferences, or it may produce locally plausible text without reliable region-level grounding.

6. Reported results and worked example

In Table 5 of the paper, four models were human-evaluated on StepSense and final-answer accuracy (Wu et al., 9 Jul 2025). The reported models are Qwen2.5-VL 7B, InternVL 2.5 8B, Gemma 3 27B, and GPT-4o mini.

Model	StepSense (%)	Final-Answer Acc (%)
Qwen2.5-VL	61.38	69.98
InternVL 2.5	48.99	48.69
Gemma 3	52.13	55.10
GPT-4o mini	53.89	59.26

The paper also gives a concrete calculation for Qwen2.5-VL 7B. Across all 689 instances, the total number of reasoning steps is approximately $q \in D$8 (Wu et al., 9 Jul 2025). The number labeled correct is approximately $q \in D$9 steps, and the number labeled incorrect is approximately $n_q$0 steps. Hence,

$n_q$1

The paper’s interpretation is that, on average, about 6 out of every 10 reasoning steps by Qwen2.5-VL were judged sound by human annotators (Wu et al., 9 Jul 2025). It further states that higher StepSense is strongly correlated with higher final-answer accuracy but also independently highlights where models still hallucinate or reason incorrectly. This indicates that StepSense is both an evaluative metric and a diagnostic signal for model failure modes in grounded visual reasoning.