ECG-Reasoning Benchmark

• ECG-Reasoning Benchmark is a multi-turn evaluation framework that defines clinical deduction in ECG interpretation by verifying each reasoning step rather than relying on pattern matching.
• It employs a structured four-step protocol—criterion selection, finding identification, ECG grounding, and diagnostic decision—to mirror the sequential logic of clinical diagnosis.
• Constructed from PTB-XL and MIMIC-IV-ECG datasets with over 6,400 curated samples, the benchmark exposes limitations in models that rely on superficial visual cues instead of signal-grounded reasoning.

Searching arXiv for the benchmark paper and closely related ECG reasoning benchmark work. {"query":"arXiv (Oh et al., 15 Mar 2026) ECG-Reasoning-Benchmark A Benchmark for Evaluating Clinical Reasoning Capabilities in ECG Interpretation", "max_results": 5} ECG-Reasoning-Benchmark is a multi-turn evaluation framework for electrocardiogram interpretation that tests whether multimodal LLMs can carry out step-by-step clinical deduction rather than relying on superficial visual cues or label-level pattern matching. It was introduced to assess the full interpretive chain of ECG reading—criterion retrieval, finding verification, waveform grounding, measurement, and final diagnostic decision—under a stricter protocol than conventional ECG question answering or single-turn classification. The benchmark comprises over 6,400 samples spanning 17 core ECG diagnoses, is constructed from PTB-XL and MIMIC-IV-ECG, and is positioned as a structured clinical exam for ECG-LLMs rather than a conventional classification leaderboard (Oh et al., 15 Mar 2026).

1. Concept and evaluative objective

The benchmark is motivated by a specific failure mode in contemporary ECG MLLMs: a model may generate a fluent interpretation, or even retrieve the correct textbook diagnostic criterion, without grounding that reasoning in the actual waveform. ECG-Reasoning-Benchmark therefore shifts evaluation away from whether an answer merely “looks plausible” and toward whether a model can execute the same staged deduction a clinician would use in practice. In this formulation, ECG interpretation is not treated as direct label prediction, but as a multi-stage process involving identification of relevant criteria, verification of their presence, localization of evidence in leads and waveform segments, quantitative measurement, and only then diagnostic confirmation (Oh et al., 15 Mar 2026).

This design addresses a broader concern in ECG AI evaluation. Earlier benchmark work on PTB-XL emphasized the need for well-defined evaluation procedures and showed that aggregate performance can obscure clinically meaningful failure modes such as hidden stratification (Strodthoff et al., 2020). More recent ECG reasoning systems, including ECG question-answering and grounded multimodal interpretation frameworks, have improved generative capability and explanation quality, but they do not by themselves establish that a model’s intermediate logic is faithful to the signal (Pham et al., 7 May 2025). ECG-Reasoning-Benchmark is specifically built to expose that gap.

A common misconception is that accurate diagnosis or polished rationale implies valid clinical reasoning. The benchmark is premised on the opposite possibility: a model can know the right diagnosis criteria yet fail to verify them in the ECG. This suggests that final-answer accuracy and reasoning fidelity should be treated as distinct quantities.

2. Dataset construction and diagnostic scope

Construction begins with an automated ECG analysis pipeline applied to PTB-XL and MIMIC-IV-ECG. The pipeline performs wave delineation using a U-Net3+ segmentation model, followed by post-processing rules that recover difficult cases such as non-conducted P waves, enforce physiological constraints, and require multi-lead consensus. From these delineations, the system computes quantitative features including P/QRS/T durations, PR/RR/QT intervals, amplitudes, ST deviation, QRS morphology, pathological Q waves, and frontal axis. Those measurements are mapped to discrete clinical findings and then to final diagnoses using hierarchical logic diagrams codified from authoritative sources such as the ECG Core Curriculum and validated by three board-certified internal medicine specialists (Oh et al., 15 Mar 2026).

The benchmark covers 17 core diagnoses organized into six clinical groups.

Clinical group	Diagnoses
AV blocks	first-, second-, third-degree AV block
Conduction disturbances	CLBBB, CRBBB, LAFB, LPFB
Hypertrophies	LVH, RVH
Ectopic beats	PAC, PVC
Myocardial infarction	anterior, inferior, lateral MI
Ischemia	anterior, inferior, lateral ischemia

The paper reports over 6,400 samples, drawn from 3,076 PTB-XL cases and 3,355 MIMIC-IV-ECG cases. The appendix further reports 2,868 unique ECGs from PTB-XL and 3,316 from MIMIC-IV-ECG, with 24,097 total QA pairs for PTB-XL and 27,025 for MIMIC-IV-ECG. The average reasoning length is around 8 turns per case. Samples are balanced across diagnoses and across the different valid reasoning paths that can lead to a diagnosis, and the dataset is filtered to retain only cases where the human label agrees with the pipeline-derived diagnosis (Oh et al., 15 Mar 2026).

The quality-control procedure is unusually explicit for an ECG-language benchmark. Three specialists validate 143 representative samples, and all authors manually review the reasoning paths under specialist supervision. This suggests that the benchmark is not merely auto-generated QA from labels, but a curated reasoning dataset with explicit clinical logic.

3. Multi-turn reasoning protocol

Each benchmark instance begins with an Initial Diagnostic Question, such as whether an ECG suggests first-degree AV block. That initial prompt is only the entry point. The core evaluation is a repeated four-step verification loop applied to each finding required by the target diagnosis (Oh et al., 15 Mar 2026).

Criterion Selection requires the model to choose which clinical criterion should be checked next. This is framed as a multiple-choice discrimination task with two types of distractors: category-based distractors from the same clinical family, and presence-based distractors that correspond to findings genuinely present in the ECG but irrelevant to the diagnosis.

Finding Identification requires the model to determine whether the selected criterion is actually present in the current ECG.

ECG Grounding verifies whether the identified finding is anchored to the signal. This stage may include three sub-questions: lead grounding, wave grounding, and measurement grounding. Lead grounding asks which leads show the abnormality. Wave grounding asks in which temporal segment or waveform the finding appears. Measurement grounding asks which value range a measured feature falls into.

Diagnostic Decision asks whether the currently verified evidence is sufficient to confirm the diagnosis or whether additional findings must still be checked.

For diagnoses requiring multiple findings, the four-step cycle repeats sequentially. The final diagnosis is only reached after all required findings have been traversed. This protocol is central to the benchmark’s semantics: it does not simply test whether a model can name a pathology, but whether it can maintain a coherent diagnostic trajectory across multiple logically coupled subdecisions. A plausible implication is that the benchmark operationalizes ECG interpretation as controlled traversal of an expert-authored logic graph rather than as free-form explanation.

4. Metrics and formal evaluation semantics

The evaluation protocol is conversational and multi-turn, with Gemini-3-Flash used to judge semantic consistency between model answers and ground truth at each step. The principal metrics are Initial Diagnosis Accuracy (IDA), Completion, Depth, and GT-Reasoning-Based Diagnosis Accuracy (GT-RDA) (Oh et al., 15 Mar 2026).

Initial Diagnosis Accuracy (IDA) measures correctness on the first diagnostic question alone, before stepwise verification begins.

Completion is the strictest metric. It is the percentage of cases for which the model answers every required reasoning question correctly across the full chain. Evaluation terminates on the first incorrect response, so Completion functions as a perfect-chain metric.

Depth measures how far a model progresses through the reasoning loop on average. Step 3 is fractionally scored when multiple grounding subtasks are present. If a finding requires lead, wave, and measurement grounding, each contributes a fraction of the Step-3 score. The appendix provides a CLBBB example: a model that answers Step 1 and Step 2 correctly and one of two Step-3 grounding subtasks correctly obtains depth 2.5 for that loop; full completion yields 4.0; success only on Step 1 yields 1.0; success on Steps 1 and 2 but failure on Step 3 yields 2.0.

GT-RDA measures final diagnostic accuracy when the model is supplied with the perfect ground-truth reasoning trajectory up to the last step. This isolates whether the model can exploit correct intermediate evidence, rather than having to generate it.

These metrics separate several behaviors that conventional ECG-QA collapses together. A model may have reasonable IDA but poor Completion, indicating early pattern recognition without sustained evidence verification. A model may also have high GT-RDA but low Completion, implying that it can use correct reasoning when given, but cannot autonomously produce it.

5. Empirical findings and identified failure modes

The benchmark’s central empirical result is that true multi-step ECG reasoning remains extremely weak. Across models and across both source datasets, Completion is near zero. The best reported scores are only around 6%: Gemini-3-Flash reaches 6.26% on PTB-XL, and ECG-R1-RL reaches 5.81% on MIMIC-IV-ECG. Many models, especially smaller open-weight or non-ECG-specific ones, are at or below 1% Completion (Oh et al., 15 Mar 2026).

Depth reveals why. Most models exceed 1.0, indicating that they can often identify the correct criterion in Step 1, but they rarely progress far beyond 2.0, which means they collapse during Finding Identification or ECG Grounding. The failure is therefore not primarily ignorance of textbook criteria. Rather, models often know what should be checked but fail when asked to verify the corresponding evidence in the ECG.

The contrast between IDA and GT-RDA further sharpens this interpretation. Many non-ECG-specific models have IDA around 50%, close to random guessing on the initial diagnosis, but their GT-RDA increases dramatically when supplied with ground-truth reasoning. Hulu-Med (32B), for example, rises from 57.49% IDA to 99.42% GT-RDA on PTB-XL, and from 51.62% to 97.20% on MIMIC-IV-ECG. ECG-specific models show the opposite pattern: they often have high IDA, suggesting strong direct diagnostic pattern recognition, yet GT-RDA drops sharply. ECG-R1-RL falls from 85.41% IDA to 22.70% GT-RDA on PTB-XL and from 80.17% to 22.95% on MIMIC-IV-ECG. PULSE also drops sharply (Oh et al., 15 Mar 2026).

The paper interprets this asymmetry as evidence of shortcut learning. General multimodal models can often follow a correct reasoning path when it is externally provided, but do not spontaneously generate one. ECG-specific models appear to have learned direct mappings from global ECG patterns to labels, without faithful intermediate deduction. This is one of the benchmark’s most consequential claims: high ECG-specific diagnostic accuracy may coexist with poor stepwise interpretability.

The failure modes reported are consistent with this interpretation. Smaller models tend to lose context in the multi-turn setting and produce repetitive or irrelevant responses. General multimodal models can display strong GT-RDA while failing autonomous reasoning. ECG-specific models can display high baseline diagnosis accuracy yet collapse under structured verification. This suggests that the benchmark is testing a capability not well captured by existing ECG model training paradigms.

6. Relationship to adjacent ECG reasoning and benchmark paradigms

ECG-Reasoning-Benchmark belongs to a broader shift in ECG AI from single-output classification toward grounded, multimodal, and clinically structured evaluation. That shift, however, is heterogeneous. ECG-QA systems such as Q-Heart target question answering over ECGs and show that lead-aware mapping, retrieval augmentation, and multimodal tuning improve exact-match accuracy on the ECG-QA benchmark, but that benchmark does not require the strict four-step verification loop used here (Pham et al., 7 May 2025). Grounded multimodal systems such as GEM define “Grounded ECG Understanding” in terms of diagnosis plus feature-level explanation and measurable evidence, and evaluate diagnosis accuracy, lead assessment, feature grounding, and clinical diagnostic fidelity, but they do not impose the same termination-based, multi-turn deduction protocol (Lan et al., 8 Mar 2025).

Other recent resources emphasize complementary aspects of ECG reasoning. Heartcare-Bench evaluates diagnosis, waveform/form analysis, rhythm interpretation, report generation, and signal prediction across signal and image modalities (Xie et al., 6 Jun 2025). MEETI provides a four-way aligned dataset of signal, image, structured features, and interpretation text for multimodal ECG reasoning and explainability (Zhang et al., 21 Jul 2025). A separate reasoning-evaluation framework decomposes ECG reasoning into Perception and Deduction, using executable verification of waveform claims and retrieval-based verification of clinical logic; that framework is closely aligned in spirit with the grounding concerns exposed by ECG-Reasoning-Benchmark, though its evaluative primitives differ (Xu et al., 27 Feb 2026).

The benchmark also sits within a longer tradition of ECG benchmarking that includes PTB-XL task standardization (Strodthoff et al., 2020), representation-level evaluation of ECG foundation models across cross-continental datasets (Filice et al., 29 Jan 2026), critiques of narrow arrhythmia-centric benchmarking and macro-AUROC-only reporting (Berger et al., 19 Feb 2026), domain-specific benchmarks such as ECG biometric recognition (Melzi et al., 2022), and even reconstructive reasoning settings such as chest vibration to ECG prediction in Vib2ECG (Lu et al., 16 Mar 2026). This suggests that “ECG reasoning benchmark” has become an umbrella notion covering at least four distinct objects: signal-grounded diagnosis, multimodal explanation, representation robustness, and cross-modal physiological inference. ECG-Reasoning-Benchmark is specifically the branch concerned with explicit clinical deduction over 12-lead waveforms.

7. Limitations, implications, and future development

The benchmark intentionally filters ambiguous cases so that the ground truth reasoning path is clean. This improves evaluability, but it also means the setting is less uncertain than real clinical practice, where ECGs often support graded confidence rather than binary criterion satisfaction. The paper therefore acknowledges that future versions should formalize uncertainty and allow models to express doubt appropriately (Oh et al., 15 Mar 2026).

It also suggests extending static logic diagrams into dynamic reasoning diagrams with severity and certainty weights, so that a model could be rewarded for clinically sensible early stopping when evidence is overwhelming. This would move evaluation closer to real-world cardiology, in which complete traversal of every possible criterion is not always necessary once a diagnosis is sufficiently supported.

The broader implication is methodological. If ECG systems continue to be evaluated mainly by final labels, judge-model fluency, or coarse QA metrics, then fundamental failures in evidence grounding may remain hidden. ECG-Reasoning-Benchmark therefore argues for reasoning-centric medical AI: models should be trained and tested to identify criteria, verify findings, localize evidence, quantify measurements, and connect those verified findings to diagnoses through transparent clinical logic (Oh et al., 15 Mar 2026).

In that sense, ECG-Reasoning-Benchmark is not primarily a performance leaderboard. It is an instrument for distinguishing criterion retrieval from perceptual verification, direct pattern recognition from explicit deduction, and fluent explanation from signal-grounded reasoning. Its main finding—that current systems remain dramatically weaker under this stricter standard than under surface-level evaluation—has made it a reference point for subsequent work on grounded ECG interpretation and trustworthy multimodal clinical AI.