---
title: 'Neural-MedBench: Multimodal Neurology Reasoning'
url: https://www.emergentmind.com/topics/neural-medbench
type: topic
---

# Neural-MedBench: Multimodal Neurology Reasoning

Neural-MedBench is a compact yet reasoning-intensive benchmark for multimodal clinical reasoning in neurology. It integrates multi-sequence MRI scans, structured electronic health records, and clinical notes, and it organizes evaluation around three core task families: differential diagnosis, lesion recognition, and rationale generation. Introduced in “Beyond Classification Accuracy: Neural-MedBench and the Need for Deeper Reasoning Benchmarks,” it is explicitly designed to expose an “evaluation illusion” in medical vision-language systems, in which strong results on broad, classification-centric tests coexist with failure on deep, multimodal diagnostic reasoning; the proposed remedy is a Two-Axis Evaluation Framework that separates breadth-oriented statistical generalization from depth-oriented reasoning fidelity [2509.22258].

## 1. Problem setting and conceptual motivation

Neural-MedBench was created to expose a pervasive evaluation illusion in medical vision-language systems: models that look strong on broad, classification-centric tests frequently fail at the core of clinical practice—deep, multimodal diagnostic reasoning. The benchmark paper argues that conventional benchmarks reward pattern recognition and surface alignment, which can inflate perceived competence while under-measuring reasoning fidelity, uncertainty management, and the ability to justify conclusions. In neurology, where risk is high and imaging findings must be integrated with clinical context under ambiguity, the benchmark targets specific failures such as misweighting MRI sequences, collapsing multi-lesion evidence into single etiologies, fabricating findings to support persuasive but incorrect narratives, and failing to connect clinical context such as pharmacologic history to imaging features [2509.22258].

Its clinical focus is narrow by design. The domain is clinical neurology, and the scenarios are diagnostically complex cases spanning common conditions such as Alzheimer’s disease, ischemic stroke, and epilepsy, together with rare or atypical entities such as autoimmune encephalitis and CNS infections. These scenarios are constructed as OSCE-style stress tests emphasizing multimodal synthesis and reasoning under uncertainty. Figure 1 in the benchmark paper places Neural-MedBench in a high-Depth, lower-Breadth space, and the reported performance drop of leading VLMs on this benchmark is used as empirical evidence that success on breadth-oriented evaluation does not guarantee competence in depth.

The benchmark therefore does not present itself as a replacement for large datasets. Instead, it operationalizes the Depth axis of evaluation with a compact, expert-curated benchmark that prioritizes reasoning density over volume. This design makes Neural-MedBench an evaluative instrument for diagnosing failure modes rather than a conventional large-scale training corpus.

## 2. Corpus design, modalities, and curation protocol

Neural-MedBench contains 120 authentic neurology cases curated from over 2,000 candidates through a four-stage pipeline: pooling/filtering, expert curation, expert ground-truth annotation, and consensus/challenge validation. This process yields 200 reasoning-intensive tasks, on average 2 per case. Data sources and proportions are reported as Radiopaedia: 45.5%, JAMA Neurology case reports: 36.4%, ADNI: 18.2%, and OASIS: 9.1% [2509.22258].

The modalities are deliberately heterogeneous. Imaging includes multi-sequence MRI with balanced representation of T1, T2, and FLAIR; CT is also present. Textual inputs include structured EHR-type fields and clinical notes, and the narratives are normalized for length to control spurious effects. The benchmark description notes examples of diffusion-related reasoning, such as “integrating diffusion restriction,” while also stating that DWI, ADC, and SWI are not explicitly enumerated as standard sequences in the dataset description.

Task-level content statistics are reported by difficulty level. Level 1 has English tokens 225 and Image Pixel $(\times 10{,}000)$ 535; Level 2 has English tokens 309 and Image Pixel $(\times 10{,}000)$ 145; Level 3 has English tokens 37 and Image Pixel $(\times 10{,}000)$ 1147. The benchmark has no train/validation/test split and is used zero-shot for evaluation. It is publicly released with a leaderboard at `https://neuromedbench.github.io/`, and the dataset is hosted on Hugging Face at `https://huggingface.co/datasets/Reisen301/Neural-MedBench`.

Licensing and privacy constraints are explicit. Radiopaedia materials are under CC-BY-NC-SA 3.0; ADNI, OASIS, and JAMA sources are de-identified with approvals and acknowledgements. All data are fully de-identified, and the ADNI publication was reviewed by its Data and Publications Committee. The benchmark’s ethical posture is thus tied to re-use of de-identified clinical material rather than prospective clinical deployment.

## 3. Task families, difficulty structure, and annotation

Neural-MedBench defines three task families and three difficulty levels. The task distribution is Differential Diagnosis: 29%, Lesion Recognition: 25%, and Rationale Generation: 46% [2509.22258].

| Task family | Share | Output and scoring |
|---|---:|---|
| Differential Diagnosis | 29% | Ranked hypothesis list with justification; pass@k |
| Lesion Recognition | 25% | Lesion type and location; text-based matching and LLM grading |
| Rationale Generation | 46% | Free-form explanatory narrative; correctness, coherence, and evidence grounding |

Differential Diagnosis takes imaging plus patient history as input and requires a ranked hypothesis list with justification. Its label space is open-world diagnoses pertinent to neurology, and pass@k measures whether the ground-truth diagnosis appears in the top-k list. Lesion Recognition uses imaging as input and requires lesion type and location, emphasizing multimodal spatial reasoning; labels include radiologically typical lesion categories and anatomical locations, and the scoring uses text-based matching and LLM grading rather than pixel-level segmentation metrics. Rationale Generation takes the case materials as input and requires a free-form explanatory narrative mirroring clinical case discussions or board exams.

Difficulty is stratified as Level 1 (Direct), Level 2 (Complex), and Level 3 (Multi-turn). Level 1 contains clear cues. Level 2 covers ambiguous or rare diseases requiring deeper inference. Level 3 uses simulated consult-style dialogues requiring iterative reasoning. This stratification enables controlled movement from direct interpretation to ambiguity management and multi-turn belief revision.

The annotation protocol is expert-led. Two senior neurologists and one neuroradiologist handled expert curation and ground-truth annotation, with disagreements resolved via consensus. Rationale references were authored as structured narratives reflecting clinician reasoning, and challenge validation used baseline models to remove trivial cases. A separate human error taxonomy study was double-annotated by two board-certified neurologists, with inter-rater agreement $\kappa = 0.82$, indicating strong consistency.

## 4. Scoring architecture and evaluation methodology

Neural-MedBench implements a hybrid, clinician-calibrated scoring protocol that combines a GPT-4o-based evaluator guided by neurology rubrics, clinician validation, and semantic similarity metrics. The grader scores correctness, logical coherence, and evidence grounding. Validation against a panel of board-certified neurologists shows very high correlation with expert consensus, with Pearson $r > 0.9$, and the paper reports no systematic bias favoring GPT-4o’s own outputs [2509.22258].

For tasks with definitive outcomes, diagnostic accuracy is reported with pass@1 and pass@5. The benchmark defines pass@k as

$$
\mathrm{pass@}k = \frac{1}{n}\sum_{i=1}^{n}\mathbf{1}[\mathrm{ground\ truth} \in \mathrm{top\mbox{-}k\ predictions}_i].
$$

For free-form rationales, semantic fidelity is measured with BERTScore using `roberta-large`. The cosine similarity component is given as

$$
s(u,v)=\frac{u \cdot v}{\|u\| \|v\|}.
$$

The LLM grader rubric tolerates minor terminology differences: core diagnosis captured counts as correct despite minor terminology differences; identifying the primary condition suffices if multiple are listed in the reference; formatting differences are ignored. Confidence intervals are reported via the 95% Wilson score interval. The benchmark does not report imaging IoU or Dice, and it does not report calibration metrics such as ECE; lesion recognition is text-evaluated via matching and LLM grading.

Human baselines are included to anchor difficulty and provide clinical context. This is important because the benchmark is intended to test reasoning fidelity rather than only automated agreement with a reference string. The released clinically calibrated LLM grader is meant to enable fully automated community evaluation after one-time expert validation.

## 5. Empirical results and dominant failure modes

Thirteen VLMs were benchmarked zero-shot: GPT-4o (2025-03-26), Gemini 2.5-Flash, Gemini 2.0-Flash, Claude 4.0 Sonnet, Claude 3.7 Sonnet, Claude 3.5 Sonnet, Qwen-VL-2.5, Doubao-1.5-vision-pro, LLaVA-Med, MedGemma-27B-it, RadFM, Med-Flamingo, and HuatuoGPT-Vision. Decoding hyperparameters included temperature = 0.7 and top\_p = 0.95, with seed = 22 for GPT-4o [2509.22258].

The main quantitative result is a sharp degradation relative to conventional medical benchmarks. On direct diagnosis, MedGemma-27B-it achieves the highest pass@1 at 30.0% and 36.7% pass@5, while Gemini 2.5-Flash reaches the highest pass@5 at 46.7%; Qwen-VL-2.5 records the highest BERTScore at 0.79. GPT-4o records 20.0% pass@1, 36.7% pass@5, and BERTScore 0.70. The human baseline is substantially higher at the top end, with Senior Physician 40.0% pass@1 and Medical Student 3.3% pass@1. On complex diseases, MedGemma-27B-it again leads pass@1 with 18.3% and pass@5 with 38.3%; the human baseline is Senior Physician 35.5% pass@1 and Medical Student 3.3% pass@1. On multi-round dialogue, Gemini 2.5-Flash achieves pass@1 10.5% and pass@5 18.5%, while the Senior Physician baseline is 15.0% pass@1.

The breadth-versus-depth gap is most directly quantified for GPT-4o. Its passing rate is reported as 53.96% on GMAI-MMBench, 29.74% on OmniMed-VQA, and 9.67% on Neural-MedBench. The absolute drop from GMAI-MMBench to Neural-MedBench is 44.29 percentage points, with a relative drop of approximately 82.1%; from OmniMed-VQA to Neural-MedBench, the drop is 20.07 points, with a relative drop of approximately 67.5%. These values are central to the paper’s argument that breadth-oriented evaluations can materially overstate clinical reasoning competence.

Error analysis on 100 incorrect responses shows that Reasoning Failure is dominant at 51%, followed by Perceptual Failure at 27%, Visual Hallucination at 13%, and Knowledge Gap at 9%. The benchmark defines reasoning failure in concrete ways: correct observations but incorrect causal inference, misweighting across sequences such as over-interpreting FLAIR without diffusion integration, and collapsing multi-lesion evidence. Visual hallucination includes fabricated findings such as “midline shift” or “ring enhancement” that then anchor incorrect diagnoses. The prominence of reasoning failure over perceptual failure is one of the benchmark’s strongest empirical claims.

## 6. Two-Axis evaluation, limitations, and safety implications

The benchmark formalizes a Two-Axis Evaluation Framework. The Breadth axis refers to large, diverse datasets for statistical generalization and coverage. The Depth axis refers to compact, diagnostically complex, expert-curated benchmarks for reasoning fidelity, multimodal synthesis, uncertainty management, and justification quality. Neural-MedBench is presented as an operationalization of the Depth axis for neurology, and the paper’s evidence is used to support the hypothesis that breadth and depth are largely uncorrelated [2509.22258].

The benchmark’s limitations are explicit. Its scale is modest—200 tasks—by design for dense reasoning assessment. It is biased toward conditions with pronounced imaging correlates, such as stroke and tumors, and functional or metabolic disorders are underrepresented. Domain shift across centers, scanners, and reporting conventions is not systematically captured yet, although future iterations aim to expand multi-center and longitudinal coverage. Statistical significance tests beyond confidence intervals are not specified.

The clinical safety implication is direct: the dominance of reasoning failures indicates that models can “see” but struggle to integrate evidence and adjudicate competing hypotheses. A plausible implication is that such systems may remain hazardous in high-stakes neurology even when their broad benchmark scores appear strong. Neural-MedBench therefore functions not only as a performance benchmark but also as a diagnostic stress test for clinically trustworthy AI.

The benchmark is also designed for cost-effective assessment. It reduces image token costs by approximately $10\times$ versus OmniMed-VQA, reported as 1K images at approximately \$2.50 cost versus 128K images at approximately \$320. This enables robust multi-sample reliability analyses without prohibitive compute, and it strengthens the benchmark’s role as an open and extensible diagnostic testbed.

## 7. Position within the benchmark landscape

Despite the similarity in naming, Neural-MedBench is distinct from the benchmarks titled “MedBench.” One MedBench is a large-scale Chinese benchmark for medical LLMs comprising 40,041 questions from the Chinese Medical Licensing Examination, the Resident Standardization Training Examination, the Doctor In-Charge Qualification Examination, and expert-annotated EHR-derived cases; it emphasizes unified evaluation of factual knowledge and clinical reasoning in Mainland China [2312.12806]. Another MedBench is a cloud-based Chinese benchmarking system with 300,901 questions spanning 43 clinical specialties and five evaluation dimensions, with physical separation of questions and ground truths and dynamic mechanisms such as circular shuffling of choices and random prompt matching [2407.10990]. Neural-MedBench differs by focusing on clinical neurology, multimodal imaging-plus-text integration, zero-shot evaluation, and compact depth-oriented stress testing.

Later benchmark directions illuminate adjacent design spaces. MedBench v5 moves toward dynamic, process-oriented evaluation with five reasoning audit nodes and hallucination propagation monitoring across initiation, propagation, anchoring, and contradiction interaction [2606.24155]. NeuroVLM-Bench, also centered on neurological disorders, evaluates 2D MRI and CT with structured outputs for diagnosis, subtype, modality, sequence, plane, and confidence, and measures abstention-aware discrimination, calibration, structured-output validity, and computational efficiency [2603.24846]. This suggests that Neural-MedBench occupies a specific middle position: deeper than breadth-oriented exam-style suites, but earlier than full process-audit and hallucination-trajectory frameworks.

Its enduring significance lies in making “reasoning fidelity” an explicit evaluative target. By showing that current medical VLMs lag in integrative clinical reasoning even when classification metrics on breadth benchmarks look strong, Neural-MedBench reframes benchmark design around clinically meaningful failure modes: misintegration across MRI sequences, unsupported rationales, hallucination, and poor handling of uncertainty. In that sense, it is best understood as a neurology-specific argument for depth-oriented benchmark construction rather than as merely another medical classification dataset.

Source: https://www.emergentmind.com/topics/neural-medbench