MM-NeuroOnco-Bench: MRI Diagnostic Benchmark

• MM-NeuroOnco-Bench is a manually annotated evaluation benchmark designed to assess diagnostic reasoning over brain tumor MRI slices, emphasizing semantic understanding over segmentation.
• It leverages a curated subset of 1000 slices from a larger multimodal dataset with 200,000 enriched instructions to systematically test diagnostic attributes like tumor type, morphology, and spread.
• The benchmark introduces a rejection-aware evaluation protocol and highlights current models’ limitations in accurate image-grounded diagnosis, prompting improvements in clinical reasoning capabilities.

MM-NeuroOnco-Bench is a manually annotated evaluation benchmark for multimodal brain tumor MRI understanding introduced within the broader MM-NeuroOnco project, which also includes a large-scale instruction-tuning dataset and the domain-adapted NeuroOnco-GPT model. Its central purpose is to evaluate diagnosis-oriented reasoning grounded in MRI manifestations rather than lesion delineation alone. In the source project, MM-NeuroOnco comprises 24,726 MRI slices from 20 data sources paired with approximately 200,000 semantically enriched multimodal instructions spanning diverse tumor subtypes and imaging modalities, while MM-NeuroOnco-Bench is a curated evaluation subset designed to test clinically grounded semantic understanding under stricter annotation rules and a rejection-aware protocol (Guo et al., 26 Feb 2026).

1. Identity and scope

MM-NeuroOnco-Bench is best understood as the evaluation component of a three-part system: MM-NeuroOnco as the multimodal instruction dataset and image pool, MM-NeuroOnco-Bench as the benchmark, and NeuroOnco-GPT as the fine-tuned model used to validate the dataset’s utility. The benchmark is explicitly described as a manually annotated benchmark for multimodal brain tumor MRI understanding, derived as a high-quality subset of the larger resource (Guo et al., 26 Feb 2026).

Its motivating claim is that existing public brain tumor MRI resources are dominated by segmentation-centric paradigms. In that framing, benchmarks such as BraTS are highly valuable for lesion delineation and, in BraTS 2021, also for MGMT promoter methylation classification on pre-operative baseline mpMRI, but they remain centered on pixel-level boundaries and limited molecular endpoints rather than diagnostic semantics, radiological attributes, or interpretable reasoning (Baid et al., 2021). MM-NeuroOnco-Bench was created to measure whether models can reason over tumor type, lesion morphology, spread, and spatial manifestations from MRI slices in a clinically structured manner rather than merely identify or segment lesions (Guo et al., 26 Feb 2026).

The benchmark is intentionally slice-centered. Its design rationale is that radiologists often rely on a small set of highly informative 2D slices with cross-modal comparison, rather than exhaustive 3D voxel-wise processing. A consequence is that MM-NeuroOnco-Bench occupies a specific niche: it is a diagnostic-reasoning benchmark for MRI slices, not a volumetric segmentation benchmark, not a radiogenomic benchmark, and not a full report-generation or longitudinal treatment-response benchmark (Guo et al., 26 Feb 2026).

2. Data basis, modalities, and benchmark isolation

The broader MM-NeuroOnco resource aggregates over 100,000 brain MRI scans from 20 public data sources, including repositories such as Kaggle, Zenodo, and TCIA. After preprocessing, 73,226 valid samples remained in the “Full Master Index,” and 19,086 of these had pixel-level segmentation masks. The final instruction dataset contains 24,726 MRI slices, approximately 200,000 semantically enriched multimodal instructions, eight tumor subtypes and healthy controls, and four MRI modalities: T1, T1CE, T2, and FLAIR (Guo et al., 26 Feb 2026).

MM-NeuroOnco-Bench is derived by independently sampling 1,000 images from the raw pool and annotating them under stricter rules than the training set. The benchmark contains over 2,000 closed-ended questions and 1,000 open-ended questions, yielding 3,000+ benchmark items in total. The benchmark therefore inherits the modality diversity and tumor-oriented scope of the full resource, but with benchmark-specific annotation controls and evaluation design (Guo et al., 26 Feb 2026).

The benchmark’s split protocol is explicitly designed to reduce leakage. Global pixel-level deduplication is performed before partitioning. For volumetric datasets, only one representative 2D slice per 3D volume is extracted. The stated goal is to ensure no exact duplicate images and no multi-slice correlations from the same volume across the instruction dataset and benchmark. Stratified partitioning is performed by tumor categories and imaging modalities. For datasets lacking patient identifiers, the protocol relies on pixel-level deduplication, stratified partitioning, and slice-level disjointness (Guo et al., 26 Feb 2026).

The project states that MM-NeuroOnco covers eight tumor subtypes and healthy controls, but the complete explicit list of all eight subtypes is not fully enumerated in the available text. The paper clearly mentions examples such as glioma, meningioma, pituitary tumor, lymphoma, metastasis, and healthy or no-tumor cases. This suggests a heterogeneous diagnostic space, but the exact full subtype ontology must be taken from the released resource rather than reconstructed from the paper excerpt alone (Guo et al., 26 Feb 2026).

3. Semantic annotation and medical information completion

A defining feature of MM-NeuroOnco-Bench is its emphasis on diagnosis-oriented semantic attributes rather than coarse class labels alone. The project distinguishes between gold labels from original datasets, silver labels produced through automated semantic completion, and benchmark ground truth produced under a stricter no-LLM rule. Gold labels include tumor type, modality, and segmentation masks. Silver labels include radiological attributes such as edema and enhancement, and the core radiological sign set is formalized as

$$S = \{\text{Shape, Margin, Texture, Enhancement, Edema, Signal Intensity}\}, \quad |S| = 6.$$

The benchmark itself follows a strict “No-LLM-Inference” policy: all benchmark attribute labels are derived exclusively via human annotation or mask mapping, and predictions or semantic completions from large models are excluded from benchmark ground-truth construction (Guo et al., 26 Feb 2026).

The broader dataset’s semantic completion pipeline is organized as “Dual-Path Extraction -- Consensus Fusion -- Visual Verification.” In Stage 1, two heterogeneous high-capability VLMs, named in the appendix as GPT-5.1 and Claude-Sonnet-4.0, independently analyze each MRI slice and produce structured medical semantics. Their prompting policy includes “Omission over Fabrication,” a “Default Null Policy,” “Location–Appearance–Certainty validation,” a “Pixel Authority Principle,” use of contralateral normal-appearing white matter as a signal reference, and MRI physics consistency rules such as requiring CSF to appear hypointense on T1. Stage 2 fuses the two outputs field by field: exact matches are retained, minor degree disagreements are downgraded to broader labels such as “Present,” and major semantic conflicts are set to null. Stage 3 uses a third visual verifier under a subtraction-only mechanism that may reject fields or discard high-risk samples but may not add new information. The verifier is described as a third high-performance visual model, but its specific identity is not given in the text provided (Guo et al., 26 Feb 2026).

The project also converts masks into structured lesion semantics using deterministic geometry. Circularity is defined as

$C = \frac{4\pi A}{P^2},$

with lower $C$ indicating more irregular shape. Centroid coordinates are defined by

$$C_x = \frac{M_{1,0}}{M_{0,0}}, \qquad C_y = \frac{M_{0,1}}{M_{0,0}},$$

with image moments

$M_{pq} = \sum_x \sum_y x^p y^q I(x,y).$

Elongation is defined as

$$E = \frac{\lambda_{\text{major}}}{\lambda_{\text{minor}}}.$$

These quantities are then mapped into discrete morphology classes: “Irregular / Spiculated” if $C < 0.5$, “Round / Oval” if $C \ge 0.8$ and $E \le 1.5$, and “Lobulated” otherwise. Relative tumor size is derived from lesion area ratio and grouped into “Small/Focal,” “Medium,” and “Large/Extensive.” Spread or multiplicity is determined from connected components and dominant-core ratio, yielding “Solitary,” “Dominant with Satellites,” or “Scattered/Multifocal.” Localization is discretized to a 3×3 visual grid rather than a neuroanatomical atlas (Guo et al., 26 Feb 2026).

The textual side of the resource is also structured. Report-style descriptions follow the template

$$D = [T_{\text{base}}, T_{\text{pathology}}, T_{\text{morphology}}],$$

where the three components encode imaging context, pathology statement, and lesion details such as size, location, shape, and spread. Missing information is explicitly represented as “unknown” or by a disclaimer rather than omitted implicitly. This design is meant to penalize hallucination and to teach models to preserve uncertainty under incomplete evidence (Guo et al., 26 Feb 2026).

Quality control over the silver-label pipeline is monitored with Average Information Rate. For each sample $C = \frac{4\pi A}{P^2},$0,

$C = \frac{4\pi A}{P^2},$1

and AIR is the dataset-level mean of $C = \frac{4\pi A}{P^2},$2 over positive cases. The paper gives heuristic interpretations of AIR below 10% as overly conservative and above 90% as implausibly high. In a manual audit of 136 reviewed attribute fields, the reported attribute-level precision was 89.70%, with an information omission rate of 17.65% (Guo et al., 26 Feb 2026).

4. Task design and evaluation protocol

The benchmark evaluates multidimensional diagnostic understanding through both closed-ended and open-ended question answering. The input is a single 2D MRI slice paired with a question. Closed-ended items provide answer options, while open-ended items require free-text responses. The paper states that benchmark question stems exclude explicit lesion hints such as size or morphology, so models must rely on visual evidence rather than lexical cues (Guo et al., 26 Feb 2026).

The closed-ended benchmark categories include Modality, Diagnosis, Size, Shape, Spread, and Location. The open-ended benchmark is evaluated along Detail, Location, and Reasoning, together with an overall score. The benchmark therefore mixes pure imaging metadata recognition with lesion characterization and diagnosis-related reasoning (Guo et al., 26 Feb 2026).

Task family	Reported formulation
Closed-ended	Modality, Diagnosis, Size, Shape, Spread, Location; Accuracy
Open-ended	Detail, Location, Reasoning; LLM-as-a-Judge score

MM-NeuroOnco-Bench’s defining procedural innovation is its rejection-aware setting. For every closed-ended question, the benchmark appends a fifth option, “None of the above.” The benchmark uses this option as an explicit rejection mechanism, motivated by the claim that ordinary multiple-choice medical VQA is susceptible to shortcut learning and the forced-choice assumption that the correct answer must always be present. Scoring remains standard accuracy: selecting “None of the above” is correct when that is the designated answer, but there is no separate abstention utility or calibration-based scoring rule for rejection (Guo et al., 26 Feb 2026).

Closed-ended evaluation uses accuracy. Open-ended evaluation uses an LLM-as-a-Judge setup with a hierarchical scoring rubric and a 0-to-10 scale based on the input question, ground truth, and model output. The rubric is described as penalizing laterality confusion and unsupported pathological claims or hallucinations. The paper contains a documentation inconsistency regarding the exact judge model: the main text names Qwen3-80B-Instruct, whereas Appendix C.3 names Qwen3-Next-80B. The reported setup clearly uses a Qwen 80B-class local judge, but the exact variant is not stably named across sections (Guo et al., 26 Feb 2026).

The ten baseline LVLMs evaluated are GPT-5.1, Gemini 3 Flash, Claude-Sonnet-4.0, HuLuMed-32B, Lingshu-7B, Lingshu-32B, HuLuMed-7B, MedGemma-27B, MedGemma-1.5-4B or MedGemma-4B, and Qwen3-VL-8B. Closed-source models are evaluated through official APIs, while open-source models run on a server with $C = \frac{4\pi A}{P^2},$3 NVIDIA H100 (80G) GPUs. NeuroOnco-GPT is trained from Qwen3-VL-8B using LLaMA-Factory, LoRA, default hyperparameters, and one epoch (Guo et al., 26 Feb 2026).

5. Empirical results

The benchmark’s central empirical finding is that current multimodal models perform poorly on diagnosis-oriented neuro-oncology MRI reasoning. The paper states that even the strongest baseline, Gemini 3 Flash, achieves only 41.88% accuracy on diagnosis-related questions. In the closed-ended results table, Gemini-3-Flash scores 41.9 on Diagnosis and 40.9 overall. Other overall closed-ended scores are 37.6 for Lingshu-7B, 37.3 for HuLuMed-32B, 37.2 for GPT-5.1, 35.9 for Claude-Sonnet-4.0, 35.6 for Lingshu-32B, 34.0 for HuLuMed-7B, 31.8 for MedGemma-27B, 30.1 for Qwen3-VL-8B, and 26.5 for MedGemma-1.5-4B. The benchmark therefore reports low absolute performance even for frontier systems, and it explicitly concludes that medical-specialized LVLMs do not show a consistent advantage over general-purpose models (Guo et al., 26 Feb 2026).

Open-ended scoring yields a different ranking profile. GPT-5.1 is the best open-ended model, with Detail 71.15, Location 63.99, Reasoning 86.17, and Overall 72.72. Gemini 3 Flash is second with an overall open-ended score of 65.67. This divergence between relatively fluent open-ended reasoning and weak diagnosis accuracy is one of the paper’s central interpretive claims: coherent medical language does not imply correct image-grounded diagnostic discrimination (Guo et al., 26 Feb 2026).

The rejection-aware evaluation makes this point more sharply. In the ablation, the average score falls from 58.60 in the standard 4-option setting to 56.47 in a 5-option setting with an ordinary distractor, but to 48.65 in the 5-option rejection-aware setting. The paper interprets this as a drop of 2.13 points for adding a normal fifth option, 9.95 points for adding the rejection option relative to 4-N, and an additional 7.82-point drop relative to the normal 5-option setting. These results are presented as evidence that much apparent performance in conventional multiple-choice setups is supported by elimination shortcuts rather than robust internal knowledge boundaries (Guo et al., 26 Feb 2026).

The fine-tuning experiment with NeuroOnco-GPT demonstrates that the benchmark is difficult but responsive to domain-specific supervision. Base Qwen3-VL-8B scores 24.3 on diagnosis questions, whereas NeuroOnco-GPT (CoT) reaches 51.4, an increase of 27.1 percentage points that the paper summarizes as a 27% absolute accuracy improvement. NeuroOnco-GPT without CoT reaches 40.6 on Diagnosis and 40.0 overall; NeuroOnco-GPT with CoT reaches 51.4 on Diagnosis and 51.4 overall. Reported deltas for CoT over non-CoT are +10.8 on Diagnosis, +0.4 on Size, +12.7 on Shape, +35.9 on Spread, and +11.4 overall, although the table’s Location delta is internally inconsistent with the displayed values. The benchmark thus also functions as a training-data validation instrument: semantically grounded supervision and explicit chain-of-thought can materially improve performance on difficult diagnostic dimensions (Guo et al., 26 Feb 2026).

6. Position in the benchmark landscape and limitations

MM-NeuroOnco-Bench sits between segmentation-heavy neuro-oncology datasets and broader neuro-domain multimodal benchmarks. Relative to BraTS 2021, which combines dense tumor subregion segmentation with a binary MGMT promoter methylation task on pre-operative baseline mpMRI, MM-NeuroOnco-Bench shifts emphasis from voxel-level delineation and a single molecular endpoint toward diagnosis-related semantics, morphology, spread, and structured reasoning over slices (Baid et al., 2021). Relative to broad neuro benchmarks such as OmniBrainBench, which spans 15 imaging modalities and 15 workflow-aligned task families across brain disorders, MM-NeuroOnco-Bench is narrower in modality coverage but deeper in tumor-focused semantic supervision (Peng et al., 2 Nov 2025). Relative to NeuroABench, which evaluates anatomy identification in neurosurgical videos, it is tumor-focused and MRI-based rather than anatomy-centered and intraoperative (Song et al., 7 Dec 2025). Relative to NeuroVLM-Bench, which uses structured JSON outputs on 2D neuroimaging and includes tumor subtype prediction, MM-NeuroOnco-Bench is more explicitly diagnosis-semantic and lesion-attribute oriented, though both are slice-based and both expose the gap between technical image understanding and clinically meaningful reasoning (Dineva et al., 25 Mar 2026).

Its limitations are explicit. First, it is primarily a single-2D-slice benchmark and therefore misses volumetric context essential in routine neuro-oncology. Second, some semantics in the broader MM-NeuroOnco training resource are generated through automated silver-labeling and may contain residual errors despite quality control. Third, the benchmark inherits annotation limitations from the original public datasets. Fourth, patient-level grouping cannot always be enforced because some original 2D datasets lack reliable patient or case identifiers, so the benchmark relies on pixel deduplication and stratification rather than universal patient-level separation (Guo et al., 26 Feb 2026).

A further structural limitation is that MM-NeuroOnco-Bench is not a full multimodal fusion benchmark in the sense of combining MRI with pathology, genomics, operative video, or longitudinal clinical records. Its multimodality is primarily MRI-modality awareness plus structured textual reasoning over image-derived semantics. This suggests that MM-NeuroOnco-Bench is best viewed as a benchmark for clinically grounded MRI-based tumor diagnostic reasoning, not as a complete neuro-oncology benchmark ecosystem covering radiogenomics, post-treatment assessment, or patient-level longitudinal decision support. Even so, its benchmark philosophy is consequential: it replaces mask-only evaluation with explicit lesion semantics, introduces a rejection-aware protocol to reduce closed-ended bias, and provides a hard testbed on which current multimodal systems remain substantially below dependable diagnostic performance (Guo et al., 26 Feb 2026).