NeuroOnco-GPT: Multimodal Brain Tumor Diagnosis

• NeuroOnco-GPT is a domain-adapted vision-language model that performs diagnostic reasoning on brain tumor MRI scans using explicit radiological semantics and multi-modal inputs.
• It is fine-tuned on the MM-NeuroOnco dataset, which pairs thousands of MRI slices with structured instructions and uses a collaborative multi-model annotation pipeline for robust quality control.
• Its Chain-of-Thought variant boosts diagnostic accuracy by 11.4%, demonstrating significant improvements over baseline models on the MM-NeuroOnco-Bench.

NeuroOnco-GPT is a domain-adapted large vision-LLM for multimodal, clinically grounded diagnosis of brain tumors from MRI scans. It is built on Qwen3-VL-8B and specialized through fine-tuning on MM-NeuroOnco, a multimodal instruction dataset comprising 24,726 MRI slices from 20 data sources paired with approximately 200,000 semantically enriched instructions. The system was introduced together with MM-NeuroOnco-Bench, a manually annotated benchmark designed to evaluate diagnosis-related question answering under a rejection-aware protocol intended to better reflect clinical uncertainty (Guo et al., 26 Feb 2026).

1. Definition and diagnostic scope

NeuroOnco-GPT is presented as a domain-adapted large vision-LLM whose objective is not merely lesion detection, but diagnosis-related reasoning grounded in MRI manifestations. The model accepts 2D MRI slices across T1, T2, FLAIR, and T1ce modalities, together with structured natural-language instructions or questions. Its language component supports both closed-form question answering and open-ended question answering, with explicit Chain-of-Thought supervision emphasizing reasoning grounded in medical evidence (Guo et al., 26 Feb 2026).

This scope distinguishes NeuroOnco-GPT from workflows centered on a single downstream endpoint such as segmentation or molecular prediction. In the formulation associated with MM-NeuroOnco, the target is multimodal MRI understanding: tumor subtype recognition, interpretation of morphology and spread, and response generation that is constrained by structured radiological semantics rather than free-form captioning. A plausible implication is that the system is intended to operationalize diagnostic semantics as a first-class training signal, rather than treat them as incidental metadata.

2. MM-NeuroOnco as the supervisory substrate

The training substrate for NeuroOnco-GPT is MM-NeuroOnco, which was created to address the limited annotation richness and diagnostic semantics of existing public brain-tumor MRI datasets. Each image is paired with gold labels such as tumor type and with silver structured attributes including morphology, location, enhancement, and edema. The instruction set spans both closed-ended and open-ended formats, and many examples include explicit Chain-of-Thought steps intended to require reasoning such as modality identification, localization, morphology assessment, and pathology inference (Guo et al., 26 Feb 2026).

The semantic schema is unusually granular. The documented attributes include tumor subtype, imaging modality, morphology, location, spread, enhancement, edema, margins, and signal characteristics. The attribute design also incorporates explicit quantitative descriptors, including circularity,

$C = \frac{4\pi A}{P^2},$

and dominant core ratio,

$f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$

Closed-ended questions may include up to five options and can contain adversarial distractors or a “none of the above” option. Open-ended questions require natural-language descriptions of findings, diagnostic rationale, or Chain-of-Thought explanations. This suggests that the dataset was designed not only for recognition but for supervision of diagnostic decomposition into evidence-bearing subproblems.

3. Automated semantic completion and quality control

A central feature of the NeuroOnco-GPT pipeline is a multi-model collaborative annotation procedure for automated medical information completion and quality control. In the first stage, two heterogeneous commercial vision-LLMs independently annotate each image’s key attributes such as shape, edema, and enhancement. In the second stage, the outputs are fused; fields are retained only on strict agreement, or are downgraded or binarized when conflicting. In the third stage, a separate strong vision model verifies the outputs and is allowed only to remove fields, not add them, to reduce hallucination risk (Guo et al., 26 Feb 2026).

The reported manual review of these silver labels gives attribute-level precision of approximately $89.7\%$, an information omission rate of approximately $17.7\%$, and an Average Information Rate intended to balance informativeness against hallucination risk. Within the paper’s own framing, this produces structured, evidence-driven annotations grounded in verified or cross-validated attributes rather than unconstrained text generation. The importance of this design is methodological: the supervision used for NeuroOnco-GPT is not simply synthetic question generation, but a curated semantic layer inserted between MRI and language output.

4. Architecture and instruction tuning

NeuroOnco-GPT uses Qwen3-VL-8B as its backbone. The model is fine-tuned on MM-NeuroOnco through LoRA-based supervised fine-tuning for a single epoch using default LLaMA-Factory settings. Instruction synthesis is performed with Qwen3-Next-80B, which is used to generate linguistically diverse and diagnostically challenging instructions from curated attributes. Inputs are image-question pairs, and outputs are either multiple-choice answers or open-ended responses with diagnostic rationale (Guo et al., 26 Feb 2026).

For closed-ended multiple-choice supervision, the training objective is standard cross-entropy:

$$\mathcal{L} = - \sum_{i} \log P(\text{label}_i \mid \text{image}_i, \text{question}_i).$$

For open-ended question answering, teacher-forcing is applied to the likelihood of the ground-truth rationale. The model also has a Chain-of-Thought variant, NeuroOnco-GPT (CoT), which explicitly incorporates reasoning supervision. In the reported results, this supervision increases closed-ended accuracy by $11.4\%$. The broader training logic is therefore not only multimodal adaptation but also explicit alignment of reasoning trajectories with the diagnostic semantics encoded in MM-NeuroOnco.

5. Benchmark design and reported performance

Evaluation is carried out on MM-NeuroOnco-Bench, a manually annotated benchmark containing 1,000 images, more than 2,000 closed-ended questions, and more than 1,000 open-ended questions. The benchmark uses a rejection-aware protocol in which each closed-ended item includes a “none of the above” option. The paper reports that introducing this rejection option decreases average accuracy by about $10\%$, and interprets this as exposing capability boundaries and reducing shortcut learning (Guo et al., 26 Feb 2026).

The reported benchmark results are summarized below.

Model	Diagnosis accuracy	Open-ended reasoning
Gemini-3-Flash	41.9%	65.67%
GPT-5.1	37.0%	72.72%
HuluMed-32B	38.2%	61.44%
Lingshu-7B	39.2%	61.53%
NeuroOnco-GPT	40.6%	—
NeuroOnco-GPT (CoT)	51.4%	—

In the paper’s abstract, the strongest baseline, Gemini 3 Flash, reaches only $41.88\%$ accuracy on diagnosis-related questions, while fine-tuned NeuroOnco-GPT achieves a $27\%$ absolute accuracy improvement on diagnostic questions. The Chain-of-Thought variant reaches $51.4\%$ diagnosis accuracy in the benchmark table. Open-ended performance is evaluated separately through an LLM-as-a-judge protocol with a hierarchical rubric from 0 to 10 that penalizes safety errors and hallucinations. Collectively, these results position NeuroOnco-GPT as a model whose main empirical contribution is improved diagnosis-related reasoning after domain-specific instruction tuning, rather than dominance across all evaluation modes.

6. Relation to adjacent systems, significance, and limitations

NeuroOnco-GPT belongs to a broader shift in neuro-oncology AI from single-task imaging models toward systems that integrate diagnosis, molecular inference, and language generation. A closely related example is GMMAS-GPT, which takes the outputs of a multimodal MRI analysis system—segmentation proportions and molecular or histological predictions—consults a knowledge base such as WHO classification and clinical guidelines, and generates a personalized prognosis and treatment recommendation report (Liu et al., 29 Jan 2025). NeuroOnco-GPT differs in emphasis: it is trained directly as a multimodal diagnostic reasoning model from MRI slices and semantically rich instructions, rather than as a report generator layered on top of a separate imaging pipeline (Guo et al., 26 Feb 2026).

Its emergence also reflects the limitations of generic multimodal LLMs in neuro-oncologic imaging. On a separate brain-tumor VQA benchmark derived from three BraTS datasets, GPT-5-mini achieved a macro-average accuracy of $f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$0, GPT-5 $f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$1, GPT-4o $f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$2, and GPT-5-nano $f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$3, and the authors concluded that performance was not acceptable for clinical use (Safari et al., 14 Aug 2025). Within that context, the MM-NeuroOnco program can be read as an attempt to make multimodal diagnostic reasoning measurable, teachable, and rejection-aware. This suggests that NeuroOnco-GPT is as much an infrastructure contribution—dataset, benchmark, and semantic supervision pipeline—as it is a standalone model.

The principal limitation stated by the benchmark itself is difficulty: even strong baselines remain near the low-$f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$4 range on diagnosis-related questions, and the rejection-aware setting reduces average accuracy by roughly $f_{\mathrm{core}} = \frac{A_{\max}}{\sum A_i}.$5. The benchmark is therefore deliberately adverse to inflated scores. The paper frames this not as a failure of a specific model, but as evidence that multimodal brain-tumor diagnostic understanding remains substantially unresolved. Code and dataset are publicly available at the MM-NeuroOnco repository, reinforcing the project’s role as a public benchmark for clinically grounded multimodal reasoning in neuro-oncology (Guo et al., 26 Feb 2026).