MM-NeuroOnco: Multimodal Neuro-Oncology

• MM-NeuroOnco is a unifying paradigm that integrates imaging, pathology, genomics, and radiomics for quantitative and clinically oriented neuro-oncology.
• It automates end-to-end MRI workflows—from scan classification and segmentation to radiomic feature extraction—ensuring reproducibility and precision.
• It extends to molecular profiling and diagnostic reasoning through PDE-constrained tumor modeling and multimodal instruction-tuned benchmarks, enhancing treatment support.

MM-NeuroOnco denotes a set of multimodal neuro-oncology frameworks that operate at different levels of abstraction, ranging from automated MRI workflow orchestration and PDE-constrained tumor modeling to multimodal foundation models and instruction-tuning benchmarks for diagnostic reasoning. In the cited literature, the term is used both for operational pipelines that transform heterogeneous neuro-oncology MRI into quantitative measurements and for broader research programs that integrate imaging, pathology, genomics, radiomics, and model-based inference for diagnosis, prognosis, and treatment support (Chakrabarty et al., 2022, Mang et al., 2020, Guo et al., 26 Feb 2026). This suggests that MM-NeuroOnco is best understood as a unifying paradigm for multimodal, quantitative, and clinically oriented neuro-oncology rather than as a single invariant software artifact.

1. Terminological scope and major formulations

Across the literature, MM-NeuroOnco appears in at least three direct formulations. One is the Multi-Modal Neuro-Oncology framework realized by the I3CR-WANO pipeline, which automates scan sorting, preprocessing, segmentation, and radiomic extraction from raw MRI DICOM data (Chakrabarty et al., 2022). A second is a “Multimodal Neuro-Oncology” framework that couples biophysical tumor-growth models with multi-parametric MRI image analysis, inverse problems, radiomics, and clinical decision support (Mang et al., 2020). A third is MM-NeuroOnco as a multimodal benchmark and instruction dataset for MRI-based brain tumor diagnosis, with associated evaluation benchmark and fine-tuned vision-LLM (Guo et al., 26 Feb 2026).

Formulation	Core inputs	Principal outputs
I3CR-WANO	multisequence neuro-oncology MRI data	scan classification, segmentation, radiomics
biophysical MM-NeuroOnco	mpMRI plus tumor-growth models	calibrated parameters, infiltration maps, decision support
MM-NeuroOnco benchmark	MRI slices plus multimodal instructions	diagnostic benchmark and NeuroOnco-GPT

This range of usage matters because it prevents an overly narrow definition. MM-NeuroOnco is not limited to lesion segmentation, even though segmentation is central in several implementations. It also includes radiogenomic prediction, multiscale mapping of microscopic variables, multimodal fusion of pathology and genomics, and evaluation of clinically grounded diagnostic reasoning.

2. End-to-end MRI workflow automation: the I3CR-WANO realization

In its most operational form, MM-NeuroOnco is realized by the I3CR-WANO pipeline as an end-to-end, highly automated system for transforming raw, heterogeneous neuro-oncology MRI studies into quantitative tumor measurements (Chakrabarty et al., 2022). The pipeline is organized into four stages: scan-type classification, reproducible preprocessing, CNN-based tumor segmentation with expert-in-the-loop refinement, and radiomic feature extraction. Each stage is encapsulated in Docker containers.

Stage I performs scan-type classification with a cascade ensemble. The first classifier is NLP-based and parses DICOM SeriesDescription ([0008,103E]) and instance counts to separate “segmentable” from “other.” The second is a CNN classifier operating on the image volume itself and assigns each segmentable series to T1-weighted, post-contrast T1 (Gd-T1), T2-weighted, FLAIR, or “non-segmentable.” Orientation is inferred via DICOM tags. On 4,404 WUSM + 644 MDA scans, the cascade achieved an overall accuracy

$$\mathrm{Accuracy}=\frac{TP+TN}{TP+TN+FP+FN}\approx 99.6\%.$$

The scan-type classifier also yielded an accuracy of over 99%, correctly identifying sequences from 380/384 and 30/30 sessions from the WUSM and MDA datasets, respectively.

Stage II performs reproducible preprocessing. Selected sequences undergo rigid and affine registration with FLIRT to a common atlas, SRI24, followed by N4 bias-field correction, ROBEX skull stripping, and intensity normalization to zero mean and unit variance after clipping at the 5th and 95th percentiles. The preprocessing container, nrg_ai_neuroonco_preproc, fixes versions of FSL, ANTs, and ROBEX to ensure bitwise reproducibility across platforms.

Stage III performs segmentation using a 3D U-Net-style CNN inspired by Isensee et al. 2017, with residual blocks, instance normalization, and deep supervision. The network predicts edema (ED), non-enhancing core (NC), and enhancing tumor (ET). Robustness to incomplete clinical protocols is handled by training separate models on every subset of {T1, Gd-T1, T2, FLAIR}. At inference, the system selects the model matching the available contrasts. If only T2/FLAIR are present, the output is binary whole-tumor segmentation; if no segmentable scan is present, the session is flagged and excluded. Expert review is built in through an XNAT + OHIF interface for manual correction of vessel mislabels, periventricular hyperintensities, choroid plexus, and related errors.

Quantitative performance was reported on external test sets. For WUSM ($n=380$ fully processed sessions), Dice scores were $0.882 \pm 0.244$ for whole tumor, $0.75 \pm 0.334$ for tumor core, and $0.91 \pm 0.240$ for enhancing tumor. For MDA ($n=30$), Dice scores were $0.977 \pm 0.040$ for whole tumor, $0.984 \pm 0.028$ for tumor core, and $0.899 \pm 0.097$ for enhancing tumor. In BraTS experiments, ET/TC performance dropped significantly without Gd-T1 ($\Delta ET \approx 0.27$, $n=380$0, both $n=380$1), which justified the fall-back to whole-tumor-only output when contrast-enhanced imaging is absent.

Stage IV extracts radiomic features with PyRadiomics. The framework computes 3D shape (14), first-order (18), and texture (75) features for every tumor subregion and image contrast, yielding 1,930 features per patient session: 70 shape and 1,860 intensity/texture features. The feature classes include GLCM, GLRLM, GLSZM, GLDM, and NGTDM families, with definitions such as

$n=380$2

3. Mechanistic and multiscale formulations

A second MM-NeuroOnco line integrates mechanistic tumor modeling with image analysis. The central construct is a patient-specific, quantitative pipeline for tumor simulation, parameter calibration, segmentation/registration, radiomic extraction, and clinical decision support (Mang et al., 2020). Its simplest tumor-growth model is a logistic reaction–diffusion equation for tumor cell density $n=380$3:

$n=380$4

The framework also considers anisotropy from DTI, multi-species tumor formulations, edema and necrosis fields, and mechanical displacement generating mass effect. Parameter estimation is posed as a PDE-constrained inverse problem with $n=380$5 data misfit and regularization. GLISTR/GLISTRboost and SIBIA are identified as leading joint segmentation-and-registration frameworks. Reported applications include BraTS 2015 segmentation, survival prediction, radiogenomics, recurrence prediction, and radiotherapy planning. Specific results reported in this framework include IDH-1 classification with accuracy $n=380$6, AUC $n=380$7; MGMT methylation with accuracy $n=380$8, AUC $n=380$9; and peritumoral edema infiltration signatures predicting recurrence location with odds-ratios $0.882 \pm 0.244$0 (Mang et al., 2020).

A distinct but related multiscale MM-NeuroOnco implementation statistically maps microscopic and molecular heterogeneity from multiparametric MR and spatially registered core biopsy (Parker et al., 2019). Twenty-nine patients underwent pre-surgical mp-MR followed by MR-guided stereotactic core biopsy. The imaging stack comprised T1w, T1w-post, T2w, T2-FLAIR, and ADC. After co-registration and white-matter normalization, voxelwise feature matrices were linked to biopsy-derived labels for IDH1 mutation, MGMT promoter methylation, cellular necrosis, and microvascular proliferation. Logistic regression quantified individual and collective predictive power, while weighted $0.882 \pm 0.244$1-nearest neighbor classification was evaluated in leave-one-out cross-validation. Voxelwise class membership probabilities were converted to $0.882 \pm 0.244$2 statistics and corrected by Benjamini–Hochberg or Gaussian random field theory.

The combined five-contrast logistic model correlated with outcome for all four microscopic variables ($0.882 \pm 0.244$3). With Benjamini–Hochberg correction, statistically significant results were obtained for IDH1, MGMT, and microvascular proliferation, with average classification accuracy $0.882 \pm 0.244$4 and average classification sensitivity $0.882 \pm 0.244$5. Random field theory improved these values to average accuracy $0.882 \pm 0.244$6 and sensitivity $0.882 \pm 0.244$7. This formulation shifts MM-NeuroOnco from lesion delineation toward whole-brain statistical mapping of microscopic and molecular properties.

4. Molecular profiling and multimodal learning extensions

Within the broader MM-NeuroOnco literature, noninvasive molecular inference is a recurrent downstream objective. A 2.5D hybrid multi-task convolutional neural network for glioma classification jointly localized tumor and predicted IDH mutation and 1p/19q codeletion from pre-operative MRI in 2,648 patients drawn from WUSM, BraTS, LGG 1p/19q, Ivy GAP, TCGA, and EGD (Chakrabarty et al., 2022). The network used post-contrast T1-weighted, T2-weighted, and T2-FLAIR scans for IDH, and T1c plus T2 for 1p/19q, together with late-fused age and tumor-location priors. It was built on Mask R-CNN with a ResNet-101 backbone and Feature Pyramid Network. On the internal, WUSM, and EGD test sets, the best IDH model achieved AUROCs of 0.925, 0.874, and 0.933, with AUPRCs of 0.899, 0.702, and 0.853. For 1p/19q, the best model achieved AUROCs of 0.782, 0.754, and 0.842, with AUPRCs of 0.588, 0.713, and 0.782.

More explicitly multimodal learning appears in UMEML, which integrates histology and genomics for glioma diagnosis and prognosis through hierarchical attention, prototype clustering, modularity-based alignment, and a registration mechanism with learnable tokens (Yi et al., 2024). On TCGA GBM-LGG, UMEML reported grading accuracy $0.882 \pm 0.244$8 and AUC $0.882 \pm 0.244$9, classification accuracy $0.75 \pm 0.334$0 and AUC $0.75 \pm 0.334$1, and survival $0.75 \pm 0.334$2-Index $0.75 \pm 0.334$3. Ablation studies showed reductions when modularity, the unified multimodal decoder, or register tokens were removed.

Foundation-model scaling has also entered the MM-NeuroOnco landscape. NeuroRAD-FM pretrained BYOL, DINO, MAE, and MoCo backbones on 7,414 three-dimensional MRI volumes from BraTS-GLI-Pre, BraTS-GLI-Post, BraTS-MEN, BraTS-MET, BraTS-PED, LUMIERE, and MU-Glioma-Post, and applied Group-DRO to reduce site and class imbalance (Bhattacharya et al., 18 Sep 2025). At CUIMC, mean balanced accuracy increased from 0.744 to 0.785 and AUC from 0.656 to 0.676, with particularly large gains for CDKN2A/2B and ATRX. For survival in IDH1 wild-type glioblastoma, $0.75 \pm 0.334$4-index improved at CUIMC, UPenn, and UCSF.

These developments sit alongside broader reviews of machine learning in neuro-oncology, which emphasize that structural MRI, diffusion, perfusion, spectroscopy, PET, and radiomics have been used for diagnosis, prognosis, and treatment-response monitoring, but that much of the evidence remains retrospective, single-centre, and vulnerable to overfitting (Booth et al., 2019, Booth, 2019).

5. MM-NeuroOnco as a benchmark and instruction dataset

A more recent usage of the name designates MM-NeuroOnco as a multimodal benchmark and instruction-tuning dataset for MRI-based brain tumor diagnosis (Guo et al., 26 Feb 2026). The dataset contains 24,726 MRI slices from 20 data sources paired with approximately 200,000 semantically enriched multimodal instructions spanning diverse tumor subtypes and imaging modalities. Four standardized MRI sequences are used: T1, T1ce, T2, and FLAIR.

The tumor subtype distribution comprises glioma 9,890, meningioma 6,783, pituitary 5,058, metastasis 1,486, lymphoma 494, schwannoma 296, craniopharyngioma 314, ependymoma 158, and healthy 147. Golden annotations include tumor type label, MRI modality tag, and segmentation mask, with masks available on 19,086 slices. Silver annotations comprise six core radiological attributes—Shape (circularity), Margin, Texture, Enhancement pattern, Edema severity, and Signal intensity—plus quantitative geometric metrics such as circularity

$0.75 \pm 0.334$5

The silver annotations are generated by a three-stage “Dual-Path → Fusion → Review” pipeline. Two heterogeneous commercial VLMs, GPT-5.1 and Claude-Sonnet-4.0, independently extract structured radiological attributes under a Default Null Policy. Consensus-based fusion retains agreements, downgrades minor degree conflicts to “Present,” and sets major conflicts to null. A third visual model then performs subtraction-only final review, meaning that it can remove or nullify fields or reject a sample, but never add new information.

MM-NeuroOnco-Bench is a manually annotated evaluation benchmark with 1,000 curated slices, 2,000 closed-ended VQA questions over Modality, Diagnosis, Size, Shape, Spread, and Location, and 1,000 open-ended questions over Detail, Location, and Reasoning. The benchmark is rejection-aware: each closed-ended question has five options, including “None of the above.” On diagnosis-related questions, even the strongest baseline, Gemini 3 Flash, achieved only 41.88% accuracy; its closed-ended overall accuracy was 40.9%. NeuroOnco-GPT, built on Qwen3-VL-8B with LoRA adapters and fine-tuned on the full 200k instruction set, reached 51.4% on closed-ended diagnosis in the Chain-of-Thought setting, a 27% absolute improvement over the open-source Qwen3-VL-8B baseline of 24.3% (Guo et al., 26 Feb 2026).

6. Clinical integration, limitations, and open problems

The literature presents MM-NeuroOnco systems as assistive infrastructures for neuro-oncology rather than autonomous replacements for expert judgment. In I3CR-WANO, the XNAT/OHIF interface explicitly embeds radiologist correction before masks are finalized, and the downstream quantitative outputs are linked to RANO criteria implementation, longitudinal tumor tracking, and decision support for personalized therapy planning (Chakrabarty et al., 2022). The broader imaging-biomarker literature similarly positions machine learning as useful for diagnosis, prognosis, and treatment-response differentiation, but notes that most evidence is low level, predominantly retrospective, and often derived from single centres (Booth et al., 2019, Booth, 2019).

Several technical limitations recur. In the BraTS 2023 intracranial meningioma challenge, skull-stripping removes any extracranial tumor extension; pre-registration and interpolation may obscure high-resolution details; and single-observer annotations leave multi-observer consensus as an open need (LaBella et al., 2023). The uncertainty-aware unified nnU-Net model for brain tumor segmentation addressed two related deficiencies—lack of voxel-wise uncertainty estimates and lack of healthy-brain context—by adding an uncertainty head and a unified normal-plus-cancer model. It reported uncertainty correlation 0.750 and RMSD 0.047 without hurting tumor accuracy, together with DSC 0.81 for brain structures and 0.86 for tumor, but the unified model used only T1 modality and the uncertainty head modeled only cancer errors (Zhou, 16 Nov 2025).

Clinical workflow extension beyond primary glioma is also visible in the management of multiple brain metastases, where contrast-enhanced MRI is the gold standard for detection and follow-up, surveillance is commonly performed every 2–3 months in the first year, and radiomics- or hazard-based AI models are proposed for personalized imaging frequency and early salvage treatment planning (Santos et al., 2023). A plausible implication is that MM-NeuroOnco pipelines will increasingly be evaluated not only by segmentation or classification accuracy, but also by their capacity to support longitudinal scheduling, uncertainty-aware review, and clinically acceptable contouring.

Taken together, the MM-NeuroOnco literature defines a technically heterogeneous but conceptually coherent field: standardized multimodal preprocessing, quantitative representation of tumor subregions and surrounding anatomy, noninvasive inference of molecular and prognostic variables, and clinically grounded human-in-the-loop deployment. The central unresolved issues are generalizability across institutions, robustness to missing modalities and protocol shift, annotation quality, interpretability, and prospective validation.