Evaluating the Generalization of Neuroimaging Foundation Models on African Brain MRI

Abstract: Neuroimaging foundation models pretrained on large, predominantly western cohorts are increasingly proposed as general-purpose backbones for brain MRI analysis. Yet, their ability to generalize to underrepresented clinical populations remains largely untested. We evaluate four recent foundation models (BrainIAC, Neuro-JEPA, NeuroVFM, and Primus) on a three-way diagnostic classification task (Control, Dementia, Parkinson's disease) using a cohort of 88 subjects from a Nigerian clinical brain MRI dataset, across four modality configurations (T1w, T2w, T1w+T2w, FLAIR), and compare against an end-to-end trained ViT3D baseline. The frozen backbones collapse to majority-class predictions, while Neuro-JEPA on FLAIR shows modest but still limited discrimination. In contrast, the end-to-end trained ViT3D achieves higher accuracy and MCC on every task (up to 53.4% accuracy, MCC=0.27) and is the only model with non-trivial recall. Our findings suggest that these frozen neuroimaging foundation models are insufficient for fine-grained diagnostic classification in small, non-western clinical cohorts, motivating parameter-efficient adaptation and broader multi-site external validation for equitable deployment in global health settings.

• The paper investigates whether four neuroimaging foundation models, pretrained on non-African data and dubbed BrainIAC, Neuro-JEPA, NeuroVFM, and Primus, effectively generalize to a diverse Nigerian clinical MRI dataset.
• End-to-end 3D Vision Transformer (ViT3D) consistently outperforms frozen foundation models across various input configurations, with the strongest baseline performance at 53.3% accuracy and MCC of 0.272 for T1w+T2w sequences.
• The study identifies potential shortcut learning in ViT3D and class distribution biases in frozen foundation models such as BrainIAC, NeuroVFM, and Primus.

The study evaluates whether neuroimaging foundation models pretrained largely on non-African cohorts transfer to a small, heterogeneous African clinical population under a frozen-backbone protocol. Using Nigerian brain MRI data, it compares four pretrained models—BrainIAC, Neuro-JEPA, NeuroVFM, and Primus—with an end-to-end 3D Vision Transformer trained from scratch. The central result is unfavorable to the assumption that large-scale pretraining alone produces broadly transferable representations: the frozen models generally underperform the task-specific ViT3D baseline, frequently collapse toward majority-class predictions, and provide weak discrimination among control, dementia, and Parkinson’s disease cases. Neuro-JEPA is a partial exception, particularly for FLAIR and multimodal inputs, but its advantage is limited and does not establish robust clinical generalization (2609.23983).

Study design and clinical setting

The evaluation uses the Nigerian Brain MRI dataset, comprising 761 MRI sequences from 88 participants recruited at three diagnostic centers. The cohort includes 31 participants with dementia, 22 with Parkinson’s disease, and 35 controls. MRI acquisition spans field strengths from 0.3 T to 1.5 T, and sequence availability is incomplete: 79 participants have T1-weighted scans, 80 have T2-weighted scans, 75 have paired T1w+T2w scans, and 80 have FLAIR scans. The resulting data reflect several sources of domain variation that are usually attenuated in benchmark datasets, including scanner heterogeneity, missing modalities, different acquisition protocols, and substantial demographic and diagnostic imbalance.

The diagnostic task is three-way classification among Control, Dementia, and Parkinson’s disease. The authors evaluate four input configurations—T1w, T2w, T1w+T2w, and FLAIR—and use subject-level five-fold cross-validation. The preprocessing pipeline includes skull stripping with HD-BET and acquisition selection using BRISQUE, a no-reference image-quality metric. These choices improve consistency across repeated intrasession acquisitions, although selecting the lowest-BRISQUE scan does not necessarily guarantee the most diagnostically representative or artifact-free acquisition.

The experimental design deliberately isolates frozen representation quality. BrainIAC, Neuro-JEPA, NeuroVFM, and Primus are used as fixed feature extractors, with only a dropout-linear probing head trained on the Nigerian data. The comparator is ViT3D, which is optimized end-to-end from scratch using the same general training framework. This comparison is important because it tests a specific deployment proposal: adapting a pretrained representation to a low-resource population without updating the backbone. It does not test whether the foundation models could perform better after full fine-tuning or parameter-efficient adaptation.

Foundation models and probing protocol

The four pretrained backbones differ substantially in their pretraining corpora and objectives. NeuroVFM is a ViT-B trained on 5.24 million CT and MRI volumes with a volumetric JEPA objective. Neuro-JEPA is a ViT-B trained on 1,551,862 multimodal MRI scans using a sparse mixture-of-experts JEPA architecture. BrainIAC is a ViT-B trained with SimCLR on 32,015 multiparametric brain MRIs from 16 datasets covering 10 neurological conditions. Primus uses a transformer encoder–convolutional decoder architecture pretrained with the Volume Contrastive objective on the OpenMind dataset. The diversity of these models allows the study to distinguish a general failure of frozen representations from an issue specific to one pretraining paradigm.

For all frozen models, the probe consists of dropout followed by a single linear projection to three classes. Optimization uses AdamW, label smoothing, a five-epoch warm-up, cosine learning-rate decay, and a maximum of 50 epochs. Multimodal inputs require modifying the first patch-embedding convolution, whose single-channel weights are replicated over the additional channels and divided by the number of channels. This preserves the pretrained parameter scale but is not equivalent to learning a modality-aware input stem. Consequently, the multimodal experiments evaluate a constrained form of transfer in which the original single-channel embedding is mechanically extended to new channel configurations.

Performance is assessed using accuracy, balanced accuracy, macro-F1, weighted F1, MCC, and macro one-versus-rest AUC. The inclusion of balanced accuracy, macro-F1, and MCC is appropriate because raw accuracy can obscure the severe class-wise asymmetry observed in the predictions. AUC is also useful in this setting because it evaluates ranking separability independently of a fixed decision threshold, but it does not by itself demonstrate clinically useful multiclass classification.

Classification performance and the frozen-backbone result

Across the four input configurations, end-to-end ViT3D generally outperforms all frozen foundation-model probes. Its strongest configuration is T1w+T2w, where it reaches 53.3% accuracy, 48.2% balanced accuracy, 43.4% macro-F1, MCC of 0.272, and AUC of 0.672. The corresponding figures on T1w are 50.7% accuracy, 46.5% balanced accuracy, 40.7% macro-F1, MCC of 0.241, and AUC of 0.651. On FLAIR, ViT3D achieves 46.6% accuracy, 43.6% balanced accuracy, 41.1% macro-F1, MCC of 0.175, and AUC of 0.675.

The implication is not merely that the from-scratch model has higher accuracy. ViT3D is also the only model that consistently achieves positive and non-trivial MCC across the evaluated configurations, indicating more meaningful agreement between predictions and labels than would be expected from a strongly biased classifier. Nevertheless, the magnitude of the best result remains modest: 53.3% accuracy in a three-class task is only moderate discrimination, and the explainability analysis raises concerns that even this performance may be partly driven by acquisition-related shortcuts rather than pathology.

The frozen models are substantially weaker. On T1w, their accuracy ranges from 38.1% to 39.3%, with MCC values between -0.022 and 0.000. On T2w, accuracy ranges from 32.3% to 40.2%, while MCC ranges from -0.158 to 0.032. For T1w+T2w, BrainIAC reaches 40.0% accuracy but has an MCC of -0.043; NeuroVFM and Primus each reach 41.3% accuracy with MCC of 0.000. These results indicate that apparent accuracy near or above the largest class proportion can coexist with almost no useful multiclass association.

The strongest frozen-backbone result is obtained by Neuro-JEPA on FLAIR: 44.8% accuracy, 40.9% balanced accuracy, 32.4% macro-F1, MCC of 0.128, and AUC of 0.684. Neuro-JEPA also produces an AUC of 0.676 on T1w+T2w, slightly above ViT3D’s 0.672. These AUC results complicate the paper’s overall conclusion. Neuro-JEPA sometimes ranks cases at least as effectively as the end-to-end baseline, even though its thresholded accuracy, balanced accuracy, and MCC are lower. The implication is that the representation may contain weakly useful ordering information that the linear decision boundary does not convert into reliable three-class predictions. Conversely, the small AUC differences should not be interpreted as evidence of superior clinical utility, particularly given the small sample size and absence of confidence intervals or external validation.

Input	Model	Accuracy	Balanced accuracy	Macro-F1	MCC	AUC
T1w+T2w	ViT3D	0.533	0.482	0.434	0.272	0.672
T1w+T2w	Neuro-JEPA	0.426	0.347	0.218	0.039	0.676
FLAIR	ViT3D	0.466	0.436	0.411	0.175	0.675
FLAIR	Neuro-JEPA	0.448	0.409	0.324	0.128	0.684
T1w	ViT3D	0.507	0.465	0.407	0.241	0.651
T2w	ViT3D	0.462	0.407	0.308	0.141	0.634

The study’s statistical analysis provides only limited inferential support for the superiority of ViT3D. Paired five-fold accuracy comparisons reach significance on T1w for BrainIAC, Neuro-JEPA, and NeuroVFM, with $p$-values of 0.044, 0.041, and 0.021, respectively. No other model-task comparison reaches $p<0.05$. The authors correctly attribute this pattern partly to limited statistical power. With only five fold-level observations and approximately 79–80 subjects per sequence, fold-level $t$-tests are unstable and cannot substitute for a larger independent test cohort.

Class-wise behavior and prediction collapse

The aggregate metrics conceal a more consequential failure mode: most frozen models do not distribute predictions across the three diagnostic categories. On T1w+T2w, BrainIAC, NeuroVFM, and Primus are strongly biased toward Dementia. Each obtains an F1 score of zero for Parkinson’s disease, while their Control F1 scores are only 0.028, 0.021, and 0.117, respectively. Neuro-JEPA performs somewhat better, with Control F1 of 0.126 and Parkinson’s F1 of 0.052, but these values remain very low.

ViT3D also performs best on the Dementia class, with an F1 of 0.629, but it produces non-zero performance for all classes: Control F1 is 0.412 and Parkinson’s F1 is 0.128. Its predictions correctly classify 11 of 22 controls, 26 of 31 dementia cases, and 8 of 22 Parkinson’s cases. This distribution is materially different from near-exclusive Dementia prediction and explains the higher MCC and macro-F1.

The class-wise results imply that the frozen models have not learned representations that preserve the distinctions necessary for this diagnostic task. In particular, their behavior is not well characterized as uniformly weak classification. It is a structured collapse toward one class, which may arise from prior prevalence, domain-specific feature mismatch, calibration failure, or a representation that encodes generic disease-related variation without separating dementia from Parkinson’s disease and controls. Because the probing head is linear, the experiment cannot determine whether the information is absent from the representations or merely inaccessible to a linear classifier.

Modality dependence and Neuro-JEPA’s partial exception

Neuro-JEPA is the only foundation model that shows a consistent, though limited, advantage on selected modalities. Its best performance occurs on FLAIR, where it obtains the highest AUC among all evaluated models at 0.684 and the strongest frozen-model results for accuracy, balanced accuracy, macro-F1, and MCC. Its relatively stronger performance on T1w+T2w also suggests that the representation can exploit complementary sequence information more effectively than the other frozen backbones.

This modality dependence is consistent with the possibility that FLAIR exposes pathology-relevant signal that is more robust to the particular diagnostic distinctions in the cohort. However, the study does not establish that FLAIR is intrinsically superior for this population. Sequence-specific sample sizes differ, missingness is not random, and the acquisition conditions are heterogeneous. Moreover, Neuro-JEPA’s advantage is not uniform across metrics or modalities. It therefore supports a narrower conclusion: the transferability of a foundation model is conditional on both the pretraining architecture and the downstream sequence configuration.

The multimodal implementation also limits interpretation. Replicating single-channel patch-embedding weights across T1w and T2w channels is a pragmatic compatibility operation, but it does not reproduce a multimodal pretraining setup. Thus, the T1w+T2w evaluation measures transfer under an input mismatch as well as population and scanner shift.

Spatial attribution and possible shortcut learning

The Grad-CAM analysis provides an important qualification to the favorable ViT3D metrics. BrainIAC and NeuroVFM produce diffuse activation throughout the field of view, with little clear localization to brain structures. Primus generates sparse, patch-like activations concentrated near the brain periphery. These patterns are compatible with weak anatomical grounding, although attribution maps alone cannot distinguish irrelevant activation from distributed pathology-related signal.

More concerningly, ViT3D repeatedly exhibits a strong hotspot at one lateral edge of the image across nearly every modality. This spatial consistency suggests a potential acquisition or preprocessing shortcut. The model may be exploiting image borders, residual skull-strip artifacts, scanner-specific structure, positioning, or other systematic features correlated with the labels. If so, its higher classification metrics do not establish that it has learned disease-specific neuroanatomy.

Figure 1: Grad-CAM attribution maps by modality and model, showing diffuse activation for several frozen backbones, peripheral activation for Primus, and a recurring lateral hotspot for ViT3D.

The implication is that explainability is not ancillary in this experiment. It changes how the numerical comparison should be interpreted. ViT3D is empirically stronger under the reported cross-validation protocol, but its performance should not be regarded as clinically grounded without tests such as spatial masking, image-border randomization, acquisition-site holdout, scanner-stratified evaluation, and perturbation-based shortcut analysis. The paper appropriately states that Grad-CAM cannot establish causality; the maps identify a robustness concern rather than proving shortcut dependence.

Limitations and open questions

The principal limitation is the small, heterogeneous cohort. The total sample contains only 88 subjects, and the diagnostic groups are imbalanced. Although subject-level cross-validation prevents direct leakage between repeated acquisitions of the same participant, five folds still provide limited statistical power and wide uncertainty. The lack of an independent Nigerian or multi-site African test set prevents assessment of external validity.

The frozen-backbone protocol is also intentionally restrictive. It evaluates whether a fixed representation plus a linear probe is sufficient, but it does not test full fine-tuning, LoRA, adapters, normalization-statistic recalibration, or nonlinear probing. A negative result under this protocol cannot establish that the pretrained models are intrinsically uninformative. It establishes that their features are not readily usable through the specified low-data linear adaptation procedure. This distinction is especially important because the paper’s conclusion about foundation-model generalization is broader than the direct experimental intervention.

The modality construction introduces another assumption. Multimodal inputs are created by replicating single-channel patch-embedding weights rather than using a modality-specific or jointly pretrained input layer. This may disadvantage models whose pretraining assumptions differ from the evaluation input. The study also does not quantify calibration, confidence reliability, site-specific performance, uncertainty, or computational and data costs.

Finally, the attribution analysis leaves the source of ViT3D’s lateral hotspot unresolved. The central open question is whether the observed performance hierarchy is driven primarily by population and scanner shift, by the frozen linear-probing constraint, by the small sample size, or by a mismatch between the models’ pretraining objectives and the three-way clinical task. Parameter-efficient adaptation and targeted shortcut-ablation experiments are needed to separate these explanations. A further unresolved issue is whether the Neuro-JEPA AUC advantage on FLAIR and T1w+T2w persists in a larger, prospectively collected, multi-site African cohort.

Conclusion

The study presents a direct and technically useful test of frozen neuroimaging foundation models outside the populations and acquisition environments that dominate their pretraining data. On the Nigerian cohort, BrainIAC, NeuroVFM, Primus, and—less severely—Neuro-JEPA generally fail to provide reliable three-way diagnostic classification through a linear probe. An end-to-end ViT3D baseline achieves the strongest thresholded performance, reaching 53.3% accuracy and MCC of 0.272 on T1w+T2w, but its recurring Grad-CAM hotspot raises the possibility of shortcut learning. Neuro-JEPA is a meaningful exception on FLAIR and multimodal AUC, although its class-wise performance remains limited. The evidence therefore supports a constrained conclusion: frozen foundation representations are not automatically transferable to small, heterogeneous African clinical MRI cohorts, and their evaluation requires both class-sensitive metrics and direct investigation of anatomical and acquisition-related shortcuts (2609.23983).