The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models

Abstract: How robust and generalisable are pathology foundation models and have their scaling limites been reached? We benchmarked twelve pathology foundation models (PFMs) and ResNet baselines using our Robustness Evaluation and Enhancement Toolbox (REET) across eleven clinically realistic perturbations and a dissimilarity-driven Non-Redundant K-fold validation (NR-Kfold) protocol. We introduce a Perturbation Performance Index (PPI) to summarise accuracy trends under controlled perturbation sweeps and analyse robustness scaling with parameter count. We show that PFMs consistently outperform CNNs in both robustness and domain generalisation, yet model scaling shows diminishing returns: mid-sized models such (UNI2/Virchow-2 etc.) achieve comparable or greater resilience than larger systems. NR-Kfold analysis further reveals systematic accuracy loss and increased variability when training-test similarity is broken, underscoring the need for explicit distribution-shift evaluation. These findings suggest that the next generation of pathology foundation models must prioritise data quality, multimodality information and domain alignment over parameter count to achieve genuine clinical reliability.

• The paper introduces a unified framework using REET and NR-Kfold to benchmark pathology foundation models against realistic perturbations and domain shifts.
• It demonstrates that transformer-based models exhibit strong robustness with pixel distortions being most detrimental while geometric changes are best tolerated.
• The study reveals that scaling beyond mid-sized architectures offers diminishing returns in model generalisation and robustness.

Benchmarking Robustness and Generalisation of Histopathology Foundation Models

Introduction

This essay critically examines "The Good, the Bad, and the Brittle: Benchmarking Robustness and Generalisation of Histopathology Foundation Models" (2607.04401), which presents a unified framework for evaluating the robustness and generalisation capacity of twelve pathology foundation models (PFMs) and convolutional neural network (CNN) baselines across a diverse array of perturbations and domain shifts. Using the Robustness Evaluation and Enhancement Toolbox (REET), the authors present a multi-faceted experimental analysis encompassing both clinically realistic perturbations and stringent domain shifts implemented via a non-redundant K-fold protocol. The study introduces the Perturbation Performance Index (PPI) for summarising model resilience and systematically interrogates the scaling behavior of PFMs, thereby offering concrete insights into the limits of current approaches in computational pathology.

Methodological Framework

The benchmarking employs REET to generate eleven perturbations spanning pixel-level (noise, brightness, compression), stain/colour-space, and geometric (blur, crop, rotation, zoom) changes, all reflective of real-world challenges in clinical histopathology. For each model-dataset combination, performance is quantified by integrating the area under ROC-AUC curves over a spectrum of perturbation intensities—the PPI metric—enabling direct comparison across diverse perturbation types. Additionally, the study introduces the Non-Redundant K-fold (NR-Kfold) evaluation paradigm, which minimizes latent similarity between training and test sets by feature-space clustering, thereby exposing susceptibility to distributional and covariate shifts.

Datasets, Model Families, and Experimental Setup

Four representative patch-level datasets (NCT, PANDA, PanNuke, PatchCamelyon) serve as the experimental benchmarks, capturing a breadth of tissue types, label distributions, and acquisition artifacts. The evaluated models range from traditional ResNet18/50 CNNs to a spectrum of PFMs: early transformer architectures (Hibou-B/L, Phikon-v1/v2, EXAONEPath, UNI), mid-sized self-supervised ViTs (Virchow, Virchow2, UNI2), and billion-parameter models (GigaPath, H-Optimus-0/1). All encoders are frozen and subjected to linear probing, facilitating a uniform assessment of feature robustness against controlled perturbations.

Perturbation Analysis and the Perturbation Performance Index

PFMs generally demonstrate substantial robustness benefits compared to convolutional baselines, with ViT-based encoders and models pretrained on diverse and multi-center data (EXAONEPath, Virchow2, UNI2, Phikon-v2) exhibiting especially high PPI scores. The study unveils structured performance degradation patterns: pixel-level distortions (noise, brightness, compression) are most detrimental (PPI ≈ 0.4–0.6), while stain/colour perturbations induce moderate loss (PPI ≈ 0.6–0.75); geometric transformations (blur, rotation, crop, zoom) are best tolerated (PPI ≈ 0.8–1.0), underscoring the spatial invariance of transformer-based models.

Figure 1: (i) ROC-AUC vs. normalised constraint for pixel noise with PPI as area under the curve; (ii) PPI heatmaps across datasets, models, and perturbations (higher is better); (iii) PPI vs model parameter counts for different datasets.

A notable finding is the saturation of robustness improvements beyond mid-sized model regimes: mean PPI does not correlate strongly with parameter count (ρ = 0.15–0.55), and models exceeding 600M parameters (e.g., GigaPath, H-Optimus-1) offer no tangible robustness gain over more efficient mid-sized PFMs. This suggests that advances in data quality and pre-training regime, rather than aggressive parameter scaling, are the key determinants of out-of-distribution stability.

Domain Shift Generalisation: NR-Kfold Evaluation

The NR-Kfold protocol compellingly demonstrates the fragility of models under stringent distribution shift conditions. While standard random folds yield high ROC-AUC, enforcing feature-space dissimilarity between training and test substantially diminishes mean performance, with ΔAUC ranging from –0.02 (NCT) to –0.23 (PanNuke), and increases variability. Despite this, PFMs (notably Virchow2, UNI2, H-Optimus-1) are less sensitive than CNNs or earlier transformer designs, displaying only moderate AUC drops (mean ΔAUC ≈ –0.07) and lower inter-fold standard deviation (SD ≈ 0.09–0.12).

Crucially, scaling to large model sizes fails to mitigate this brittleness: parameter count shows weak correlation with NR-Kfold ROC-AUC (ρ=0.22, p=0.096), and the accuracy plateau is accompanied by a mild negative trend for models above the 600M threshold (bootstrap slope –0.16, CI95% [–0.16, 0]). These results indicate an empirical limit to generalisation via model capacity in contemporary PFMs.

Implications and Future Directions

The key findings indicate that merely scaling foundational vision architectures for histopathology does not yield commensurate returns in robustness or cross-domain generalisation. Instead, three core axes—diversity and realism of pre-training data, domain-specific architecture and objective choices, and systematic evaluation against distributional shift—are critical for progress towards clinically deployable AI. There is a strong implication that future work must prioritise multimodal pre-training, targeted augmentation, and explicit domain-alignment mechanisms. Integration of clinical covariates, tissue microenvironment context, or molecular signals represents promising directions for improving the real-world fidelity of computational pathology systems.

Systematic robustness assessment, such as that operationalised via the REET suite and NR-Kfold protocols, should become standard in model validation, particularly for applications with high safety or generalisation requirements. Beyond frozen encoders and patch-level inference, further studies will be needed to extend these analyses to end-to-end WSI pipelines and joint visual-textual models.

Conclusion

This study sets a rigorous benchmark for evaluating the robustness and generalisation of PFMs in computational pathology. The methodology and findings establish that the dominant dimensions of model reliability now center on data, pre-training, and clinical realism, rather than sheer architectural scale. As PFMs approach a saturation point in parameter-driven improvement, advances in domain adaptation, data strategy, and multimodal integration are imperative for clinically robust pathology AI.