PancreasDG: 3D MRI Segmentation Benchmark

• PancreasDG is a large-scale, multi-center 3D MRI pancreas segmentation benchmark focused on evaluating domain generalization under varying acquisition sequences.
• It employs a double-blind, two-pass protocol to create pixel-accurate pancreas masks while distinguishing same-sequence cross-center from cross-sequence shifts.
• Empirical results demonstrate that specialized methods are required for robust cross-sequence adaptation, as improvements in same-phase performance do not guarantee out-of-phase success.

Searching arXiv for PancreasDG and related benchmark papers. I’ll look for the PancreasDG arXiv entry and closely related pancreas MRI DG benchmarks. PancreasDG is a large-scale multi-center 3D MRI pancreas segmentation benchmark created to study domain generalization in medical imaging, with particular emphasis on the distinction between cross-center and cross-sequence shift. Introduced in “Rethink Domain Generalization in Heterogeneous Sequence MRI Segmentation” (Zhang et al., 30 Jul 2025), it comprises 563 MRI scans from six institutions, spanning venous-phase and out-of-phase acquisitions, and provides pixel-accurate pancreas masks created by a double-blind, two-pass protocol. The benchmark was motivated by the observation that pancreas segmentation remains unusually difficult because the organ is small, irregularly shaped, surrounded by structures such as the stomach, duodenum, and visceral fat, and often exhibits low contrast on T1 MRI; it was also framed as a response to the relative underrepresentation of the pancreas in public domain-generalization benchmarks (Zhang et al., 30 Jul 2025).

1. Clinical and methodological rationale

PancreasDG was designed around a specific claim about robustness in abdominal MRI: existing medical domain-generalization benchmarks had focused primarily on cross-center variability, while a more clinically important source of shift often arises from cross-sequence or cross-phase variation. In the benchmark’s formulation, the key comparison is between venous-phase and out-of-phase T1 MRI, rather than between institutions alone (Zhang et al., 30 Jul 2025).

The pancreas is a particularly demanding target for this question. The benchmark paper identifies four recurring sources of difficulty: the organ’s small size, its irregular morphology, its proximity to anatomically confusing neighbors, and low T1 contrast. It further notes that models achieving more than 90% Dice on organs such as the liver or kidneys can still miss 20–30% of the pancreas. This supports the benchmark’s choice of organ as a stress test for out-of-distribution segmentation rather than as a routine extension of multi-organ abdominal segmentation (Zhang et al., 30 Jul 2025).

A central conceptual contribution of PancreasDG is its rejection of the assumption that “different hospital” is the dominant form of domain shift in MRI. The benchmark instead separates same-sequence cross-center transfer from cross-sequence transfer and argues that these two settings behave differently both empirically and methodologically. This suggests that robustness claims based only on cross-center evaluation may overestimate clinical deployability when acquisition phases differ (Zhang et al., 30 Jul 2025).

2. Dataset composition and annotation protocol

The dataset contains 563 total MRI scans from six institutions: NYU Langone Health, Mayo Clinic Florida, Northwestern University, Istanbul University Faculty of Medicine, Erasmus Medical Center, and a private consortium in-house source. It focuses on 3D T1 MRI pancreas segmentation and includes 463 venous-phase scans and 100 out-of-phase scans. Reported scanner vendors include GE, Siemens, and Philips, and magnet strengths include 1.5T and 3T (Zhang et al., 30 Jul 2025).

The pancreas masks were created with a double-blind, two-pass annotation protocol and are described as pixel-accurate. In the context of pancreas segmentation, this matters because annotation uncertainty is itself a major limiting factor, especially near poorly defined boundaries and in small distal regions. The benchmark therefore treats annotation quality as part of the dataset design rather than as a secondary implementation detail (Zhang et al., 30 Jul 2025).

The domain structure is explicit:

Benchmark component	Cases	Role
Source-domain venous phase	313	Train/validation/test, split 3:1:1
Target-domain venous phase	150	Cross-center evaluation
Target-domain out-of-phase	100	Cross-sequence evaluation

Within the target domains, same-sequence cross-center evaluation uses venous-phase scans from EMC, IU, and NU, with 50 scans from each center. Cross-sequence evaluation uses out-of-phase scans from NU and IH, again with 50 scans from each center. This organization makes the benchmark simultaneously multi-center and multi-sequence, but does so in a way that keeps those two forms of shift analytically separable (Zhang et al., 30 Jul 2025).

3. Benchmark protocol and problem formulation

PancreasDG uses a source-only validation design. The 313 source-domain venous cases are split 3:1:1 for training, validation, and test, and no target-domain data are used for validation. The remaining 250 cases from other centers and phases are held out for testing. The benchmark paper explicitly presents this as a safeguard against leakage, arguing that some prior medical domain-generalization work had mixed target data into validation and thereby biased model selection (Zhang et al., 30 Jul 2025).

This evaluation design supports two distinct tasks. The first is cross-center generalization under matched sequence conditions, where source and target are all venous-phase. The second is cross-sequence generalization, where training is restricted to venous-phase data and testing is performed on out-of-phase data. The latter is treated as the more severe and qualitatively different setting (Zhang et al., 30 Jul 2025).

The benchmark’s empirical analysis yields three headline findings. First, limited sampling introduces significant variance that may be mistaken for distribution shift. Second, cross-center performance correlates with source-domain performance for identical sequences. Third, cross-sequence shifts require specialized solutions. Taken together, these findings imply that poor transfer across centers is not always evidence of a deep domain mismatch, whereas poor transfer across sequences more plausibly reflects a genuinely different generalization regime (Zhang et al., 30 Jul 2025).

4. Reported results and what they show about domain shift

PancreasDG reports that same-phase cross-center performance often improves with more training data or larger models, while cross-sequence performance on out-of-phase data can degrade as training continues or model capacity increases. The benchmark therefore treats cross-sequence failure not as a simple extension of cross-center heterogeneity but as a different failure mode (Zhang et al., 30 Jul 2025).

On the out-of-phase NU test set, the baseline Dice was 43.55 ± 26.33. Classical domain-generalization methods improved this only modestly: MixStyle reached 49.71 ± 20.35, VREX 46.78 ± 24.49, EQRM 44.68 ± 25.73, GroupDRO 43.46 ± 26.50, and IBERM 43.55 ± 26.33. Large segmentation models also remained limited, with SAM at 36.58 ± 10.52, SAM-Med2D at 9.72 ± 4.17, and SegVol at 32.80 ± 15.17. STUNet variants similarly underperformed. The proposed method reached 70.39 ± 7.82 Dice, which the paper reports as a +61.63% Dice improvement over the baseline (Zhang et al., 30 Jul 2025).

On the out-of-phase IH test set, the baseline Dice was 35.62. MixStyle reached 43.25, VREX 38.10, EQRM 36.11, GroupDRO 35.96, and IBERM 35.22. Among large segmentation models, SAM reached 35.35, SAM-Med2D 12.90, and SegVol 45.81. The proposed method achieved 66.61 Dice, reported as a +87.00% improvement over the baseline (Zhang et al., 30 Jul 2025).

A compact view of representative cross-sequence results is given below.

Method	OOP NU Dice	OOP IH Dice
Baseline	43.55 ± 26.33	35.62
MixStyle	49.71 ± 20.35	43.25
SegVol	32.80 ± 15.17	45.81
Proposed method	70.39 ± 7.82	66.61

These numbers were used to support a broader claim: generic DG methods and generic large segmentation models do not adequately solve the venous-to-out-of-phase transfer problem. A plausible implication is that the dominant failure is not merely insufficient regularization, but a mismatch between appearance-driven representations and the sequence-dependent physics of MRI contrast (Zhang et al., 30 Jul 2025).

5. Semi-supervised method and ablation findings

The benchmark is accompanied by a semi-supervised large-scale pretraining strategy intended to learn anatomically meaningful and sequence-invariant representations. The downstream segmentation backbone is nnU-Net, but the pretraining paradigm is promptable and is described as inspired by SAM and SegVol. The broader pretraining setup can accept text prompts, bounding boxes, and point annotations, while the pancreas evaluation focuses on point and box prompts during fine-tuning (Zhang et al., 30 Jul 2025).

The pretraining corpus includes 25 open-source CT segmentation datasets and 10 MRI datasets, with MRI oversampled to maintain a 1:1 CT:MRI ratio. MRI datasets named in the paper include BRATS, TotalSegmentatorMRI, AMOS22-MRI, CirrMRI600, Duke Liver, and ACDC Cardiac MRI. The method additionally uses 6.5K unlabeled 3D MRI scans. Its core assumption is that anatomy is more stable than appearance across MRI sequences, so models that learn stronger anatomical structure from many sequences and centers should transfer better to unseen sequence domains (Zhang et al., 30 Jul 2025).

Optimization uses an EMA teacher-student framework. The teacher is updated by exponential moving average and generates pseudo-labels for unlabeled scans from randomly sampled foreground points. To reduce collapse to trivial segmentations, the method adds graph-based segmentation regularization using the Felzenszwalb-Huttenlocher algorithm. Reported training details are 10 pretraining epochs at learning rate $1 \times 10^{-5}$, with training time of about one week, followed by downstream fine-tuning for 100 steps at learning rate $1 \times 10^{-4}$ (Zhang et al., 30 Jul 2025).

The ablations are notable because they clarify what the benchmark regards as sequence-robust adaptation. Fine-tuning more network blocks improved same-phase and same-center performance but harmed cross-phase performance; the best trade-off was obtained by fine-tuning the last two blocks. Prompt type also mattered. On NU, Dice progressed from 65.70 with no additional prompts, to 67.78 with points, to 70.39 with boxes, and then fell slightly to 68.89 with points plus boxes. On IH, the corresponding values were 60.95, 61.73, 66.61, and 65.10. The paper interprets this to mean that prompts help, bounding boxes work best, and the joint use of points and boxes does not help further because the information is likely redundant (Zhang et al., 30 Jul 2025).

These ablations also serve as a correction to a common misconception in transfer learning for medical segmentation: more adaptation capacity is not necessarily better under severe sequence shift. In PancreasDG, capacity that is beneficial for same-sequence fitting can be detrimental for cross-sequence robustness (Zhang et al., 30 Jul 2025).

6. Limitations, misconceptions, and position in the literature

The benchmark paper notes three explicit limitations: limited sequence diversity, annotation variability, and single-organ focus. Sequence diversity is restricted to venous-phase and out-of-phase T1 MRI, with future work suggested to include T2, DWI, arterial phase, and additional protocols. The paper also acknowledges that pancreas masks remain difficult and may contain inter-observer noise. Finally, it cautions that conclusions drawn from pancreas-only experiments may not directly transfer to all abdominal organs (Zhang et al., 30 Jul 2025).

Within the broader pancreas segmentation literature, PancreasDG occupies a specific place. Earlier large-scale work on “Large-Scale Multi-Center CT and MRI Segmentation of Pancreas with Deep Learning” assembled 767 MRI scans from five centers and 1,350 CT scans for benchmarking, and reported Dice coefficients of 85.0% on T1W MRI and 86.3% on T2W MRI using PanSegNet (Zhang et al., 2024). That work emphasized cross-modality and cross-center segmentation accuracy. PancreasDG, by contrast, isolates generalization failure under heterogeneous MRI sequence conditions, especially when only venous-phase labels are available (Zhang et al., 30 Jul 2025).

Later work extended this emphasis. “CrossPan: A Comprehensive Benchmark for Cross-Sequence Pancreas MRI Segmentation and Generalization” introduced 1,386 annotated 3D scans across T1-weighted, T2-weighted, and Out-of-Phase sequences from eight centers and reported that models achieving Dice scores above 0.85 in-domain could collapse to near-zero, below 0.02, under cross-sequence transfer. The same study found that state-of-the-art DG methods provided negligible benefit in hard single-source settings, while foundation models such as MedSAM2 retained moderate zero-shot performance through contrast-invariant shape priors (Peng et al., 20 Apr 2026). This later benchmark reinforces the core thesis already present in PancreasDG: cross-sequence generalization, not center diversity alone, is the primary obstacle to clinically deployable pancreas MRI segmentation.

PancreasDG is therefore best understood as both a dataset and an argument about evaluation. Its empirical message is not merely that out-of-phase pancreas MRI is difficult, but that domain-generalization claims in abdominal imaging must distinguish limited sampling, same-sequence cross-center heterogeneity, and genuine cross-sequence shift. That distinction has become central to subsequent pancreas MRI benchmark design (Zhang et al., 30 Jul 2025).