Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark

Abstract: Surgical resection remains the only potentially curative treatment for pancreatic ductal adenocarcinoma (PDAC), and eligibility depends on accurate assessment of vascular invasion (VI), i.e., tumor extension into adjacent critical vessels. Despite its importance for preoperative staging and surgical planning, computational VI assessment remains underexplored. Two major challenges are the lack of public datasets and the diagnostic ambiguity at the tumor-vessel interface, which leads to substantial inter-rater variability even among expert radiologists. To address these limitations, we introduce the CURVAS-PDACVI Dataset and Challenge, an open benchmark for uncertainty-aware AI in PDAC staging based on a densely annotated dataset with five independent expert annotations per scan. We also propose a multi-metric evaluation framework that extends beyond spatial overlap to include probabilistic calibration and VI assessment. Evaluation of six state-of-the-art methods shows that strong global volumetric overlap does not necessarily translate into reliable performance at clinically critical tumor-vessel interfaces. In particular, methods optimized for binary segmentation perform competitively on average overlap metrics, but often degrade in high-complexity cases with low expert consensus, either collapsing in volume or overextending at uncertain boundaries. In contrast, methods that model inter-rater disagreement produce better calibrated probabilistic maps and show greater robustness in these ambiguous cases. The benchmark highlights the limitations of volumetric accuracy as a proxy for localized surgical utility, motivating uncertainty-aware probabilistic models for preoperative decision-making.

• The paper presents the CURVAS-PDACVI dataset and challenge, which rigorously captures inter-rater variability through multi-expert CT annotations.
• It benchmarks six nnU-Net based segmentation methods, evaluating both global DSC and vessel-specific invasion metrics using uncertainty quantification.
• Findings reveal that uncertainty-aware, disagreement-modeling approaches outperform deterministic methods in accurately delineating complex tumor-vessel interfaces.

Comprehensive Assessment of Vascular Invasion in PDAC: The PDACVI Benchmark

Introduction

The precise assessment of vascular invasion (VI) in Pancreatic Ductal Adenocarcinoma (PDAC) directly influences surgical management and patient outcomes. However, the complexity of PDAC segmentation—driven by heterogeneous tumor appearance, ill-defined margins, and proximity to major vessels—results in pronounced inter-rater variability, challenging the reliability of computational methods. The paper "Assessing Pancreatic Ductal Adenocarcinoma Vascular Invasion: the PDACVI Benchmark" (2604.27582) introduces the CURVAS-PDACVI dataset and challenge, addressing these challenges by focusing on uncertainty-aware AI benchmarking with multi-expert annotations and multi-metric evaluation.

Dataset Construction and Inter-Rater Variability

The CURVAS-PDACVI dataset comprises 109 abdominal CT scans, each annotated by five independent radiologists with varied experience, alongside refined vascular segmentations for five major vessels. This multi-annotator design robustly captures the aleatoric uncertainty present in PDAC delineation, especially at the tumor-vessel interface.

Human-to-human Dice Similarity Coefficient (DSC) analyses reveal a mean agreement of $58.4\% \pm 25.7\%$, demonstrating substantial diagnostic divergence. Notably, agreement variance correlates with rater experience, and specific cases exhibit spatially disjoint annotations, emphasizing the irreducible uncertainty inherent to this task.

Figure 1: Independent segmentations from five experts illustrate the high diagnostic divergence and spatial uncertainty in PDAC and adjacent vessel delineation.

Aggregated agreement matrices further highlight that less experienced annotators are systematic outliers in both inter-rater and consensus comparisons.

Figure 2: The interrater agreement matrix demonstrates lower mean DSC for the most junior expert, confirming experience-linked annotation variability.

Methodological Overview of Benchmarked Algorithms

The CURVAS-PDACVI challenge evaluated six state-of-the-art segmentation algorithms, all based on nnU-Net architectures but differing in their treatment of multi-rater supervision and uncertainty quantification:

• TwinTrack: Ensemble-based cascade with post-hoc isotonic calibration against human consensus.
• CorpuSeg: Separate expert-specific models whose outputs are fused at inference to express rater variability.
• BreizhSeg: Bayesian neural network architecture with stochastic inference, producing probabilistic segmentations.
• MIC DKFZ: Ensemble fine-tuned with both individual and consensus annotations, calibrated via temperature scaling.
• ROISeg: Single-model training using a pre-fused STAPLE consensus, with no explicit ambiguity modeling.
• OrdSTAPLE: Joint ordinal and binary modeling to explicitly encode the distribution of annotator agreement.

Multi-Metric Evaluation Framework

Recognizing the inadequacy of single-metric benchmarks, the evaluation integrates:

• Standard and Threshold-averaged Dice: Capturing both binarized and probabilistic overlap with annotations.
• Multi-rater Expected Calibration Error (MR-ECE): Quantifying confidence calibration across all annotators.
• Continuous Ranked Probability Score (CRPS): Penalizing deviations in predicted vs. empirical volume distributions.
• Vascular Invasion Scores: Wasserstein distances between predicted and empirical angle distributions for contact with PORTA, SMV, SMA, Celiac Trunk, and Aorta.

Metric correlation analyses confirm that overlap metrics are only weakly predictive of vascular interface accuracy, justifying a comprehensive, multi-metric approach.

Figure 3: Inter-correlation analysis reveals strong internal consistency of overlap metrics and weak correlation with vessel-specific errors.

Results and Robustness Analysis

Global Performance and Metric Trade-Offs

BreizhSeg achieved the highest mean DSC ($71.04\%$) and calibration scores (MR-ECE $25.7 \times 10^{-3}$), confirming the efficacy of Bayesian uncertainty modeling in global segmentation and probabilistic reliability. However, TwinTrack and CorpuSeg outperformed others on four out of five vessel-specific VI metrics, indicating that methods explicitly incorporating annotator variability provide superior anatomical localization at surgically critical interfaces.

The leaderboard stability, rigorously validated through bootstrap resampling and Wilcoxon signed-rank analysis, affirms consistency across both global and vessel-specific domains. Yet, vascular invasion metrics did not reach pairwise statistical significance as often, underscoring the high variance induced by clinical ambiguity.

Figure 4: Heatmaps of Wilcoxon signed-rank test p-values show significant differences in overlap/calibration, but weaker significance for vessel-specific metrics due to high ambiguity.

Performance in High-Complexity and Ambiguous Cases

Focusing on cases with mean inter-rater DSC below $30\%$, the benchmark exposes an inversion in ranking: methods such as OrdSTAPLE, which explicitly model disagreement, rise to the top, whereas baseline deterministic models lose spatial overlap and calibration. This indicates enhanced robustness of disagreement-aware approaches under extreme uncertainty, evidencing their superior clinical reliability for complex surgical decision-making.

Failure case analyses show that all methods, regardless of strategy, share a subset of 'hard' studies that degrade performance—confirming that modeling ambiguity is a necessity rather than an optional refinement.

Figure 5: Failure analysis demonstrates both distinct algorithm-specific profiles and shared consistent failures on recurrent hard cases.

Representative predictions further showcase that binary-target models are prone to overconfident, sharp boundaries in ambiguous scenarios, whereas disagreement-aware models generate diffuse, uncertainty-reflective probability maps.

Figure 6: Predictions on ambiguous cases highlight sharper boundaries for binary models and more diffuse outputs for uncertainty-aware methods, aligning with the spread of expert opinion.

Theoretical and Practical Implications

This benchmark amplifies key observations for the design of AI systems in clinical oncology:

• Volumetric overlap is not a sufficient surrogate for clinical utility in PDAC staging, especially where surgical outcomes hinge on accurate, confidence-quantified assessment of critical tumor-vessel interfaces.
• Explicit modeling of annotation ambiguity and uncertainty improves system robustness and interpretability, directly supporting safer clinical translation in ambiguous cases.
• Reliance on consensus ground truth alone can obscure or even misdirect model development, as consensus strategies risk discarding diagnostically meaningful variation present in true expert opinion.

These findings position fully probabilistic, disagreement-aware segmentation as critical for future AI-enabled oncological tools, particularly in high-stakes, high-uncertainty domains.

Limitations and Future Directions

Noted restrictions include the limited sample size for extreme ambiguity analysis and the absence of histopathological ground truth for vascular involvement, confining the benchmark to radiological consensus. Additionally, all methods assessed were built on the nnU-Net backbone; expanding architectural diversity and integrating richer clinical endpoints remain open research objectives.

Extending the benchmark to broader datasets, including longitudinal outcomes and surgical findings, and encouraging the adoption of epistemic uncertainty quantification, are promising avenues for advancing AI reliability in clinical staging.

Conclusion

The PDACVI Benchmark establishes a robust, multi-rater, uncertainty-aware framework to evaluate segmentation algorithms for PDAC and its vascular invasion. The results underscore that deterministic, binary segmentation approaches are insufficiently robust for high-stakes clinical applications, particularly under high ambiguity. Methods that explicitly encode expert disagreement yield both superior calibration and more faithful uncertainty representation, critical for surgical planning. The benchmark paves the way for future probabilistic, interpretable AI systems co-developed with clinical experts.