AutoPET Challenge: PET/CT Segmentation Benchmark

• AutoPET Challenge is a series of benchmarks for automated, tracer-specific lesion segmentation in whole-body PET/CT imaging.
• It evolved from FDG-only segmentation to multitracer, multicenter, and interactive human-in-the-loop settings, enhancing clinical biomarker assessments.
• Key evaluation metrics include Dice score, FPV, and FNV, highlighting the balance between overlap accuracy, missed lesions, and false positives.

The AutoPET Challenge is a benchmark series for automated lesion segmentation in whole-body PET/CT, originally centered on FDG PET/CT and later extended to multitracer, multicenter, and interactive settings. Its clinical motivation is the use of lesion burden and longitudinal lesion change as biomarkers for staging, restaging, therapy monitoring, treatment planning, and response assessment, while its technical difficulty comes from the fact that tracer uptake is not lesion-specific: high signal can reflect malignant disease, physiological uptake, inflammation, infection, or tracer-specific normal biodistribution (Heiliger et al., 2022, Dexl et al., 7 May 2026, Huang et al., 2 Sep 2025).

1. Clinical motivation and problem definition

AutoPET targets automatic delineation of tumor or lesion masks from paired PET and CT volumes. In the original FDG-oriented setting, the motivating observation was that tumor volume and changes in tumor characteristics over time are important biomarkers for cancer therapy, and that FDG-PET/CT is routinely used for staging and re-staging because fluorodeoxyglucose is taken up in regions of high metabolism. The central challenge is that these regions are not specific to tumors and can also represent physiological uptake by normal functioning organs, inflammation, or infection, making detailed and reliable tumor segmentation demanding (Heiliger et al., 2022).

This problem remains central across later editions, but the confounders broaden. In the multitracer setting, FDG and PSMA exhibit different physiological uptake patterns, so lesion segmentation cannot be reduced to thresholding PET intensity. The challenge literature repeatedly emphasizes discrimination between physiological and pathological uptake, with tracer-specific nuisance regions including the brain, heart, urinary system, kidneys, submandibular glands, parotid glands, lacrimal glands, liver, spleen, and other tracer-avid normal structures (Kalisch et al., 2024, Mesbah et al., 2024).

Manual delineation is consistently described as time-consuming, labor-intensive, and prone to inter-observer variability. A recurring implication across the series is that AutoPET is not only a segmentation benchmark but also a test of whether quantitative PET/CT analysis can be scaled beyond sparse manual measurements toward reproducible whole-body lesion characterization (Mazher et al., 29 Aug 2025, Huang et al., 2 Sep 2025).

2. Challenge evolution and dataset structure

The series evolved from a single-tracer FDG benchmark into a multitracer, multicenter, and then human-in-the-loop benchmark. The progression is summarized below.

Edition	Core focus	Data summary
AutoPET 2022	Whole-body FDG-PET/CT lesion segmentation	1,014 PET/CT scans from 900 patients; 200 hidden test scans
AutoPET III	Multitracer, multicenter, compositional generalization	1,611 studies; 200 held-out studies across four tracer-center conditions
autoPET IV	Interactive lesion segmentation	Multi-tracer, multi-center setting with click-guided interaction

The original AutoPET training dataset contained 1,014 PET/CT scans from 900 patients from University Hospital Tübingen. Among these were 513 scans without lesions, and 188 melanoma, 168 lung cancer, and 145 lymphoma scans. The hidden challenge test set contained 200 PET/CT scans, split evenly between Tübingen and LMU Munich, and was evaluated through container submission on Grand Challenge (Heiliger et al., 2022).

AutoPET II continued the FDG whole-body lesion-segmentation setting and became notable for organ-aware and multilabel strategies. One AutoPET II paper describes the challenge data as 900 patients and 1,014 studies in the training set, with a held-out test set of 150 studies, including 100 studies from the same hospital as the training data and 50 studies from a different hospital but with a similar acquisition protocol (Murugesan et al., 2023). Another AutoPET 2023 system paper describes a hidden test set of 205 images, with one quarter from the same source distribution and three quarters from a different distribution (Alloula et al., 2023). This suggests that published system descriptions did not always present the same split summary, even when describing the same challenge year.

AutoPET III introduced explicit multitracer, multicenter generalization. Its training data comprised 1,014 FDG PET/CT studies from University Hospital Tübingen and 597 PSMA PET/CT studies from LMU University Hospital Munich, for a total of 1,611 studies. The held-out test set contained 200 studies equally split across four tracer-center conditions: 50 UKT FDG, 50 LMU PSMA, 50 LMU FDG, and 50 UKT PSMA. Two of these were unseen tracer-center compositions, making compositional generalization a benchmark objective rather than a by-product of ordinary domain shift (Dexl et al., 7 May 2026).

The fourth challenge broadened the task definition further. One AutoPET/CT IV system paper describes Task 1 as whole-body PET/CT lesion analysis and Task 2 as longitudinal CT lesion tracking, while another AutoPET IV paper states that the new element is a human-in-the-loop scenario using foreground and background clicks for iterative refinement (Mazher et al., 29 Aug 2025, Huang et al., 2 Sep 2025).

3. Evaluation protocol and ranking logic

The AutoPET series consistently evaluates segmentation with overlap and connected-component-aware burden metrics. In the earlier FDG setting, the official metrics were foreground Dice score, false positive volume (FPV), and false negative volume (FNV). For non-healthy patients, all three metrics were used; for healthy patients, only FPV was considered. Final ranking was based on weighted metric ranks with weights $(0.5, 0.25, 0.25)$ for Dice, FPV, and FNV, respectively (Heiliger et al., 2022).

The challenge metrics were later formalized for autoPET3 as connected-component-aware quantities. With prediction $P$ and ground truth $G$,

$\mathrm{DSC}(P,G)=\frac{2|G\cap P|}{|G|+|P|}.$

If predicted mask $P$ is decomposed into connected components $\{P_l\}_{l=1}^{L_P}$ and ground truth $G$ into connected components $\{G_i\}_{i=1}^{L_G}$, then

$$\mathrm{FPV} = v\sum_{l=1}^{L_P}|P_l|\cdot \mathbf{1}(|P_l\cap G|=0),$$

$$\mathrm{FNV} = v\sum_{i=1}^{L_G}|G_i|\cdot \mathbf{1}(|G_i\cap P|=0),$$

with voxel volume $P$0 and an overlap criterion based on at least one voxel. In autoPET3, DSC and FNV were computed only on lesion-positive cases, while if $P$1, only FPV was evaluated (Dexl et al., 7 May 2026).

For AutoPET 2024 system papers, the ranking formula is written explicitly as

$P$2

In the compositional autoPET3 benchmark, this ranking was applied after computing the three metrics separately on the four tracer-center subsets, averaging subset ranks per metric, and then combining them into the final score (Ahamed, 2024, Dexl et al., 7 May 2026).

A distinctive feature of autoPET III was the addition of a data-centric award category. In that track, architecture was fixed to a baseline model and only data handling, preprocessing, augmentation, synthetic data generation, and related strategies could be modified, isolating the contribution of data-centric decisions from backbone design (Dexl et al., 7 May 2026).

4. Dominant methodological patterns

Across the challenge literature, the dominant technical pattern is 3D PET/CT segmentation with early modality fusion. In autoPET3, seventeen teams submitted twenty-seven algorithms, predominantly nnU-Net-based 3D networks with PET/CT channel concatenation. Most teams used 5-fold ensembling; six teams used TotalSegmentator organ masks; and two broad paradigms emerged: a single global model for both tracers, or tracer-routing systems that first infer tracer identity and then dispatch the case to tracer-specific experts (Dexl et al., 7 May 2026).

Early AutoPET submissions established several motifs that recurred later. One 2022 system combined nn-U-Net and Swin UNETR, augmented by an ensemble of PET maximum-intensity-projection classifiers acting as a gating mechanism. If the MIP ensemble predicted lesion absence, the final output was the empty mask; otherwise, PET and CT were concatenated channel-wise, passed to both segmentation models, and fused by late averaging of softmax outputs. In 5-fold cross-validation on non-healthy cases, this ensemble achieved Dice $P$3, FNV $P$4, and FPV $P$5, improving over standalone nn-U-Net and Swin UNETR (Heiliger et al., 2022).

Another 2022 contribution used a “step-by-step” 3D segmentation strategy in which a standard 3D U-Net predicted one main lesion mask plus seven auxiliary channels derived from progressively dilated lesion-related targets. The paper framed this as encouraging coarse-to-fine localization and reported Dice 0.80 on a 40-patient validation set and 0.92 on a preliminary test set, although it explicitly acknowledged the lack of ablation experiments (Liu et al., 2022).

AutoPET II made organ-aware supervision a prominent theme. A multilabel approach trained a 3D nnUNet to predict lesions together with eight additional organ labels—liver, kidneys, urinary bladder, spleen, lung, brain, heart, and stomach—generated automatically with TotalSegmentator. In held-out experiments, the multilabel model consistently outperformed lesion-only training, for example improving Dice from 0.66 to 0.79 in the larger-data regime while drastically reducing false positive volume, and the paper states that this strategy achieved top ranking in AutoPET II (Murugesan et al., 2023).

The multitracer era amplified tracer-aware routing and anatomy-aware multitask learning. One AutoPET III submission used a tracer classifier based on coronal and sagittal PET MIPs, followed by two tracer-specific nnUNet ensembles with anatomical labels incorporated as a multi-label task; its reported cross-validation Dice scores were 76.90% for FDG and 61.33% for PSMA (Kalisch et al., 2024). Another AutoPET III system posed the “tracer frontier” as a routing problem: a MIP-CNN selected between FDG-specific and PSMA-specific nnUNetv2 ensembles, yielding preliminary-test Dice 74.91%, FNV 40.72, and FPV 0.760, with a marked reduction in false positives relative to a single-model nnUNet baseline (Mesbah et al., 2024).

The winning model-centric autoPET III solution, Team LesionTracer, combined a stronger nnU-Net ResEncL backbone, large 3D patches, PET/CT misalignment augmentation, multimodal CT/MR/PET pretraining, and organ supervision via a second prediction head. In five-fold cross-validation, the reported progression improved Dice from 57.61 for default nnU-Net to 68.40 for the final retrained system, while reducing FPvol from 19.32 to 7.82 and FNvol from 15.69 to 10.35 (Rokuss et al., 2024).

Data-centric approaches became a separate methodological strand. In the fixed-baseline track, a “subtleDA + misalDA” strategy reduced augmentation aggressiveness and added CT-only misalignment augmentation, improving mean Dice from 53.27 to 54.08 and reducing FNvol from 30.90 mL to 24.75 mL, at the cost of increased FPvol (Kovacs et al., 2024). A different AutoPET III paper adapted DiffTumor to generate synthetic PET/CT lesions and showed Dice improvements in all tested transform settings, including 0.5398 to 0.6143 at 15 transforms per image (Chan et al., 2024).

The move to autoPET IV added interactive click conditioning. One system incorporated tracer classification, organ supervision, and simulated foreground/background click maps into nnUNetv2 Residual Encoder models, evaluating behavior from 0 clicks to 10 clicks and showing that stochastic click-count sampling during training produced more stable performance across the interaction spectrum than dense-guidance-only training (Huang et al., 2 Sep 2025).

5. Official benchmark findings

The most comprehensive benchmark-level analysis is available for autoPET3. Across the four tracer-center test conditions, the top-ranked algorithm achieved mean DSC 0.6616, mean FNV 3.1842 mL, and mean FPV 2.7849 mL. Relative to the provided data-centric baseline, this improved DSC from 0.5816 to 0.6616 and reduced FNV from 8.2107 mL to 3.1842 mL. The ranking was reported as stable across 2,000 bootstrap resamples and several alternative ranking schemes (Dexl et al., 7 May 2026).

Performance was strongly condition-dependent. For the winning method, DSC was 0.7702 on UKT FDG, 0.6433 on LMU PSMA, 0.6619 on LMU FDG, and 0.5711 on UKT PSMA. The benchmark analysis concluded that in-domain multitracer PET/CT segmentation is sufficient and probably approaching reader agreement, whereas compositional generalization to unseen tracer-center combinations remains an open problem mainly driven by systematic volume overestimation (Dexl et al., 7 May 2026).

A central scientific result was that case heterogeneity dominated algorithmic differences among top teams. In a linear mixed-effects analysis over the top algorithms, unexplained DSC variance was partitioned as 61% patient heterogeneity, 38% residual, and 1.3% team differences. The same paper states that the expected DSC difference between two randomly selected patients, averaged across algorithms, is roughly seven times as large as the difference between two randomly selected algorithms (Dexl et al., 7 May 2026).

Lesion-level behavior was equally revealing. Detection sensitivity increased with lesion volume and SUV$P$6. Very small lesions below 0.1 mL were detected only roughly 40–60% of the time, whereas lesions with SUV$P$7 were detected above 95%. Frequent false positives included physiologic muscle uptake, injection-site infiltration, lacrimal glands and other head glands in PSMA, bladder-adjacent uptake, atelectasis, and spinal degenerative uptake. The unseen tracer-center compositions were systematically shifted toward oversegmentation, with roughly 1.7 times oversegmentation reported for the out-of-domain datasets (Dexl et al., 7 May 2026).

6. Limitations, controversies, and future directions

The series also exposes structural limitations of PET/CT benchmarking. The autoPET3 challenge report explicitly notes confounding of center and tracer with scanner hardware, reconstruction, annotation style, and demographics. It also acknowledges an annotation shift between the UKT data and the other datasets, possible label errors, and single-annotator bias. A further practical limitation is the five-minute per-case runtime cap, which constrained ensembling and test-time augmentation strategies (Dexl et al., 7 May 2026).

System descriptions across editions often reinforce these limitations from another angle. Many papers report strong cross-validation or preliminary-test results but do not include final hidden-test placement, exhaustive ablations, or complete implementation details. Several also show that aggressive false-positive suppression can increase false-negative burden, especially in tracer-aware systems that favor conservative predictions. This suggests that AutoPET is not merely an overlap-optimization problem but a three-way tradeoff among overlap, missed lesion volume, and spurious lesion volume (Mesbah et al., 2024, Ahamed, 2024).

The next methodological direction in the series is explicit. The autoPET3 overview argues that the remaining frontier is not another near-identical nnU-Net comparison but interactive segmentation and expanded data heterogeneity, including additional tracers and improved labels. autoPET IV operationalizes that shift by introducing a human-in-the-loop scenario in which a clinician can incrementally correct the model using foreground and background clicks, and challenge submissions are evaluated from zero guidance to dense guidance (Dexl et al., 7 May 2026, Huang et al., 2 Sep 2025).

A plausible implication is that the AutoPET Challenge has evolved from a benchmark for fully automatic FDG lesion delineation into a broader research program on tracer-aware, anatomy-aware, and interaction-aware PET/CT segmentation. That trajectory is already visible in the transition from FDG-only lesion masking, to compositional multitracer generalization, to organ supervision, data-centric augmentation, synthetic lesion generation, and click-guided refinement (Heiliger et al., 2022, Rokuss et al., 2024, Huang et al., 2 Sep 2025).