ULS23 Segmentation Model

Updated 7 July 2026

The model’s main contribution is a semi-supervised 3D residual-encoder U-Net that segments lesion-centered VOIs for accurate clinical measurements.
It employs a cropped volume of interest centered on lesion coordinates, preserving key spatial priors vital for RECIST follow-up and radiomic analysis.
Benchmark results demonstrate robust Dice performance with sensitivity to prompt jitter, highlighting challenges in longitudinal lesion tracking.

ULS23 Segmentation Model denotes the lesion-centered universal lesion segmentation systems developed around the ULS23 challenge for computed tomography, most directly the semi-supervised 3D baseline model released with the benchmark and subsequently examined in longitudinal and derivative settings. Its defining formulation is not full-volume segmentation but segmentation from a cropped volume of interest (VOI) centered on a click-point or lesion centroid, with the model returning a 3D mask for the lesion of interest. In the original benchmark, this design was intended to support clinically measured target lesions across chest–abdomen–pelvis CT, including RECIST-style follow-up, volumetry, and registration-based lesion tracking (Grauw et al., 2024).

1. Origin, clinical role, and terminological scope

The ULS23 challenge was introduced to address a gap between organ-specific lesion segmentation benchmarks and routine oncologic CT, where clinically relevant lesions span multiple organs and tissue types. The challenge paper defines universal lesion segmentation as segmentation of the full spectrum of clinically relevant lesions in chest–abdomen–pelvis CT, including difficult pancreatic, colon, and bone lesions, and pairs this scope with a public benchmark and a released baseline model (Grauw et al., 2024).

Clinically, the model family is tied to the radiologist’s target-lesion workflow rather than to exhaustive scene parsing. The challenge motivation explicitly links ULS to repeated lesion measurement for treatment response, automated extraction of long- and short-axis diameters, volumetric and radiomic analysis, and registration-based lesion tracking across time. This lesion-centric framing is important: the benchmark assumes that a target lesion has already been localized sufficiently to define a lesion-centered VOI, so ULS23 is best understood as an interactive segmentation system rather than a detection-free whole-scan segmenter (Grauw et al., 2024).

The term “ULS23” broadened in subsequent literature. In the longitudinal stress test, “the ULS23 segmentation model” refers to the released baseline model deployed on baseline and follow-up CT examinations to assess temporal robustness (Rocholl et al., 25 Jul 2025). In a later multimodal study, “the ULS23 DeepLesion dataset” denotes a distinct 2D lesion-centered slice dataset tailored from DeepLesion for text-conditioned segmentation experiments (Cheng et al., 8 Aug 2025). This suggests that the phrase “ULS23 Segmentation Model” can refer either to the original 3D challenge baseline or, more loosely, to model variants and datasets descended from the ULS23 benchmark ecosystem.

2. Benchmark construction and lesion-centered data model

The original ULS23 benchmark couples a large development corpus with a curated clinical evaluation set. The training corpus combines fully annotated 3D lesion datasets and partially annotated datasets with RECIST-like measurements, for an overall development pool of 38,693 lesions across chest–abdomen–pelvis CT. Fully annotated sources include kidney, liver, abdominal and mediastinal lymph nodes, lung nodules, lung, colon, pancreas, bone, and a DeepLesion3D subset; partially annotated sources include DeepLesion and CCC18 (Grauw et al., 2024).

For evaluation, the challenge assembled target lesions derived from routine clinical reporting. The initial curation comprised 775 lesions from 284 patients drawn from Radboudumc and Jeroen Bosch Hospital, with lesion types including lymph node, kidney, colon, pancreas, bone, lung, liver, peritoneum, breast, adrenal, abdominal wall, and others. After a separate validation carve-out of 16 patients and 50 lesions, the final test set comprised 268 patients and 725 lesions (Grauw et al., 2024).

The benchmark input representation is a lesion-centered 3D VOI of $256 \times 256 \times 128$ voxels in original scan spacing, with resampling disabled during VOI preparation. The center voxel is randomly selected from within the lesion mask to simulate a radiologist’s click, every VOI contains exactly one annotated lesion, and disconnected lesion masks inside the VOI are removed. If cropping exceeds scan bounds, padding uses $(\text{min intensity of VOI} - 1)$ and a padding flag is stored (Grauw et al., 2024).

This lesion-centered input design became one of the most consequential properties of the model family. In later longitudinal analysis, the same centered-VOI assumption was identified as the source of a strong spatial prior: the network had learned to segment the central lesion while disregarding peripheral structures, a behavior that is beneficial for single-timepoint challenge evaluation but fragile under centroid misalignment over time (Rocholl et al., 25 Jul 2025).

3. Baseline architecture and semi-supervised training procedure

The challenge baseline is a semi-supervised 3D residual-encoder U-Net built within the nnU-Net ecosystem. In the challenge paper, the released model is described as nnU-Net v2, specifically a 3D residual-encoder U-Net upscaled relative to the auto-configured baseline, with class ResidualEncoderUNet, base features 32, max features 384, num_pool_per_axis: [5, 6, 6], seven stages of $3 \times 3 \times 3$ convolution kernels, encoder convolution counts $[1, 3, 4, 6, 6, 6, 6]$ , and decoder convolution counts $[1, 1, 1, 1, 1, 1]$ (Grauw et al., 2024).

The documentary description is not fully uniform across papers. The longitudinal stress-test paper characterizes ULS23 as implemented within nnU-Net v1 with a 3D Residual Encoder U-Net, extended to seven encoder–decoder levels with upper feature-channel capacity increased from 320 to 384. It also states that default nnU-Net resampling and input patching were disabled to preserve spatial priors, with the model operating directly on fixed-size VOIs of $256 \times 256 \times 128$ voxels (Rocholl et al., 25 Jul 2025). This suggests that “ULS23” in later usage refers to a released baseline model family whose operational behavior is more stable across papers than every software-version detail.

A defining architectural assumption is that the lesion of interest is centered in the VOI. In the longitudinal study, the model is described as trained on contrast-enhanced CT in portal-venous phase, with VOIs cropped around known or suspected lesion coordinates and the network trained to segment the central lesion while ignoring peripheral structures (Rocholl et al., 25 Jul 2025). In the challenge formulation, the full VOI is processed as one patch, with resampling disabled in both training and inference, explicitly to preserve the cue that the lesion is at the VOI center (Grauw et al., 2024).

The training pipeline is semi-supervised. The challenge baseline first derives pseudo-masks from 2D RECIST-like measurements using GrabCut, with seeds based on an ellipse fitted to the measurements and multiple runs under different intensity normalizations. Stage 1 pretrains nnU-Net for 1000 epochs on 2D pseudo-masks from DeepLesion and CCC18, then fine-tunes for 500 epochs on fully annotated 3D datasets at $25\%$ of the initial learning rate. The resulting model generates 3D pseudo-masks for partially annotated lesions; predictions whose long- and short-axis errors are each $\leq 5$ pixels are retained, yielding 18,264 lesions. Stage 2 then trains a second residual-encoder nnU-Net on these 3D pseudo-masks and fine-tunes again on fully annotated 3D data (Grauw et al., 2024).

The challenge paper does not explicitly list the loss functions, but notes that nnU-Net typically uses a sum of soft Dice loss and cross-entropy. The Dice coefficient is given as

$\mathrm{Dice}(P,G) = \frac{2 \sum_i p_i g_i}{\sum_i p_i + \sum_i g_i}.$

The longitudinal paper similarly states that specific loss functions and CT intensity preprocessing are not reported there, although nnU-Net defaults would commonly imply combinations of Dice and cross-entropy losses and intensity normalization; those details are explicitly identified as unspecified in that paper (Grauw et al., 2024, Rocholl et al., 25 Jul 2025).

4. Evaluation methodology and baseline performance

ULS23 evaluates segmentation quality, diameter measurement accuracy, and click robustness jointly. The benchmark ranking metric is the ChallengeScore,

$CS = 0.8 \cdot SP + 0.05 \cdot LAE + 0.05 \cdot SAE + 0.1 \cdot SCS,$

where $(\text{min intensity of VOI} - 1)$ 0 is average 3D Dice, $(\text{min intensity of VOI} - 1)$ 1 and $(\text{min intensity of VOI} - 1)$ 2 are symmetric mean absolute percentage errors for long- and short-axis diameters, and $(\text{min intensity of VOI} - 1)$ 3 is a segmentation consistency score computed as Dice between re-aligned predictions for the same lesion when VOI center coordinates vary (Grauw et al., 2024).

On the ULS23 test set of 725 lesions, the baseline nnUnet-ResEnc+SS achieved Dice $(\text{min intensity of VOI} - 1)$ 4, long-axis SMAPE $(\text{min intensity of VOI} - 1)$ 5, short-axis SMAPE $(\text{min intensity of VOI} - 1)$ 6, SCS Dice $(\text{min intensity of VOI} - 1)$ 7, and ChallengeScore $(\text{min intensity of VOI} - 1)$ 8 (Grauw et al., 2024). The same paper reports nnUnet at Dice $(\text{min intensity of VOI} - 1)$ 9, nnUnet-ResEnc at $3 \times 3 \times 3$ 0, and, with test-time augmentation enabled, nnUnet-ResEnc+SS at Dice $3 \times 3 \times 3$ 1 (Grauw et al., 2024).

Performance varied by lesion type. Reported test-set Dice values were $3 \times 3 \times 3$ 2 for lung, $3 \times 3 \times 3$ 3 for liver, $3 \times 3 \times 3$ 4 for lymph node, $3 \times 3 \times 3$ 5 for pancreas, $3 \times 3 \times 3$ 6 for kidney, $3 \times 3 \times 3$ 7 for colon, and $3 \times 3 \times 3$ 8 for bone, with some categories marked as small-sample subsets (Grauw et al., 2024). The paper attributes failures to under-segmentation of large lesions, low contrast in pancreas and colon lesions, irregular shapes, subjective boundary definitions, and confluent lesions.

The challenge also enforced practical runtime constraints. Participants had to submit a Dockerized automatic pipeline operating on a Grand Challenge environment with an NVIDIA T4, 8 vCPU, and 32 GB RAM, and the job had to process 100 lesions within 9 minutes. The released baseline disabled test-time augmentation for this setting and reported inference in less than 2 seconds per VOI on an NVIDIA RTX 2080 Ti (Grauw et al., 2024). A common misconception is therefore that ULS23 is solely a pure-accuracy benchmark; in fact, its design explicitly balances segmentation quality with measurement fidelity, consistency, and clinical runtime.

5. Longitudinal deployment and the centered-lesion failure mode

A major later analysis examined whether the single-timepoint ULS23 model could be used for longitudinal lesion analysis. Using a public University Hospital Tübingen dataset of 300 scans from metastatic melanoma patients undergoing systemic therapy, with manual 3D segmentations of all visible malignant lesions at baseline and follow-up, the study evaluated ULS23 under a baseline-to-follow-up workflow in which lesion centroids were propagated by conventional image registration and used to center the follow-up VOI (Rocholl et al., 25 Jul 2025).

The longitudinal pipeline first segmented each lesion-centered VOI and then assigned a lesion of interest by connected-component selection. If $3 \times 3 \times 3$ 9 is the set of connected components, $[1, 3, 4, 6, 6, 6, 6]$ 0 their centroids, and $[1, 3, 4, 6, 6, 6, 6]$ 1 the VOI center, the selected component was

$[1, 3, 4, 6, 6, 6, 6]$ 2

Registration error for propagated follow-up centroids was measured as

$[1, 3, 4, 6, 6, 6, 6]$ 3

and per-lesion segmentation accuracy used the Dice similarity coefficient

$[1, 3, 4, 6, 6, 6, 6]$ 4

The study identified two interconnected failure modes. First, follow-up segmentation quality degraded sharply when inter-scan registration errors displaced the true lesion away from the VOI center. The registration-error histogram showed a long tail, with a non-trivial number of lesions displaced by more than 10 mm, and follow-up Dice scores were significantly lower than baseline by Wilcoxon signed-rank test ( $[1, 3, 4, 6, 6, 6, 6]$ 5) (Rocholl et al., 25 Jul 2025). Second, once segmentation failed, lesion correspondence also failed: the combined rate of incorrect assignments plus false negatives was already 26.3% at baseline and worsened to 31.8% at follow-up, while true negatives increased from 0% at baseline to 31% at follow-up because some lesions had resolved (Rocholl et al., 25 Jul 2025).

A controlled displacement experiment isolated the effect of off-centering. Thirty lesions with the highest Dice in the longitudinal evaluation were re-cropped after rigid translations of magnitude $[1, 3, 4, 6, 6, 6, 6]$ 6 mm, using

$[1, 3, 4, 6, 6, 6, 6]$ 7

The results showed a threshold-like collapse rather than a smooth decline: at $[1, 3, 4, 6, 6, 6, 6]$ 8 mm, 100% of assignments were correct; by $[1, 3, 4, 6, 6, 6, 6]$ 9 mm, correct assignments fell below 50%; around $[1, 1, 1, 1, 1, 1]$ 0 mm, Dice collapsed to near zero across lesions; and beyond $[1, 1, 1, 1, 1, 1]$ 1 mm, correct assignments were virtually absent and false negatives dominated (Rocholl et al., 25 Jul 2025). The paper interprets this as an “unstable prompt” effect: the propagated centroid acts as a prompt, and moderate prompt jitter can drive the model outside its effective center radius.

This finding substantially reframed the ULS23 baseline. High single-timepoint challenge performance did not imply temporal robustness. The paper concludes that robust oncological tracking requires moving away from cascades of registration, prompting, single-timepoint segmentation, and geometric correspondence toward integrated end-to-end models designed for temporal analysis (Rocholl et al., 25 Jul 2025).

6. Later adaptations, derivative models, and comparative research

Subsequent work extended the ULS23 ecosystem in different directions rather than simply reusing the original baseline unchanged. One line of work, ULS+, retained the click-centered 3D paradigm but altered the data and input scale. ULS+ uses a 3D nnU-Net v2 residual encoder, “size L,” reduces the input VOI from $[1, 1, 1, 1, 1, 1]$ 2 to $[1, 1, 1, 1, 1, 1]$ 3, adds fully annotated public datasets, omits the original weakly supervised pretraining, and introduces train-time click-point augmentation by sampling two additional points within each lesion mask (Weber et al., 6 Jan 2026). On the ULS23 test set, ULS improved from Dice $[1, 1, 1, 1, 1, 1]$ 4 and robustness $[1, 1, 1, 1, 1, 1]$ 5 to ULS+ Dice $[1, 1, 1, 1, 1, 1]$ 6 and robustness $[1, 1, 1, 1, 1, 1]$ 7; on a Longitudinal-CT subset, the change was from Dice $[1, 1, 1, 1, 1, 1]$ 8 and robustness $[1, 1, 1, 1, 1, 1]$ 9 to Dice $256 \times 256 \times 128$ 0 and robustness $256 \times 256 \times 128$ 1, with all improvements statistically significant at $256 \times 256 \times 128$ 2 (Weber et al., 6 Jan 2026).

A second direction used ULS23-branded data in a different problem formulation. “Text Embedded Swin-UMamba for DeepLesion Segmentation” operates on 2D lesion-centered $256 \times 256 \times 128$ 3 crops from a tailored ULS23 DeepLesion dataset, adds short-form report text, and uses a Swin-UMamba backbone with BioLord-based text embeddings fused into the decoder at five stages (Cheng et al., 8 Aug 2025). On its test set, it reported Dice $256 \times 256 \times 128$ 4, Jaccard $256 \times 256 \times 128$ 5, Hausdorff95 $256 \times 256 \times 128$ 6 pixels, sensitivity $256 \times 256 \times 128$ 7, and specificity $256 \times 256 \times 128$ 8; it outperformed LanGuideMedSeg by 37% in Dice and slightly exceeded image-only xLSTM-UNet and 2D nnUNet baselines (Cheng et al., 8 Aug 2025). This is not a direct replacement for the original ULS23 baseline, because the task is slice-level 2D segmentation on lesion-centered crops rather than 3D VOI segmentation in the challenge setting.

A third strand is comparative algorithmic exploration within the challenge context. One comparative study screened nnUNetv2, DeepLabV3, DeepLabV3+, Medical Transformer, SwinUnet, and TransUNet, primarily on Bone and then Bone+Pancreas subsets. It reports nnUNetv2 baseline Dice $256 \times 256 \times 128$ 9 on Bone, DeepLabV3 (2D) Dice $25\%$ 0, DeepLabV3+ (3D) Dice $25\%$ 1 after 25 epochs, SwinUnet overfitting without a final reported Dice, and TransUNet achieving training and validation Dice $25\%$ 2 on Bone and $25\%$ 3 on Bone+Pancreas (Shi et al., 14 Feb 2025). The same study emphasizes memory difficulty when adapting some transformer architectures to 3D VOIs of $25\%$ 4, with attempted Medical Transformer configurations exceeding 100 GB RAM (Shi et al., 14 Feb 2025).

System	Formulation	Representative reported result
ULS23 baseline	3D lesion-centered VOI segmentation	Dice $25\%$ 5, ChallengeScore $25\%$ 6 (Grauw et al., 2024)
ULS23 in longitudinal use	3D baseline applied to baseline/follow-up tracking	Incorrect + FN rises from 26.3% to 31.8% (Rocholl et al., 25 Jul 2025)
ULS+	3D click-centered adaptation with smaller VOI	ULS23 Dice $25\%$ 7, robustness $25\%$ 8 (Weber et al., 6 Jan 2026)
Text-Swin-UMamba	2D lesion-centered crop segmentation with report text	Dice $25\%$ 9, HD95 $\leq 5$ 0 px (Cheng et al., 8 Aug 2025)

Taken together, these studies show that “ULS23 segmentation model” is no longer a single immutable system. It denotes a benchmark-centered lineage whose members differ in dimensionality, prompting regime, training data, and intended clinical use.

7. Limitations, recurrent misconceptions, and future directions

Several limitations recur across the literature. The most fundamental is the centered-lesion prior. The original challenge design intentionally simulates a radiologist click within a lesion-centered VOI, and the longitudinal paper shows that this prior can become a liability when applied to follow-up scans with imperfect propagated centroids. A common misconception is therefore that universal lesion segmentation implies robust lesion tracking; the longitudinal evidence shows that single-timepoint ULS23 is highly sensitive to prompt or ROI jitter and can fail sharply beyond moderate mis-centering (Grauw et al., 2024, Rocholl et al., 25 Jul 2025).

A second limitation is incomplete specification or reproducibility of some implementation details. The longitudinal paper explicitly states that loss functions and CT intensity preprocessing are not reported there; the challenge paper likewise does not explicitly list optimizer and training-schedule details beyond what nnU-Net defaults imply. The longitudinal study also treats registration as a black box, noting that the conventional image-registration method used to propagate centroids is unspecified, so error magnitudes cannot be attributed to rigid, affine, or deformable choices (Grauw et al., 2024, Rocholl et al., 25 Jul 2025).

A third limitation is dataset scope and generalizability. The original benchmark is chest–abdomen–pelvis CT with curated clinical target lesions; the longitudinal evaluation is limited to 300 melanoma scans under a standardized portal-venous protocol; the text-conditioned 2D study uses lesion-centered crops that simplify the task relative to full $\leq 5$ 1 DeepLesion slices; and ULS+ notes that its smaller VOI may not capture very large or elongated lesions in one pass (Rocholl et al., 25 Jul 2025, Cheng et al., 8 Aug 2025, Weber et al., 6 Jan 2026). These constraints do not invalidate the reported results, but they delimit the regimes in which each reported number should be interpreted.

The principal future direction is convergence toward integrated temporal or end-to-end systems. The longitudinal paper advocates models that jointly learn detection, segmentation, correspondence, and even registration, citing possible mechanisms such as multi-timepoint nnU-Net extensions, 4D spatio-temporal encoders, learned deformable alignment, and deformable attention (Rocholl et al., 25 Jul 2025). ULS+ advances robustness within the click-centered paradigm through data-driven adaptation and click augmentation rather than through a full temporal redesign (Weber et al., 6 Jan 2026). The 2D multimodal work points in yet another direction, namely injecting report-derived semantics into decoder refinement (Cheng et al., 8 Aug 2025). A plausible implication is that future ULS-derived systems will combine these lines—improved data curation, prompt robustness, temporal coupling, and multimodal conditioning—rather than relying on any single mechanism alone.