Exploiting Longitudinal Context in Clinician-Verified Interactive Lesion Tracking

Abstract: Tracking tumor lesions across serial CT scans is essential for oncological response assessment. Existing automated methods face a fundamental trade-off: end-to-end trackers achieve high automation but offer no opportunity to correct silent tracking failures, while decoupled registration-segmentation pipelines permit user verification yet discard the lesion's prior appearance, limiting accuracy in ambiguous cases. In this work, we propose a Verified Tracking paradigm: a clinician verifies a registration-proposed prompt, which the model leverages alongside the baseline lesion appearance to resolve segmentation ambiguities. We present a unified framework combining early spatial prompt fusion with latent temporal difference weighting for longitudinally-informed segmentation. To address data scarcity, we leverage large-scale synthetic pretraining, proving essential for exploiting longitudinal context, improving performance by up to 4.5 Dice points over training from scratch. Our approach secured first place in the MICCAI autoPET IV challenge. We further curate and release PanTrack, a new longitudinal pancreatic cancer benchmark, to assess out-of-distribution generalization. Experiments show that our model outperforms prior work in both fully automatic and the proposed verified tracking setting offering a clinically safe middle ground between automation and control. Code, model and dataset will be released at https://github.com/MIC-DKFZ/LongiSeg

• The paper presents a verified tracking paradigm that integrates automated registration with clinician corrections to exploit both baseline and longitudinal context for lesion tracking.
• It employs a two-stage framework using uniGradICON for follow-up lesion localization and Difference Weighting Blocks for effective temporal feature fusion.
• Synthetic longitudinal pretraining boosts performance by up to 4.5 Dice points and demonstrates robust generalization across datasets like autoPET IV and PanTrack.

Summary of "Exploiting Longitudinal Context in Clinician-Verified Interactive Lesion Tracking"

Motivation and Context

Automated lesion tracking across serial CT scans is critical for oncological response assessment, yet existing workflows face persistent challenges: black-box end-to-end trackers preclude clinician correction and decoupled registration-segmentation pipelines discard vital longitudinal information, resulting in frequent errors especially in ambiguous follow-up cases. Point-based trackers lack volumetric context, and automated longitudinal models are not promptable—blocking any meaningful clinician interaction. The paper introduces the "Verified Tracking" paradigm, a workflow designed to combine the robustness of automation with clinician oversight and explicit longitudinal context.

Verified Tracking Paradigm and Framework

The proposed framework fuses registration-based lesion retrieval and interactive verification with a longitudinally-informed segmentation network. The process is two-stage: (1) registration (via uniGradICON) predicts the follow-up lesion location from an initial baseline center; (2) the clinician verifies/corrects the proposal, and the model segments the lesion utilizing both the verified follow-up prompt and the baseline appearance.

Figure 1: Schematic of the framework combining registration-proposed/corrected prompts, shared encoder, explicit temporal fusion via Difference Weighting, and longitudinally-informed segmentation.

This approach directly satisfies three criteria: leveraging the baseline lesion appearance (longitudinal prior), enabling clinician correction for tracking failures, and ensuring robustness to mislocalized prompts due to registration error or manual correction. The segmentation architecture employs early pixel-level prompt fusion (concatenating image and spatial heatmap) and latent temporal fusion via Difference Weighting Blocks (DWB) inserted at all U-Net skip connections. DWB computes instance-normalized feature differences and gates them to emphasize temporal changes, preventing cross-sectional collapse and forcing the model to attend to longitudinal change.

Synthetic Longitudinal Pretraining and Prompt Simulation

The architecture's efficacy is contingent on large-scale synthetic longitudinal pretraining given real annotated multi-timepoint CT data scarcity. The synthetic dataset simulates tumor growth, shrinkage, and acquisition variability, with prompts generated both on-lesion and from registered centers to train robust localization across imperfect prompts. Without synthetic pretraining, longitudinal architectures default to single-timepoint operations, discarding the baseline scan and failing to exploit temporal context.

Datasets and Benchmarks

Experimental validation utilizes the autoPET IV dataset (serial whole-body CT for melanoma with challenging lesion changes, 670 scans) and introduces PanTrack—a new longitudinal pancreatic cancer benchmark (45 patients, 161 scans)—to rigorously test out-of-distribution (OOD) generalization, addressing the lack of publicly available lesion-level, multi-timepoint CT annotation.

Experimental Results and Ablations

A systematic ablation on autoPET IV reveals strong numerical results:

• Naive longitudinal fusion underperforms single-timepoint baselines when trained from scratch (54.0 Dice vs. 55.8 Dice), exposing the necessity for architectural and data design.
• Synthetic longitudinal pretraining activates the architecture, yielding performance improvements (up to 4.5 Dice points, 69.1→70.6 NSD, 75.8→77.7 LDR).
• Difference Weighting Blocks maximize the longitudinal prior by explicitly gating features, achieving the highest accuracy (58.5 Dice, 72.3 NSD).
• Prompt simulation is critical; reliance on perfect center prompts leads to catastrophic degradation on realistic off-center scenarios.

Comparison with baselines demonstrates strong numerical superiority:

• In automatic tracking (no user correction), the proposed method achieves 60.7 Dice on autoPET IV and 58.2 Dice on PanTrack, outperforming decoupled and end-to-end competitors.
• In verified tracking (clinician-corrected prompts), the framework attains 73.7 Dice (autoPET IV) and 60.0 Dice (PanTrack), outstripping promptable foundation models (e.g., ULS: 68.7 Dice, 53.4 Dice).

The framework exhibits exceptional robustness: on PanTrack, automatic performance surpasses competing verified models, suggesting OOD generalization and reliable tracking even with domain shift and difficult lesion boundaries. The margin gained from longitudinal context is visually confirmed by qualitative comparisons: baseline-only models fail in ambiguous volumetric scenarios, while longitudinal fusion accurately isolates lesions bordering complex anatomy.

Figure 2: Qualitative comparison on autoPET IV and PanTrack; longitudinal context resolves ambiguity in lesion boundaries that single-timepoint models misclassify.

Practical and Theoretical Implications

This paradigm shifts the clinical workflow by integrating accurate, safe lesion tracking with minimal clinician burden (prompt verification/correction). Catastrophic retrieval failures are averted, and the model reliably handles topological changes (splitting, merging, vanishing lesions), isolating delineation as the primary error source. The paper establishes synthetic pretraining and temporal fusion as universal requirements for exploiting multi-timepoint information. The release of PanTrack significantly lowers the barrier for development and evaluation of robust lesion tracking models, fostering rigorous cross-domain testing.

Future directions include prospective clinical reader studies, investigation of richer prompt modalities (free-text, scribbles), and extension to RECIST 1.1 workflows and broader 3D segmentation foundation models.

Conclusion

The paper presents a rigorously validated, promptable lesion tracking framework unifying registration, clinician verification, and longitudinal context fusion. The method is numerically demonstrated to outperform both decoupled and end-to-end baselines in both automatic and interactive tracking modes, with strong generalization to new anatomical domains. The proposed paradigm delivers robust, clinically safe automation while laying the groundwork for further advances in AI-driven longitudinal imaging analysis. Release of the PanTrack dataset and model weights is poised to accelerate future developments in generalizable, promptable medical image segmentation.