MedSeqFT: Sequential Fine-Tuning in 3D Segmentation

• MedSeqFT is a sequential fine-tuning framework for 3D medical image segmentation that preserves generalizable representations while adapting to new clinical tasks.
• It employs Maximum Data Similarity (MDS) selection to maintain connection to the pre-training distribution, ensuring stable adaptation and preventing catastrophic forgetting.
• K&G RFT integrates full fine-tuning with LoRA-based knowledge distillation to balance task-specific performance with retention of fundamental knowledge, improving Dice and reducing HD95.

MedSeqFT is a sequential fine-tuning framework designed to maximize the adaptability and knowledge retention of 3D medical image segmentation foundation models in settings where new clinical tasks arise over time. Unlike parallel fine-tuning, which neglects knowledge transfer, or classical multi-task approaches, which require all datasets simultaneously and resist incremental updates, MedSeqFT enables progressive task adaptation while preserving the original model's generalizable representations. It introduces two primary innovations: Maximum Data Similarity (MDS) selection to maintain connection to the pre-training distribution and Knowledge and Generalization Retention Fine-Tuning (K&G RFT), a LoRA-based distillation scheme that aligns task-specific performance with knowledge stability.

1. Sequential Fine-tuning Challenges in Medical Image Segmentation

Sequential integration of new segmentation tasks is essential in clinical environments where datasets for novel conditions or anatomical regions become available incrementally. Standard fine-tuning strategies often induce catastrophic forgetting, wherein previously acquired representations and generalization ability degrade as the model adapts to new data. This challenge is compounded in the domain of 3D imaging, where the diversity and scale of medical datasets demand efficient and knowledge-preserving adaptation protocols.

MedSeqFT directly addresses these limitations by structuring fine-tuning as an iterative, knowledge-aware process, in contrast to naive full fine-tuning (FFT) and prior parameter-efficient fine-tuning (PEFT) methods. Its design is informed by the necessity of maintaining high performance across both experienced and unseen segmentation tasks while minimizing the retraining burden.

2. Maximum Data Similarity (MDS) Selection

The MDS component establishes a mechanism for continual connection to the initial pre-training distribution. For each downstream dataset $D_t$, all training samples are passed through the original self-supervised pre-trained model $M_0$ with its associated SSL (self-supervised learning) head. The SSL loss for each sample reflects its similarity to the pre-training regime. To mitigate stochastic fluctuations inherent in SSL (e.g., due to random masking), the loss is computed as an average over 1000 runs per sample.

The $K$ samples with the lowest average SSL loss are retained in a buffer $B_t$ and serve as representatives of the original distribution during subsequent task adaptation. This sample selection ensures that, as the model is fine-tuned on task $t+1$, it continuously encounters data that reinforce general feature representations, acting as an implicit regularizer against overspecialization.

Step	Process	Role
Forward Pass	Compute SSL loss via $M_0$ for all samples	Quantifies distributional similarity
Averaging	Repeat loss computation for 1000 runs per sample	Reduces effect of SSL randomness
Buffering	Retain $K$ samples with lowest average SSL loss	Preserves most representative samples

3. Knowledge and Generalization Retention Fine-Tuning (K&G RFT)

K&G RFT is a hybrid mechanism combining full fine-tuning, knowledge distillation, and LoRA (Low-Rank Adaptation) to balance adaptation to new segmentation tasks with retention of general knowledge.

KD-based Full Fine-Tuning (KD-FFT):

• For current task $D_t$, the model $M_{t-1}$ (encoder $E_{t-1}$ and decoder) initializes $M_{t-init}$.
• $M_{t-init}$ undergoes full fine-tuning on $D_t$ using a segmentation loss $L_{seg}$ (Dice + cross-entropy).
• Simultaneously, $E_{t-1}$ is frozen, and buffered samples are passed through both $E_{t-init}$ and $E_{t-1}$. A distillation loss $L_{kd}$ (MSE) between their outputs constrains $E_{t-init}$ to retain prior representational capacity.

LoRA-based Knowledge Distillation:

• LoRA modules (low-rank adapters) are inserted in $E_{t-1}$’s linear layers.
• A secondary distillation trains these modules using $E_{t-mid}$ (the KD-FFT-updated encoder) as teacher, optimizing an MSE-based loss $L_{refine}$.
• All modifications are confined to the LoRA parameters, with the encoder backbone frozen.

Reparameterization:

• The learned LoRA deltas are consolidated into the encoder backbone by updating weights:

$W' = W + BA$

where $W \in \mathbb{R}^{d \times k}$ is the base weight matrix, and $$A \in \mathbb{R}^{r \times k},\ B \in \mathbb{R}^{d \times r}$$ are the low-rank matrices, with $r \ll \min(d, k)$.

This process ensures that essential general representations survive the adaptation to new tasks, and the marginal parameter growth inherent in LoRA is collapsed back into the base model, keeping the overall parameter count stable.

4. Empirical Performance and Robustness

MedSeqFT yields consistent improvements across multi-task datasets comprising 10 segmentation tasks in both CT and MRI modalities. When compared to FFT and various PEFT baselines, MedSeqFT delivers an average increase of 3.0% Dice Similarity Coefficient (DSC) and a 10 mm reduction in 95th Percentile Hausdorff Distance (HD95). These gains are observed not only on seen tasks but are particularly notable for transfer to clinically significant unseen targets such as COVID-19-20 lung lesions and complex kidney/tumor segmentation.

Visualization of loss landscapes demonstrates that MedSeqFT-trained models exhibit smoother, less rugged loss surfaces relative to FFT, indicating improved optimization stability and robustness. Parameter variation analysis shows that task-specific adaptation via MedSeqFT is primarily localized to deeper encoder layers, while shallow layers—crucial for general features—remain comparatively unchanged. The maximum mean parameter change is on the order of $0.012$–$0.016$, supporting the claim that MedSeqFT implements refinement, not wholesale reparameterization.

5. Technical Structure and Loss Formulations

The MedSeqFT pipeline closely integrates several loss components:

• Segmentation Loss: $L_{seg} = L_{Dice} + L_{CE}$, where $L_{Dice}$ is the Dice loss and $L_{CE}$ is voxelwise cross-entropy.
• Knowledge Distillation Loss: For each buffered sample, $L_{kd}$ is the mean squared error (MSE) between current and previous encoder outputs.
• LoRA-based Refinement Loss: $L_{refine}$ is the MSE between LoRA-adapted outputs and teacher outputs from the first KD-FFT stage.
• Reparameterization: After training LoRA modules, the weights are merged as $W' = W + BA$.

Evaluation is performed using Dice and HD95 metrics, assessing both volume overlap and boundary precision.

6. Clinical Relevance and Extension

Sequential, knowledge-retentive adaptation is vital in medical imaging where segmentation requirements are not static, but evolve as annotation efforts mature and as new modalities or indications enter clinical practice. MedSeqFT’s preservation of general knowledge enables continual learning without recurrent catastrophic forgetting, improving both immediate performance and the foundation for future extension.

Future research directions include optimizing which network layers are refined in each task transition, extending the scheme to explicitly multi-modal imaging data, and integrating more advanced or hybrid PEFT approaches to further minimize retraining costs. Fine-grained analysis of representation dynamics during MedSeqFT is also poised to yield deeper understanding of continual learning processes in high-dimensional medical vision models.

7. Summary

MedSeqFT establishes a sequential fine-tuning paradigm that uses Maximum Data Similarity selection to anchor the evolving model to its pre-training distribution and a dual-stage, LoRA-based distillation (K&G RFT) to ensure task-specific adaptation does not come at the expense of knowledge generalization. Empirical results across multi-task and transfer segmentation scenarios confirm consistent improvements in segmentation accuracy, robustness, and transferability. This framework provides a foundation for continual clinical deployment of 3D medical image segmentation models as new requirements and datasets emerge (Ye et al., 7 Sep 2025).