Seg-TTO: Test-Time Optimization for Segmentation

• Seg-TTO is a framework that dynamically adapts segmentation models using per-sample, on-the-fly optimization with unsupervised or weakly supervised losses.
• It refines model parameters through lightweight updates at inference time, improving robustness against domain shifts, unseen categories, and prompt variations.
• Applied in medical imaging and vision-language tasks, Seg-TTO significantly boosts performance metrics such as Dice Similarity Coefficient and Intersection-over-Union.

Segmentation by Test-Time Optimization (Seg-TTO) refers to a family of frameworks that enable segmentation models—particularly in contexts requiring adaptation to imaging, domain, or prompt shifts—to perform sample-specific or session-specific adaptation at inference time, using only local data or immediate supervision. By leveraging unsupervised or weakly-supervised objectives, Seg-TTO methods dynamically refine segmentation networks’ parameters or prompts to boost robustness and accuracy, especially under distributional shift, unseen categories, or user-specific constraints. The framework is influential in both medical imaging, such as adaptive radiotherapy, and vision-language segmentation, where distributional generalization, sample-wise adaptation, and open-world recognition are critical.

1. Core Principles of Seg-TTO

Seg-TTO architectures operationalize test-time optimization for segmentation by refining model parameters or prompt representations at inference using only the current sample or immediate context. A prototypical Seg-TTO pipeline includes:

• Deployment of a pre-trained segmentation model, potentially population-based or zero-shot, as the base.
• Sample-specific adaptation, involving one or more gradient-based optimization steps performed on the model using unsupervised (e.g., entropy minimization, image similarity) or weak/self-supervised losses constructed from the test data.
• Objective regularization, typically achieved through smoothness, deformation field regularization, or prompt ensemble variance to prevent overfitting to spurious statistics.
• Lightweight update schemes, such as tuning only prompt encoders, LayerNorm parameters, or context vocabularies, to maintain computational feasibility (Liang et al., 2022, Noori et al., 28 May 2025, Silva et al., 8 Jan 2025).

Unlike classic domain adaptation, no access to labeled or unlabeled training data from the target domain is assumed at adaptation; the optimization is performed “online” on-the-fly, per sample or image sequence.

2. Seg-TTO Methodologies Across Applications

Medical Image Segmentation (DIR-based Seg-TTO)

The classical instantiation for online adaptive radiotherapy employs a deep unsupervised deformable image registration (DIR) network, such as Voxelmorph or cascaded VTN, pre-trained on population data. At test time, Seg-TTO initiates from these population weights $\theta_0$ and performs sample-specific gradient descent over an image similarity loss (MSE or NCC) combined with deformation field regularization:

$$L_{TTO}(\theta) = \alpha \cdot L_{sim}(I_f, I_m\circ f_\theta(I_m, I_f)) + \beta \cdot L_{reg}(f_\theta(I_m, I_f))$$

Iterative optimization is stopped upon loss-convergence, yielding individualized weights $\theta^*$ for the patient or session. For later sessions, intra-patient TTO reuses $\theta^*$ as the initialization, reducing both adaptation time and further improving accuracy (Liang et al., 2022).

Interactive Prompt-based Segmentation

In prompt-driven models such as SAM, Seg-TTO can manifest as per-sample, click-driven adaptation. The DC-TTA framework partitions user interaction cues (clicks) into coherent units, optimizing a prompt encoder and mask decoder for each subset, and merging the resulting models and masks via task-vector arithmetic. The per-unit loss incorporates click-level and mask-level pseudo-supervision, with adaptation steps anchored to user-provided guidance (Kim et al., 29 Jun 2025).

Open-Vocabulary and Vision-LLMs

Here, Seg-TTO is used for open-vocabulary semantic segmentation (OVSS), repurposing VLMs such as CLIP for dense prediction. Approaches such as MLMP-SegTTO jointly minimize pixel-wise and global (CLS token) entropy across multiple layers and text-prompt variants:

$$L_{SegTTO}(\theta) = \frac{1}{T} \sum_{t=1}^T \left[H_{local}^t(\theta) + H_{CLS}^t(\theta)\right]$$

Only normalization parameters (e.g., LayerNorms) are adapted, with no new data or ground-truth required. Other Seg-TTO frameworks employ self-supervised objectives mixing entropy losses and pseudo-label cross-entropy over augmented image views, while adaptively refining text and visual prompts (Noori et al., 28 May 2025, Silva et al., 8 Jan 2025).

Open-World and Incremental Adaptation

Seg-TTO also includes segmentation-assisted incremental test-time adaptation, where segmentation maps generated from patch-level CLIP similarities guide active labeling of uncertain or previously unseen classes, amplifying adaptation to emerging categories under constrained labeling budgets (Sreenivas et al., 27 Aug 2025).

3. Algorithmic Structure and Test-Time Workflow

Representative Seg-TTO workflows share the following steps:

1. Initialization: Start from a pre-trained population or zero-shot segmentation model.
2. Data Preparation: Ingest the current test sample, possibly with local augmentations or user interactions (prompts, clicks).
3. Auxiliary Signal Construction: Generate supervision via image similarity metrics, patch-level segmentation maps, user feedback, or self-supervised objectives such as entropy minimization or pseudo-labels.
4. Parameter Update: Backpropagate unsupervised/weakly-supervised loss via a lightweight optimizer (typically Adam or AdamW); update only a small subset of network parameters (e.g., prompt encoders, normalization layers, context vectors) or entire models in select cases.
5. Stopping Criteria: Use explicit convergence thresholds on loss change (e.g., $\delta = 0.005$ for DIR) or fixed adaptation steps.
6. Prediction: Apply the adapted parameters to obtain the segmentation map.
7. Progressive/Incremental Adaptation: For sequential sessions (e.g., patient fractions), reuse adapted weights to further minimize runtime.

A selection of Seg-TTO instantiations and their key characteristics is shown in the following table:

Application Domain	Adapted Components	Loss/Objective Function
CBCT DIR (Liang et al., 2022)	Full network weights	Image similarity (MSE/NCC) + DVF regularization
Prompted IS (Kim et al., 29 Jun 2025)	Prompt encoder & mask decoder	BCE on pseudo-masks and user clicks
OVSS (MLMP) (Noori et al., 28 May 2025)	LayerNorm in ViT vision encoder	Multi-layer, multi-prompt entropy minimization
OVSS (domain) (Silva et al., 8 Jan 2025)	Text/visual prompts, contexts	Patch-level SSL: entropy + pseudo-label CE
Open-world ITTA (Sreenivas et al., 27 Aug 2025)	N/A; segmentation selects oracles	Patchwise VLM segmentation guides AL

4. Performance and Benchmarks

In medical adaptive radiotherapy, Seg-TTO achieves substantial quantitative improvements. For example, on 17 head-and-neck structures in 39 patients, Voxelmorph with TTO shows up to +0.04 (5%) increase in Dice Similarity Coefficient (DSC) and up to –0.98 mm (25%) reduction in 95% Hausdorff Distance (HD95), with adaptation times of approximately four minutes for first fraction and one minute for subsequent fractions (Liang et al., 2022).

In interactive segmentation, DC-TTA consistently reduces the number of user clicks required to reach 90% IoU (“NoC$_{90}$”) and the failure rate (“FR$_{90}$”) across challenging datasets, outstripping zero-shot SAM and conventional TTA baselines by a margin of 5–15% in NoC and up to 20 percentage points in failure rate (Kim et al., 29 Jun 2025).

For OVSS, MLMP-based Seg-TTO increases mean Intersection-over-Union (mIoU) by 7.9 points on Pascal-VOC20 and 8.6 points under synthetic corruptions, with consistent gains across seven multi-domain datasets and robustness to single-sample adaptation (Noori et al., 28 May 2025). Domain-adaptive Seg-TTO yields up to 4.5 absolute mIoU points on hard-shifted medical or industrial datasets (Silva et al., 8 Jan 2025).

Open-world incremental approaches leveraging segmentation as an active-labeling heuristic improve harmonic mean class accuracy (HM) and reduce detection delay (ICDD) across DomainNet and ImageNet variants, outperforming random, MSP, entropy, and margin query methods (Sreenivas et al., 27 Aug 2025).

5. Limitations and Practical Constraints

Common limitations observed in Seg-TTO research include:

• Compute Overhead: Test-time adaptation, especially per-sample gradient updates, introduces runtime and memory costs, although limiting updates to small model subsets and warm-starting from population weights mitigate this.
• Hyperparameter Robustness: Performance is sensitive to loss weighting, adaptation step-size, click thresholding (for IS), IoU/entropy cutoffs (for unit assignment or view selection), and temperature scaling. Hyperparameter tuning or dynamic strategies may be necessary for deployment.
• Label-free or Weak Supervision Constraints: All Seg-TTO variants eschew labeled adaptation data, depending on the quality of pseudo-supervision or self-supervised objectives. Poor click quality, ambiguous prompts, or confounding self-supervised minima may reduce gains.
• Open-world Detection Misses: Segmentation-driven active labeling modules may miss unseen classes that are spatially small or semantically similar to existing classes; robustness to prompt or context quality is a recognized challenge.
• Progressive Adaptation Strategies: The practical benefit of further intra-session adaptation plateaus for strong architectures; outlier cases, however, benefit most.

6. Extensions and Future Directions

Leading future directions and open extensions include:

• Adaptive learning rates, early stopping, and multi-scale loss aggregation for improved convergence (Liang et al., 2022).
• Dynamic unit/cluster management in interactive division strategies (e.g., merging/splitting units during adaptation) (Kim et al., 29 Jun 2025).
• Test-time joint text-visual adaptation leveraging large pretrained LLMs for context-aware prompting, with confidence-driven attribute aggregation (Silva et al., 8 Jan 2025).
• Segmentation-guided query mechanisms for more sample-efficient active labeling and open-class discovery (Sreenivas et al., 27 Aug 2025).
• Scaling Seg-TTO to video and streaming settings by incorporating temporal consistency terms and fast meta-adapter architectures (Silva et al., 8 Jan 2025).
• Application to medical, remote sensing, and adversarially-shifted domains where distributional dynamics challenge classic static segmentation paradigms.

7. References and Representative Implementations

Major Seg-TTO contributions and frameworks include:

• "Segmentation by Test-Time Optimization (TTO) for CBCT-based Adaptive Radiation Therapy" (Liang et al., 2022)—first generic per-patient/fraction adaptation in DIR-based radiotherapy.
• "DC-TTA: Divide-and-Conquer Framework for Test-Time Adaptation of Interactive Segmentation" (Kim et al., 29 Jun 2025)—partitioned prompt-based adaptation in interactive segmentation.
• "Test-Time Adaptation of Vision-LLMs for Open-Vocabulary Semantic Segmentation" (MLMP) (Noori et al., 28 May 2025)—multi-level, multi-prompt pixel-entropy minimization for VLMs.
• "Test-Time Optimization for Domain Adaptive Open Vocabulary Segmentation" (Silva et al., 8 Jan 2025)—self-supervised per-sample adaptation of both text and visual representations in OVSS.
• "Segmentation Assisted Incremental Test Time Adaptation in an Open World" (Sreenivas et al., 27 Aug 2025)—segmentation-driven active labeling for continuous open-class/distribution adaptation.

These frameworks have established a diverse, robust foundation for segmentation test-time optimization, evidencing measurable improvements in personalization, domain robustness, and open-world adaptability across a spectrum of segmentation tasks.