SegDINO: Introducing Multi-Scale Structure into DINO for Efficient Medical Image Segmentation

Abstract: Self-supervised DINO models provide strong transferable visual representations, yet applying them directly to image segmentation remains challenging. Existing approaches commonly rely on heavy decoders with complex upsampling, introducing substantial parameter and computational overhead. We observe that introducing scale into DINO features is far more critical than increasing decoder capacity. In this work, we present SegDINO, an efficient segmentation framework that integrates a DINOv3 backbone with lightweight scale modeling. SegDINO introduces Token Pyramid Adaptation (TPA) to reorganize intermediate DINO features into a pseudo multi-scale hierarchy, and Scale-Aware Decoding (SAD) for efficient intra-scale refinement and top-down multi-scale propagation. We further curate PanCT, a new CT dataset containing 284 patients with expert-annotated pancreatic tumors, to assess SegDINO's ability to handle difficult small-lesion cases. Extensive experiments on PanCT and three public benchmarks demonstrate that SegDINO achieves state-of-the-art results with high efficiency. The code is available at https://github.com/script-Yang/segdino_v2.

• The paper introduces an efficient framework that integrates explicit multi-scale structure into DINO-based segmentation to enhance accuracy, particularly for small and subtle lesions.
• It utilizes a novel Token Pyramid Adaptation (TPA) and Scale-Aware Decoding (SAD) to reorganize and refine multi-depth features with minimal computational overhead.
• Experimental results across datasets such as PanCT, TN3K, Kvasir-SEG, and ISIC show significant improvements in Dice scores and Hausdorff Distance, achieving real-time performance at 51 FPS.

SegDINO: Multi-Scale Structure in DINO for Efficient Medical Image Segmentation

Introduction and Background

Medical image segmentation remains a foundational task for numerous clinical workflows, including treatment planning and computer-aided diagnosis. While convolutional networks and, more recently, transformer-based architectures have achieved substantial progress, strong generalization and efficiency are still lacking, especially for scenarios with small training datasets, protocol heterogeneity, or computational constraints.

Self-supervised visual foundation models, notably those in the DINO family, have demonstrated robust transferable representations. Nevertheless, direct adaptation of DINO-based features for dense segmentation is impeded by their lack of explicit multi-scale structure and the inefficiency of conventional heavy decoders, which are typically inherited from natural image segmentation pipelines. Addressing these limitations, SegDINO proposes an efficient framework that leverages DINOv3 representations with novel multi-scale modeling strategies, effectively enabling high-accuracy and high-efficiency medical image segmentation.

Architectural Overview

The core contribution of SegDINO is the introduction of lightweight, explicit multi-scale structure into DINO-based pipelines. The architecture consists of three major components:

1. DINOv3 Feature Extraction

SegDINO utilizes a pretrained, frozen DINOv3-S encoder to extract intermediate feature representations from multiple depths. Unlike segmentation-tailored models, raw DINO patch tokens from several transformer layers are retained, capturing semantics varying from low-level structures to high-level abstractions. Non-patch tokens such as class or register tokens are discarded, focusing solely on spatially aligned descriptors.

2. Token Pyramid Adaptation (TPA)

Recognizing that DINO’s features reside on a uniform spatial grid, TPA reorganizes these tokens into a pseudo feature pyramid. Tokens are reshaped into 2D feature maps, projected into a unified embedding space with a 1×1 convolution, and subjected to lightweight spatial resizing via strided convolutions. The result is a multi-scale hierarchy analogous to FPN architectures but tailored for transformer features, supporting coarse-to-fine semantic integration critical for segmentation, especially for small or indistinct lesions.

3. Scale-Aware Decoding (SAD)

SAD comprises two streams: intra-scale updating and inter-scale propagation. Intra-scale updating refines each scale-specific feature using an efficient residual operator involving depthwise and pointwise convolutions, group normalization, and GELU activation, with minimal additional parameterization. Inter-scale propagation fuses scales recursively from coarse to fine via residual pathways and upsampling. The final segmentation mask is predicted by a lightweight linear head applied to the highest-resolution output. Both TPA and SAD are implemented with substantial computational efficiency, avoiding heavy upsampling or redundant fusion modules.

Experimental Validation

Dataset Suite

SegDINO is evaluated on four datasets:

• PanCT: A new, curated CT dataset for small pancreatic lesion segmentation, comprising 284 patients with rigorous radiologist annotations.
• TN3K: Large-scale thyroid nodule segmentation.
• Kvasir-SEG: Polyp segmentation from colonoscopy images.
• ISIC: Skin lesion segmentation covering diverse cases.

Segmentation performance is measured primarily with Dice Similarity Coefficient (DSC) and 95th percentile Hausdorff Distance (HD95).

Quantitative Results

SegDINO establishes new state-of-the-art (SOTA) segmentation performance across all benchmarks. On TN3K, SegDINO outperforms the strongest baseline (TransUNet) by 3.64% in DSC and reduces HD95 by 7.50. For Kvasir-SEG and ISIC, DSC improvements are 4.64% and 2.41% respectively, again accompanied by significant HD95 reductions. On PanCT—a challenging small-lesion task—SegDINO not only achieves the best DSC (0.8657) but also the lowest HD95 (2.61), reflecting robust localization for subtle targets.

In terms of efficiency, SegDINO contains 27.68M parameters, with only 6.08M outside the DINO backbone. Inference throughput reaches 51 FPS, outperforming most transformer-based counterparts and approaching or surpassing several lightweight CNN alternatives.

Ablation Analysis

Comprehensive ablations confirm that TPA, which introduces explicit scale diversity, is the dominant factor in performance improvement, particularly for small-target tasks (e.g., +7.7% DSC on PanCT). SAD serves a complementary role, offering additional refinements with minimal computational burden. The findings contradict the common practice of addressing DINO's limitations with heavy decoders, highlighting that architectural simplicity and efficient scale modeling are sufficient when leveraging strong DINO pretraining.

Implications and Future Directions

SegDINO’s results have substantive implications. Practically, they demonstrate that high-performing segmentation is attainable using foundation model features with lightweight, well-structured adaptation layers—substantially reducing the marginal deployment costs in medical AI scenarios, where efficiency and resource constraints are common. Theoretically, the findings underscore the criticality of scale-aware feature adaptation over brute-force decoder complexity for vision transformer backbones.

The segmentation superiority on small, poorly contrasted lesions in PanCT underscores the generality and robustness of the approach. However, the current framework processes 2D axial slices, omitting rich inter-slice context. Future directions include extending SegDINO to 2.5D/3D segmentation, further generalization to other imaging modalities, and detailed studies on cross-institutional robustness. The lightweight nature of TPA and SAD also suggests applicability for real-time and point-of-care medical inference scenarios.

Conclusion

SegDINO introduces an efficient, multi-scale adaptation strategy for DINO-based medical image segmentation. By focusing on explicit scale hierarchy induction and scale-aware lightweight decoding, SegDINO attains SOTA performance with significant parameter and computational savings. This paradigm challenges prevailing trends of increasing decoder complexity and demonstrates the effectiveness of architectural minimalism when leveraging strong self-supervised vision backbones, especially for high-precision, real-world biomedical applications (2606.17972).