SAM2-UNeXT: Dual-Encoder Segmentation

• The paper introduces a dual-encoder framework that fuses SAM2’s high-resolution Hiera encoder with DINOv2’s global semantic features for improved binary segmentation.
• It employs a dense glue layer and dual-resolution strategy to integrate fine details with robust semantic context, enhancing segmentation accuracy.
• The framework uses parameter-efficient adapters and a simplified U-Net decoder, achieving significant performance gains over SAM2-UNet across multiple benchmarks.

SAM2-UNeXT is a high-resolution segmentation framework that adapts the Segment Anything Model 2 (SAM2) to downstream segmentation by combining the SAM2 Hiera encoder with an auxiliary DINOv2 encoder, a dual-resolution strategy, a dense glue layer, and a simple U-Net–style decoder. It is positioned as an extension of SAM2-UNet that increases encoder-side representational capacity rather than relying on complex decoder engineering, and it is used as a fully automatic model rather than as an interactive prompt-driven system (Xiong et al., 5 Aug 2025).

1. Origin and conceptual positioning

SAM2-UNeXT was introduced as “an improved high-resolution baseline for adapting foundation models to downstream segmentation tasks,” with the stated motivation that recent SAM adaptation studies had shown promise, but that “constructing a more powerful and generalizable encoder” remained open (Xiong et al., 5 Aug 2025). The framework explicitly builds on SAM2-UNet, which had already shown that SAM2’s Hiera backbone can function as a strong frozen encoder for a U-shaped segmentation model, with adapters inserted for parameter-efficient fine-tuning and a classic U-shaped decoder used for dense prediction (Xiong et al., 2024).

The conceptual distinction from SAM2-UNet is not a change in the basic encoder–decoder paradigm, but an augmentation of the encoder stack. SAM2-UNeXT preserves the use of SAM2’s Hiera encoder, freezes the original SAM2 weights, and retains parameter-efficient adapters; however, it adds a second frozen foundation encoder, DINOv2, to compensate for limitations attributed to single-source SAM2 representations. The stated design logic is that SAM2 is strong at high-resolution, detail-rich features, whereas DINOv2 contributes stronger global semantic information; the resulting model is intended to achieve “more accurate segmentation with a simple architecture,” thereby relaxing the need for elaborate decoder modules (Xiong et al., 5 Aug 2025).

The reported evaluation is confined to four binary segmentation problem families: dichotomous image segmentation, camouflaged object detection, marine animal segmentation, and remote sensing saliency detection. In that sense, SAM2-UNeXT is best understood as a high-resolution automatic binary segmentation baseline rather than as a promptable SAM2 derivative or a general-purpose multi-class semantic segmentation framework (Xiong et al., 5 Aug 2025).

2. Dual-encoder architecture

The architecture is organized around four components: a SAM2 encoder, a DINOv2 encoder, a dense glue layer, and a U-Net–style decoder (Xiong et al., 5 Aug 2025).

The SAM2 branch uses Hiera-L as the main detailed, high-resolution representation. The original SAM2 parameters are frozen, while small adapters are inserted before each Hiera block. These adapters are bottleneck MLP modules with reduction to 32 channels, GeLU nonlinearity, and residual addition. In the notation given for the model,

$$\text{Adapter}(x) = x + \text{GeLU}(W_2\, \text{GeLU}(W_1 x)),$$

with $W_1 \in \mathbb{R}^{d \times 32}$ and $W_2 \in \mathbb{R}^{32 \times d}$. The Hiera branch outputs a four-level feature pyramid with channel dimensions

$C^{\text{SAM}} = \{144, 288, 576, 1152\}.$

The DINOv2 branch uses DINOv2-L as a frozen auxiliary encoder. Unlike Hiera, DINOv2 is non-hierarchical in this formulation and outputs a single-scale, patch-wise embedding with channel dimension 1024. The model therefore treats DINOv2 not as a replacement encoder but as a semantic augmentation path whose features are reshaped and projected into the SAM2 hierarchy.

A central architectural feature is the dual-resolution strategy. SAM2 operates on a high-resolution image, with the reported final choice being $1024 \times 1024$, while DINOv2 operates on a lower-resolution version of the same image, with the reported final choice being $448 \times 448$. The rationale is computational: DINOv2 uses standard ViT self-attention, so moving it to full SAM2 resolution would be impractical, whereas SAM2’s Hiera is designed to handle high resolutions more efficiently. The resulting division of labor is explicit in the design: SAM2 handles fine detail and thin structures, while DINOv2 supplies global semantics (Xiong et al., 5 Aug 2025).

No interactive prompts are used in either training or inference. SAM2 is treated purely as a frozen visual backbone, and prompt mechanisms are not part of the operative architecture (Xiong et al., 5 Aug 2025).

3. Dense glue layer and decoding pathway

The dense glue layer is the core fusion mechanism. Its purpose is to align DINOv2’s single-scale semantic feature map with SAM2’s four-stage hierarchical pyramid. For each SAM2 stage $s \in \{1,2,3,4\}$, let

$$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$

denote the SAM2 feature at that scale, and let

$$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$

denote the reshaped DINOv2 feature map. The fusion proceeds in four steps (Xiong et al., 5 Aug 2025):

1. Channel alignment

$$\tilde{F}^{\text{DINO}}_s = \text{Conv}^{1 \times 1}_s(F^{\text{DINO}})$$

so that $W_1 \in \mathbb{R}^{d \times 32}$0 matches the SAM2 channel dimension at scale $W_1 \in \mathbb{R}^{d \times 32}$1.

2. Spatial resizing

$W_1 \in \mathbb{R}^{d \times 32}$2

3. Concatenation

$W_1 \in \mathbb{R}^{d \times 32}$3

4. Compression

$W_1 \in \mathbb{R}^{d \times 32}$4

The “dense” character of the glue layer refers to the fact that DINOv2 is fused into all four SAM2 stages rather than only into the bottleneck. The paper also highlights an empirical observation that DINOv2 features, when projected or visualized via PCA, naturally highlight foreground regions even without fine-tuning; this is treated as evidence that DINOv2 behaves like a semantic attention source for the high-resolution SAM2 pathway (Xiong et al., 5 Aug 2025).

The decoder is U-Net–style and deliberately simple. It has four decoder stages, each based on Conv–BN–ReLU blocks, and it upsamples fused features while using skip connections from the higher-resolution fused encoder maps. Relative to SAM2-UNet, SAM2-UNeXT adds an extra “partial decoder” stage without feature concatenation, raising the output resolution from $W_1 \in \mathbb{R}^{d \times 32}$5 of the input to $W_1 \in \mathbb{R}^{d \times 32}$6 of the input. A final $W_1 \in \mathbb{R}^{d \times 32}$7 convolution maps the decoder output to a single-channel foreground probability map. The reported architectural claim is that higher output resolution is crucial for better boundary fidelity, especially for thin structures and small objects (Xiong et al., 5 Aug 2025).

4. Adaptation strategy and optimization

SAM2-UNeXT is explicitly parameter-efficient. The SAM2 Hiera backbone is frozen except for the inserted adapters; DINOv2 is entirely frozen; and the trainable components are the adapters, the dense glue layer, and the decoder (Xiong et al., 5 Aug 2025).

The training loss follows the F$W_1 \in \mathbb{R}^{d \times 32}$8Net formulation: $W_1 \in \mathbb{R}^{d \times 32}$9 The paper does not re-derive the weighting functions, but it identifies the objective as weighted cross-entropy plus weighted IoU. Optimization uses AdamW with an initial learning rate of $W_2 \in \mathbb{R}^{32 \times d}$0, cosine decay, batch size 1, and 20 epochs for all tasks. Training is reported on a single NVIDIA RTX 4090 (24 GB), with random horizontal and vertical flips as the only augmentations (Xiong et al., 5 Aug 2025).

Two ablation series are central to the reported design choices. The first concerns the auxiliary encoder. On MAS3K, removing the auxiliary encoder yields mIoU $W_2 \in \mathbb{R}^{32 \times d}$1; adding conventional auxiliaries such as ResNet-101 or PVTv2-b5 yields only marginal changes; using DINOv2 improves performance, with DINOv2-S at 0.836, DINOv2-B at 0.843, and DINOv2-L at 0.853, which is the best reported result in that ablation. The second concerns resolution. On MAS3K, using both branches at $W_2 \in \mathbb{R}^{32 \times d}$2 gives mIoU $W_2 \in \mathbb{R}^{32 \times d}$3; using SAM2 at $W_2 \in \mathbb{R}^{32 \times d}$4 and DINOv2 at $W_2 \in \mathbb{R}^{32 \times d}$5 improves to 0.842; using $W_2 \in \mathbb{R}^{32 \times d}$6 and $W_2 \in \mathbb{R}^{32 \times d}$7 both reaches 0.853, with $W_2 \in \mathbb{R}^{32 \times d}$8 chosen as the final compromise because it matches $W_2 \in \mathbb{R}^{32 \times d}$9 performance at lower cost (Xiong et al., 5 Aug 2025).

These ablations establish the intended reading of the architecture: gains come from foundation–foundation fusion, not merely from adding parameters, and from asymmetric resolution rather than uniform high or low resolution (Xiong et al., 5 Aug 2025).

5. Benchmarks and reported performance

SAM2-UNeXT is evaluated on DIS5K for dichotomous image segmentation, four standard camouflaged object detection benchmarks, two marine animal segmentation benchmarks, and two remote sensing saliency datasets (Xiong et al., 5 Aug 2025).

On dichotomous image segmentation, the paper reports consistent improvements over BiRefNet. On DIS-VD, BiRefNet achieves $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$0, $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$1, $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$2, and MAE $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$3, whereas SAM2-UNeXT achieves $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$4, $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$5, $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$6, and MAE $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$7. On the DIS-TE(1–4) aggregate, BiRefNet reports $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$8, $C^{\text{SAM}} = \{144, 288, 576, 1152\}.$9, $1024 \times 1024$0, MAE $1024 \times 1024$1, while SAM2-UNeXT reports $1024 \times 1024$2, $1024 \times 1024$3, $1024 \times 1024$4, MAE $1024 \times 1024$5 (Xiong et al., 5 Aug 2025).

On camouflaged object detection, comparisons against SAM2-UNet are particularly direct. On CHAMELEON, SAM2-UNet reports $1024 \times 1024$6, $1024 \times 1024$7, $1024 \times 1024$8, MAE $1024 \times 1024$9, while SAM2-UNeXT reports $448 \times 448$0, $448 \times 448$1, $448 \times 448$2, MAE $448 \times 448$3. On COD10K, SAM2-UNet reports $448 \times 448$4, MAE $448 \times 448$5, while SAM2-UNeXT reports $448 \times 448$6, MAE $448 \times 448$7. The same pattern holds on CAMO and NC4K, with substantial gains in both structure-aware and error metrics (Xiong et al., 5 Aug 2025).

On marine animal segmentation, the paper reports strong gains over both SAM2-UNet and specialized methods. On MAS3K, SAM2-UNet attains mIoU $448 \times 448$8, $448 \times 448$9, $s \in \{1,2,3,4\}$0, $s \in \{1,2,3,4\}$1, MAE $s \in \{1,2,3,4\}$2, whereas SAM2-UNeXT reaches mIoU $s \in \{1,2,3,4\}$3, $s \in \{1,2,3,4\}$4, $s \in \{1,2,3,4\}$5, $s \in \{1,2,3,4\}$6, MAE $s \in \{1,2,3,4\}$7. On RMAS, mIoU improves from 0.738 to 0.774 (Xiong et al., 5 Aug 2025).

On remote sensing saliency detection, the model also surpasses strong non-foundation baselines. On ORSI-4199, SFANet reports $s \in \{1,2,3,4\}$8, $s \in \{1,2,3,4\}$9, $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$0, $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$1, MAE $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$2, while SAM2-UNeXT reaches $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$3, $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$4, $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$5, $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$6, MAE $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$7. On EORSSD, SFANet reports $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$8, $$F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}$$9, $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$0, $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$1, MAE $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$2, whereas SAM2-UNeXT reports $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$3, $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$4, $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$5, $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$6, MAE $$F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}$$7 (Xiong et al., 5 Aug 2025).

The overall empirical pattern is consistent across the four benchmark families: stronger encoder-side feature fusion combined with a simple decoder outperforms both the earlier SAM2-UNet baseline and several task-specific systems that rely more heavily on decoder complexity (Xiong et al., 5 Aug 2025).

6. Relation to adjacent SAM2 hybrids, limitations, and outlook

Within the SAM2 adaptation literature, SAM2-UNeXT sits between two established design lines. One line is represented by SAM2-UNet, which established the frozen Hiera encoder plus adapter plus U-shaped decoder pattern across natural and medical segmentation tasks (Xiong et al., 2024). A second line is represented by dual-encoder hybrids such as DGSUnet, which also combine SAM2 and DINOv2 under frozen-backbone training, using attention-based cross-model fusion and an attention-aware U-shaped decoder for salient object detection and camouflaged object detection (Xu et al., 27 Mar 2025). SAM2-UNeXT belongs to this second line but is distinguished by its dense glue layer, its explicit dual-resolution strategy, and its emphasis on a simple decoder rather than cross-modal attention-heavy decoding (Xiong et al., 5 Aug 2025).

A broader representational context is provided by feature-universality analyses of SAM2. In frozen-encoder transfer experiments, SAM2 specialization improves depth estimation relative to Hiera but underperforms Hiera on pose estimation and image captioning, indicating a measurable loss of broader semantic information (Atani et al., 19 Oct 2025). This suggests that the auxiliary DINOv2 branch in SAM2-UNeXT can plausibly be read as a response to the same trade-off: SAM2 contributes high-resolution, segmentation-biased detail, while DINOv2 reintroduces global semantics. That interpretation is inferential rather than explicit, but it is consistent with both the architectural design and the ablation evidence (Atani et al., 19 Oct 2025).

The limitations reported for SAM2-UNeXT are straightforward. The framework uses SAM2-L and DINOv2-L simultaneously, so inference memory and compute are substantial even though the trainable fraction is small. The evaluated scope is binary segmentation; extensions to multi-class semantic segmentation, instance segmentation, panoptic segmentation, video segmentation, and 3D or medical volume segmentation remain open. Prompt-based adaptation is likewise not explored, because the model discards prompts entirely and uses SAM2 only as a backbone (Xiong et al., 5 Aug 2025).

The paper identifies several extensions as natural next steps: replacing or augmenting DINOv2 with other encoders, integrating prompt mechanisms into the dual-encoder setup, and extending the framework to multi-class, instance, panoptic, video, and medical settings. In that sense, SAM2-UNeXT functions both as a competitive benchmark model and as a modular encoder-fusion template for later SAM2-derived segmentation systems (Xiong et al., 5 Aug 2025).