---
title: 'SAM2-UNeXT: Dual-Encoder Segmentation'
url: https://www.emergentmind.com/topics/sam2-unext
type: topic
---

# SAM2-UNeXT: Dual-Encoder Segmentation

SAM2-UNeXT is a high-resolution segmentation framework that adapts the Segment Anything Model 2 (SAM2) to downstream segmentation by combining the SAM2 Hiera encoder with an auxiliary DINOv2 encoder, a dual-resolution strategy, a dense glue layer, and a simple U-Net–style decoder. It is positioned as an extension of SAM2-UNet that increases encoder-side representational capacity rather than relying on complex decoder engineering, and it is used as a fully automatic model rather than as an interactive prompt-driven system [2508.03566].

## 1. Origin and conceptual positioning

SAM2-UNeXT was introduced as “an improved high-resolution baseline for adapting foundation models to downstream segmentation tasks,” with the stated motivation that recent SAM adaptation studies had shown promise, but that “constructing a more powerful and generalizable encoder” remained open [2508.03566]. The framework explicitly builds on SAM2-UNet, which had already shown that SAM2’s Hiera backbone can function as a strong frozen encoder for a U-shaped segmentation model, with adapters inserted for parameter-efficient fine-tuning and a classic U-shaped decoder used for dense prediction [2408.08870].

The conceptual distinction from SAM2-UNet is not a change in the basic encoder–decoder paradigm, but an augmentation of the encoder stack. SAM2-UNeXT preserves the use of SAM2’s Hiera encoder, freezes the original SAM2 weights, and retains parameter-efficient adapters; however, it adds a second frozen foundation encoder, DINOv2, to compensate for limitations attributed to single-source SAM2 representations. The stated design logic is that SAM2 is strong at high-resolution, detail-rich features, whereas DINOv2 contributes stronger global semantic information; the resulting model is intended to achieve “more accurate segmentation with a simple architecture,” thereby relaxing the need for elaborate decoder modules [2508.03566].

The reported evaluation is confined to four binary segmentation problem families: dichotomous image segmentation, camouflaged object detection, marine animal segmentation, and remote sensing saliency detection. In that sense, SAM2-UNeXT is best understood as a high-resolution automatic binary segmentation baseline rather than as a promptable SAM2 derivative or a general-purpose multi-class semantic segmentation framework [2508.03566].

## 2. Dual-encoder architecture

The architecture is organized around four components: a SAM2 encoder, a DINOv2 encoder, a dense glue layer, and a U-Net–style decoder [2508.03566].

The SAM2 branch uses Hiera-L as the main detailed, high-resolution representation. The original SAM2 parameters are frozen, while small adapters are inserted before each Hiera block. These adapters are bottleneck MLP modules with reduction to 32 channels, GeLU nonlinearity, and residual addition. In the notation given for the model,
\[
\text{Adapter}(x) = x + \text{GeLU}(W_2\, \text{GeLU}(W_1 x)),
\]
with \(W_1 \in \mathbb{R}^{d \times 32}\) and \(W_2 \in \mathbb{R}^{32 \times d}\). The Hiera branch outputs a four-level feature pyramid with channel dimensions
\[
C^{\text{SAM}} = \{144, 288, 576, 1152\}.
\]

The DINOv2 branch uses DINOv2-L as a frozen auxiliary encoder. Unlike Hiera, DINOv2 is non-hierarchical in this formulation and outputs a single-scale, patch-wise embedding with channel dimension 1024. The model therefore treats DINOv2 not as a replacement encoder but as a semantic augmentation path whose features are reshaped and projected into the SAM2 hierarchy.

A central architectural feature is the dual-resolution strategy. SAM2 operates on a high-resolution image, with the reported final choice being \(1024 \times 1024\), while DINOv2 operates on a lower-resolution version of the same image, with the reported final choice being \(448 \times 448\). The rationale is computational: DINOv2 uses standard ViT self-attention, so moving it to full SAM2 resolution would be impractical, whereas SAM2’s Hiera is designed to handle high resolutions more efficiently. The resulting division of labor is explicit in the design: SAM2 handles fine detail and thin structures, while DINOv2 supplies global semantics [2508.03566].

No interactive prompts are used in either training or inference. SAM2 is treated purely as a frozen visual backbone, and prompt mechanisms are not part of the operative architecture [2508.03566].

## 3. Dense glue layer and decoding pathway

The dense glue layer is the core fusion mechanism. Its purpose is to align DINOv2’s single-scale semantic feature map with SAM2’s four-stage hierarchical pyramid. For each SAM2 stage \(s \in \{1,2,3,4\}\), let
\[
F^{\text{SAM}}_s \in \mathbb{R}^{H_s \times W_s \times C^{\text{SAM}}_s}
\]
denote the SAM2 feature at that scale, and let
\[
F^{\text{DINO}} \in \mathbb{R}^{H_l' \times W_l' \times 1024}
\]
denote the reshaped DINOv2 feature map. The fusion proceeds in four steps [2508.03566]:

1. **Channel alignment**
   \[
   \tilde{F}^{\text{DINO}}_s = \text{Conv}^{1 \times 1}_s(F^{\text{DINO}})
   \]
   so that \(\tilde{F}^{\text{DINO}}_s\) matches the SAM2 channel dimension at scale \(s\).

2. **Spatial resizing**
   \[
   \hat{F}^{\text{DINO}}_s = \text{Resize}(\tilde{F}^{\text{DINO}}_s; H_s, W_s)
   \]

3. **Concatenation**
   \[
   F^{\text{cat}}_s = \text{Concat}\big(F^{\text{SAM}}_s,\; \hat{F}^{\text{DINO}}_s\big)
   \]

4. **Compression**
   \[
   F^{\text{fused}}_s = \text{Conv}^{1 \times 1}_{\text{compress}, s}(F^{\text{cat}}_s),
   \quad F^{\text{fused}}_s \in \mathbb{R}^{H_s \times W_s \times 128}.
   \]

The “dense” character of the glue layer refers to the fact that DINOv2 is fused into all four SAM2 stages rather than only into the bottleneck. The paper also highlights an empirical observation that DINOv2 features, when projected or visualized via PCA, naturally highlight foreground regions even without fine-tuning; this is treated as evidence that DINOv2 behaves like a semantic attention source for the high-resolution SAM2 pathway [2508.03566].

The decoder is U-Net–style and deliberately simple. It has four decoder stages, each based on Conv–BN–ReLU blocks, and it upsamples fused features while using skip connections from the higher-resolution fused encoder maps. Relative to SAM2-UNet, SAM2-UNeXT adds an extra “partial decoder” stage without feature concatenation, raising the output resolution from \(1/4\) of the input to \(1/2\) of the input. A final \(1 \times 1\) convolution maps the decoder output to a single-channel foreground probability map. The reported architectural claim is that higher output resolution is crucial for better boundary fidelity, especially for thin structures and small objects [2508.03566].

## 4. Adaptation strategy and optimization

SAM2-UNeXT is explicitly parameter-efficient. The SAM2 Hiera backbone is frozen except for the inserted adapters; DINOv2 is entirely frozen; and the trainable components are the adapters, the dense glue layer, and the decoder [2508.03566].

The training loss follows the F\(^3\)Net formulation:
\[
\mathcal{L} = \mathcal{L}^{\omega}_{\text{BCE}} + \mathcal{L}^{\omega}_{\text{IoU}}.
\]
The paper does not re-derive the weighting functions, but it identifies the objective as weighted cross-entropy plus weighted IoU. Optimization uses AdamW with an initial learning rate of \(2 \times 10^{-4}\), cosine decay, batch size 1, and 20 epochs for all tasks. Training is reported on a single NVIDIA RTX 4090 (24 GB), with random horizontal and vertical flips as the only augmentations [2508.03566].

Two ablation series are central to the reported design choices. The first concerns the auxiliary encoder. On MAS3K, removing the auxiliary encoder yields mIoU \(= 0.832\); adding conventional auxiliaries such as ResNet-101 or PVTv2-b5 yields only marginal changes; using DINOv2 improves performance, with DINOv2-S at 0.836, DINOv2-B at 0.843, and DINOv2-L at 0.853, which is the best reported result in that ablation. The second concerns resolution. On MAS3K, using both branches at \(352 \times 352\) gives mIoU \(= 0.820\); using SAM2 at \(1024 \times 1024\) and DINOv2 at \(224 \times 224\) improves to 0.842; using \(1024/672\) and \(1024/448\) both reaches 0.853, with \(1024/448\) chosen as the final compromise because it matches \(672 \times 672\) performance at lower cost [2508.03566].

These ablations establish the intended reading of the architecture: gains come from foundation–foundation fusion, not merely from adding parameters, and from asymmetric resolution rather than uniform high or low resolution [2508.03566].

## 5. Benchmarks and reported performance

SAM2-UNeXT is evaluated on DIS5K for dichotomous image segmentation, four standard camouflaged object detection benchmarks, two marine animal segmentation benchmarks, and two remote sensing saliency datasets [2508.03566].

On dichotomous image segmentation, the paper reports consistent improvements over BiRefNet. On DIS-VD, BiRefNet achieves \(S_\alpha = 0.898\), \(F_{\beta}^{w} = 0.854\), \(E_{\phi} = 0.931\), and MAE \(= 0.038\), whereas SAM2-UNeXT achieves \(S_\alpha = 0.910\), \(F_{\beta}^{w} = 0.864\), \(E_{\phi} = 0.938\), and MAE \(= 0.034\). On the DIS-TE(1–4) aggregate, BiRefNet reports \(S_\alpha = 0.901\), \(F_{\beta}^{w} = 0.858\), \(E_{\phi} = 0.934\), MAE \(= 0.035\), while SAM2-UNeXT reports \(S_\alpha = 0.911\), \(F_{\beta}^{w} = 0.867\), \(E_{\phi} = 0.940\), MAE \(= 0.032\) [2508.03566].

On camouflaged object detection, comparisons against SAM2-UNet are particularly direct. On CHAMELEON, SAM2-UNet reports \(S_\alpha = 0.914\), \(F_\beta = 0.863\), \(E_\phi = 0.961\), MAE \(= 0.022\), while SAM2-UNeXT reports \(S_\alpha = 0.942\), \(F_\beta = 0.916\), \(E_\phi = 0.972\), MAE \(= 0.013\). On COD10K, SAM2-UNet reports \(S_\alpha = 0.880\), MAE \(= 0.021\), while SAM2-UNeXT reports \(S_\alpha = 0.924\), MAE \(= 0.013\). The same pattern holds on CAMO and NC4K, with substantial gains in both structure-aware and error metrics [2508.03566].

On marine animal segmentation, the paper reports strong gains over both SAM2-UNet and specialized methods. On MAS3K, SAM2-UNet attains mIoU \(= 0.799\), \(S_\alpha = 0.903\), \(F_{\beta}^{w} = 0.848\), \(E_\phi = 0.943\), MAE \(= 0.021\), whereas SAM2-UNeXT reaches mIoU \(= 0.853\), \(S_\alpha = 0.926\), \(F_{\beta}^{w} = 0.900\), \(E_\phi = 0.960\), MAE \(= 0.014\). On RMAS, mIoU improves from 0.738 to 0.774 [2508.03566].

On remote sensing saliency detection, the model also surpasses strong non-foundation baselines. On ORSI-4199, SFANet reports \(S_\alpha = 0.876\), \(F_{\beta}^{mean} = 0.866\), \(F_{\beta}^{max} = 0.871\), \(E_\phi = 0.939\), MAE \(= 0.029\), while SAM2-UNeXT reaches \(S_\alpha = 0.887\), \(F_{\beta}^{mean} = 0.873\), \(F_{\beta}^{max} = 0.886\), \(E_\phi = 0.942\), MAE \(= 0.026\). On EORSSD, SFANet reports \(S_\alpha = 0.935\), \(F_{\beta}^{mean} = 0.868\), \(F_{\beta}^{max} = 0.883\), \(E_\phi = 0.973\), MAE \(= 0.006\), whereas SAM2-UNeXT reports \(S_\alpha = 0.948\), \(F_{\beta}^{mean} = 0.892\), \(F_{\beta}^{max} = 0.905\), \(E_\phi = 0.978\), MAE \(= 0.004\) [2508.03566].

The overall empirical pattern is consistent across the four benchmark families: stronger encoder-side feature fusion combined with a simple decoder outperforms both the earlier SAM2-UNet baseline and several task-specific systems that rely more heavily on decoder complexity [2508.03566].

## 6. Relation to adjacent SAM2 hybrids, limitations, and outlook

Within the SAM2 adaptation literature, SAM2-UNeXT sits between two established design lines. One line is represented by SAM2-UNet, which established the frozen Hiera encoder plus adapter plus U-shaped decoder pattern across natural and medical segmentation tasks [2408.08870]. A second line is represented by dual-encoder hybrids such as DGSUnet, which also combine SAM2 and DINOv2 under frozen-backbone training, using attention-based cross-model fusion and an attention-aware U-shaped decoder for salient object detection and camouflaged object detection [2503.21187]. SAM2-UNeXT belongs to this second line but is distinguished by its dense glue layer, its explicit dual-resolution strategy, and its emphasis on a simple decoder rather than cross-modal attention-heavy decoding [2508.03566].

A broader representational context is provided by feature-universality analyses of SAM2. In frozen-encoder transfer experiments, SAM2 specialization improves depth estimation relative to Hiera but underperforms Hiera on pose estimation and image captioning, indicating a measurable loss of broader semantic information [2510.17051]. This suggests that the auxiliary DINOv2 branch in SAM2-UNeXT can plausibly be read as a response to the same trade-off: SAM2 contributes high-resolution, segmentation-biased detail, while DINOv2 reintroduces global semantics. That interpretation is inferential rather than explicit, but it is consistent with both the architectural design and the ablation evidence [2510.17051].

The limitations reported for SAM2-UNeXT are straightforward. The framework uses SAM2-L and DINOv2-L simultaneously, so inference memory and compute are substantial even though the trainable fraction is small. The evaluated scope is binary segmentation; extensions to multi-class semantic segmentation, instance segmentation, panoptic segmentation, video segmentation, and 3D or medical volume segmentation remain open. Prompt-based adaptation is likewise not explored, because the model discards prompts entirely and uses SAM2 only as a backbone [2508.03566].

The paper identifies several extensions as natural next steps: replacing or augmenting DINOv2 with other encoders, integrating prompt mechanisms into the dual-encoder setup, and extending the framework to multi-class, instance, panoptic, video, and medical settings. In that sense, SAM2-UNeXT functions both as a competitive benchmark model and as a modular encoder-fusion template for later SAM2-derived segmentation systems [2508.03566].

Source: https://www.emergentmind.com/topics/sam2-unext