---
title: 'SAMSA 2.0: Early-Fusion in Hyperspectral Segmentation'
url: https://www.emergentmind.com/topics/samsa-2-0
type: topic
---

# SAMSA 2.0: Early-Fusion in Hyperspectral Segmentation

SAMSA 2.0 is an interactive segmentation framework for hyperspectral medical imaging that introduces spectral angle prompting to guide the Segment Anything Model (SAM) using spectral similarity alongside spatial cues. Its defining mechanism is an early-fusion prompt representation in which a pseudo-RGB projection of a hyperspectral cube is concatenated with an equalized spectral similarity map derived from user clicks, producing a 4-channel input that guides segmentation. In the reported configuration, the SAM image encoder is frozen, the prompt encoder and mask decoder are trainable, and the method is evaluated on hyperspectral clinical datasets with improvements over RGB-only and prior spectral fusion variants in Dice score, including low-data, noisy, few-shot, and zero-shot settings [2508.00493].

## 1. Position within hyperspectral interactive segmentation

SAMSA 2.0 is formulated for hyperspectral medical images rather than conventional RGB imagery. The central premise is that segmentation can be improved by coupling spatial cues from user interaction with spectral similarity computed directly from hyperspectral pixel vectors. In the reported pipeline, the inputs are a pseudo-RGB image \(X_{rgb}\in\mathbb R^{H\times W\times 3}\), an equalized spectral similarity map \(\hat Y_{\mathrm{SA}^{\mathrm{eq}}}\in[0,1]^{H\times W}\), and user clicks \(\mathcal I=\{(x_i,y_i)\}_{i=1}^N\) [2508.00493].

This design places SAMSA 2.0 between two simpler alternatives discussed in the experiments: spectral-only prompting and spatial-only prompting. The paper reports that spectral-only methods plateau at Dice@0.5 \(\approx 0.20\) on Heipor and \(\approx 0.37\) on Hib, while spatial-only frozen SAM2.1 achieves \(\approx 0.64/0.54\), and fusion yields \(> 0.80\). This suggests that the framework is explicitly intended to exploit complementarities between spatial and spectral evidence rather than to replace one with the other.

The framework is described as an early-fusion successor to prior SAMSA, which used late fusion of a spectral map into the prompt decoder. In SAMSA 2.0, spectral guidance is introduced at the prompt-encoding stage, and the reported ablation attributes consistent gains to this change.

## 2. Architecture and early-fusion mechanism

The base model is SAM with a frozen image encoder pretrained on RGB. The trainable components are the SAM prompt encoder and mask decoder. The hyperspectral image \(X\) is first projected to a pseudo-RGB representation, while a spectral angle prompt is computed from user-selected reference pixels. The high-level flow is:

\[
X \text{ (HSI)} \rightarrow \text{pseudo-RGB projection}
\]
\[
\searrow \qquad \text{compute spectral angle prompt}
\]
\[
\rightarrow \text{concatenate} \rightarrow \text{prompt encoder} \rightarrow \text{mask decoder} \rightarrow \hat Y
\]

The fusion itself is defined as

\[
X_{fuse}=\mathrm{concat}\bigl(X_{rgb},\,\hat Y_{\mathrm{SA}^{\mathrm{eq}}}\bigr)\in\mathbb R^{H\times W\times 4},
\]

or equivalently, pixelwise,

\[
X_{fuse}(h,w,:)=\bigl[\,X_{rgb}(h,w,1\!:\!3),\;\hat Y_{\mathrm{SA}^{\mathrm{eq}}}(h,w)\bigr].
\]

The architectural modification is minimal and localized: the prompt encoder of SAM is modified to accept 4-channel input instead of 3, while all subsequent layers in the image encoder remain unchanged. If the first convolution in the prompt encoder is \(\mathrm{Conv}_{\mathrm{orig}}:\mathbb R^{3\times k\times k}\to\mathbb R^{F}\), it becomes \(\mathrm{Conv}_{\mathrm{new}}:\mathbb R^{4\times k\times k}\to\mathbb R^{F}\). Only the prompt encoder and mask decoder are fine-tuned on fused inputs [2508.00493].

The reported ablation states that replacing SAMSA’s late fusion with early fusion yields \(+1.5\)–\(2.0\) percentage points in Dice across model scales. A plausible implication is that injecting spectral structure before prompt-feature formation is more effective than appending it after spatial prompt processing.

## 3. Spectral angle prompting

The spectral prompt is derived from spectral angle distance. Given hyperspectral pixel vectors \(\mathbf x\in\mathbb R^C\) and a reference spectrum \(\mathbf r\in\mathbb R^C\) taken from a user click, the spectral angle distance is

\[
\theta(\mathbf x,\mathbf r)
=\arccos\!\biggl(
\frac{\mathbf x\cdot\mathbf r}
{\lVert \mathbf x\rVert_2\,\lVert \mathbf r\rVert_2}
\biggr).
\]

To obtain a similarity score in \([0,1]\), the angle is normalized and inverted:

\[
s(\mathbf x,\mathbf r)=1-\frac{\theta(\mathbf x,\mathbf r)}{\pi}.
\]

For multiple clicks \(\{(i,j)\in\mathcal I\}\), the pixelwise prompt map is aggregated by a maximum over click-induced similarities:

\[
\hat Y_{\mathrm{SA}}(h,w)=
\max_{(i,j)\in\mathcal I}
\Bigl[1-\frac{\theta\bigl(\mathbf x_{h,w},\mathbf x_{i,j}\bigr)}{\pi}\Bigr].
\]

The technical summary also expresses the same construction as

\[
\theta_{h,w}^{(i)}=
\cos^{-1}\biggl(
\frac{\langle\mathbf x_{h,w},\mathbf x_i\rangle}
{\|\mathbf x_{h,w}\|_2\,\|\mathbf x_i\|_2}
\biggr),\qquad
s_{h,w}^{(i)}=1-\frac{\theta_{h,w}^{(i)}}{\pi}\in[0,1],
\]
\[
\hat Y_{\mathrm{SA}}(h,w)=\max_i s_{h,w}^{(i)}.
\]

Histogram equalization is then applied to \(\hat Y_{\mathrm{SA}}\) to produce \(\hat Y_{\mathrm{SA}^{\mathrm{eq}}}\) [2508.00493].

This prompting mechanism differs from purely spatial clicking in that the click is used not only as a location cue but also as a spectral reference. The reported clinical recommendation to always apply per-dataset histogram equalization to spectral prompts indicates that the equalization stage is treated as operationally important rather than incidental.

## 4. Training objective and interaction protocol

During interactive fine-tuning, at each click step \(k\) the model predicts \(\hat Y^{(k)}\). Optimization uses a weighted combination of soft Dice loss and binary cross-entropy:

\[
\mathcal L_{\mathrm{BCE}}^{(k)}
=
-\frac1{HW}\sum_{h,w}
\bigl[
y_{h,w}\log \hat y^{(k)}_{h,w}
+
(1-y_{h,w})\log(1-\hat y^{(k)}_{h,w})
\bigr],
\]

\[
\mathcal L_{\mathrm{Dice}}^{(k)}
=
1-\frac{2\sum_{h,w} y_{h,w}\hat y^{(k)}_{h,w}+\varepsilon}
{\sum_{h,w} y_{h,w}+\sum_{h,w}\hat y^{(k)}_{h,w}+\varepsilon},
\]

combined as

\[
\mathcal L^{(k)}
=
\lambda\,\mathcal L_{\mathrm{Dice}}^{(k)}
+
(1-\lambda)\,\mathcal L_{\mathrm{BCE}}^{(k)},
\qquad \lambda=0.5.
\]

The per-class loss is averaged over all \(K\) clicks:

\[
\bar{\mathcal L}_c=\frac1K\sum_{k=1}^K\mathcal L^{(k)}.
\]

No additional regularization terms were used [2508.00493].

The evaluation protocol reports Dice coefficient, equivalent to \(F_1\),

\[
\mathrm{Dice}=\frac{2\,|\hat Y\cap F|}{|\hat Y|+|F|},
\]

with results given as Dice@0.5, using a fixed threshold, and Dice@Max, using the best threshold. Interactive performance is reported for 1-click and 5-click settings. IoU and average surface distance were not reported. The explicit click-conditioned formulation is central: the loss is not a single static segmentation loss, but one indexed by interaction step and averaged across the click sequence.

## 5. Datasets, baselines, and reported performance

The experiments use two datasets. Heipor (Heidelberg Porcine HyperSPECTRAL) contains 20 porcine organs, \(H\times W\) images, and \(C\approx 100\) spectral bands in the visible–NIR. Hib (Hyperspectral Brain tumor) contains intra-operative human brain tissue, \(H\times W\), and \(C\approx 90\) bands spanning approximately \(400\)–\(1000\) nm. Both datasets provide sparse ground-truth organ or tumor masks, and ignored labels are masked out during evaluation [2508.00493].

The baselines and variants are grouped into four classes: spectral-only zero-shot methods (PCC, SA, SA\(_{\mathrm{eq}}\)); RGB-only SAM2.1 variants (Tiny, Small, Base+, Large) in frozen zero-shot and fine-tuned modes; prior SAMSA, which uses late fusion of the spectral map into the prompt decoder; and SAMSA 2.0, which uses the early-fusion strategy above.

The key quantitative result reported for 1-click Dice@0.5 is that SAMSA 2.0-Small reaches \(0.8221\) on Heipor and \(0.8615\) on Hib. The best RGB-only model, Small-FT, reaches \(0.7988\) and \(0.8083\), respectively. Prior SAMSA reaches \(0.8044\) and \(0.8179\). Averaged over the two datasets, the reported improvement for SAMSA 2.0-Small is \(+3.8\) percentage points over the best RGB-only model and \(+3.1\) percentage points over prior SAMSA on 1-click evaluation. Paired two-sided \(t\)-tests over 30 test cases yielded \(p<0.01\) for all reported gains [2508.00493].

The abstract further states that SAMSA 2.0 achieves up to \(+3.8\%\) higher Dice scores compared to RGB-only models and up to \(+3.1\%\) over prior spectral fusion methods, and that the approach enhances few-shot and zero-shot performance across diverse spectral datasets. Within the boundaries of the reported evidence, the central empirical claim is therefore not merely that spectral information helps, but that the specific early-fusion implementation helps more than the previously tested late-fusion alternative.

## 6. Robustness, transfer, and practical use

The robustness analyses focus on low-data and noisy settings that are described as common in clinical imaging. When training with only 10% labeled images, SAMSA 2.0-Small drops by less than 2 percentage points in Dice, whereas RGB-only drops by more than 5 percentage points. Under additive Gaussian noise with \(\sigma\) up to \(0.1\), SAMSA 2.0 maintains Dice \(>0.78\), versus \(<0.70\) for SAM2.1 [2508.00493].

The generalization results reported in the technical summary include zero-shot evaluation on unseen organs such as liver and bladder, with average Dice@0.5 \(=0.82\) without any fine-tuning. Cross-scanner transfer is reported to incur minimal degradation, specifically less than 3 percentage points in Dice when moving from one hyperspectral system to another. These values are consistent with the paper’s broader emphasis on few-shot and zero-shot performance.

Implementation details are also specified. Fine-tuning takes approximately 10 hours on \(8\times\)A100 with batch size \(=4\) and gradient accumulation. Inference is approximately \(0.5\) seconds per click on a single A100. The additional memory cost is \(+10\%\) due to the extra spectral channel. Clinical recommendations in the summary are correspondingly concrete: always apply per-dataset histogram equalization to spectral prompts; use \(\lambda=0.5\) for balanced BCE versus Dice loss; and, for very low-contrast tissues, consider multiple reference clicks to stabilize the spectral map.

Two recurrent simplifications are directly challenged by the reported evidence. First, the results do not support treating spectral-only prompting as sufficient for high-quality segmentation, since the spectral-only baselines plateau well below the fused models. Second, the data do not support treating fusion placement as immaterial, because early fusion outperforms late fusion across scales. A plausible implication is that SAMSA 2.0’s main contribution lies less in the mere addition of hyperspectral information than in the stage at which that information is introduced into the prompt pathway.

Source: https://www.emergentmind.com/topics/samsa-2-0