---
title: 'DepSeg: Depth-Guided Surgical Segmentation'
url: https://www.emergentmind.com/topics/depth-guided-surgical-scene-segmentation-depseg
type: topic
---

# DepSeg: Depth-Guided Surgical Segmentation

Depth-guided surgical scene segmentation (DepSeg) encompasses computational frameworks for partitioning images of surgical scenes into semantically distinct regions using depth information as a principal cue, notably improving segmentation quality and annotation efficiency in minimally annotated or high-complexity scenarios. DepSeg approaches leverage monocular depth priors or explicit RGB-D fusion to guide object proposals and disambiguate visually challenging structures, capitalizing on both geometric and appearance-based features [2512.05529][2407.19714].

## 1. Monocular Depth-Guided Segmentation Frameworks

Training-free depth-guided segmentation, as exemplified by the DepSeg pipeline [2512.05529], operates in a strictly modular, non-end-to-end manner. The input is a single RGB frame $x \in \mathbb{R}^{H \times W \times 3}$; a monocular depth estimator (e.g., DepthAnythingV2) infers a dense relative depth map $d(x, y) = f_\text{depth}(x) \in \mathbb{R}^{H\times W}$. This map is median-centered on a per-image basis:
\[
d_\mathrm{rel}(p) = d(p) - \mathrm{median}_{q \in \Omega}\{d(q)\}, \; p \in \Omega.
\]

Spatial segmentation seeds are extracted by a deterministic geometric operator $G$ acting on $d_\mathrm{rel}$:

- **K-means clustering** in depth space to group pixels by contiguous depth,
- **Canny edge detection** and **Otsu thresholding** to delineate boundaries,
- **Watershed segmentation** to split overlarge regions.

Morphological closing/opening produces cleaned candidate regions $\tilde R_m$. For each, the Euclidean distance transform $D_m(p)$ finds local maxima that serve as point prompts $S_m$, spaced at least $d_\mathrm{min}$ apart and away from the image border.

Point prompts $S = \bigcup_m S_m$ are consumed by a pretrained, frozen SAM2 image predictor, yielding soft score maps per prompt. Binary masks $M^{(0)}_i(p) = \mathbf{1}\{s_i(p) > \tau_s\}$ are extracted, morphologically filtered, and merged via a small-area-first priority scheme to yield a non-overlapping set $\widehat{\mathcal{M}} = \{M_j\}_{j=1}^J$.

## 2. Template-Based Mask Classification

Region descriptors are computed for each $M_j$ by masked average pooling of DINOv3 vision foundation model tokens. A template bank $\mathcal{T}_c$ is assembled offline by pooling DINOv3 features over class-masked regions in $N$ annotated frames:
\[
h_{n, c} = \frac{1}{Z_{n, c}} \sum_{u,v} M'_{n, c}(u,v) T^{(n)}(u,v,:), \quad Z_{n,c} = \sum_{u,v} M'_{n, c}(u,v)
\]
with L2 normalization. Per-class semantic assignment for predicted masks proceeds by cosine similarity matching to templates, summing the top-$k$ template scores $S_{j, c}$ and assigning class $\hat c_j = \arg\max_c S_{j, c}$. The final pixelwise map $y = F(x)$ results from merged labeled masks.

## 3. RGB-D Fusion Architectures for Surgical Segmentation

Alternatives to prompt-driven segmentation include end-to-end trainable RGB-D fusion networks. SurgDepth [2407.19714] exemplifies this paradigm, integrating RGB and predicted depth maps in a joint ViT encoder–decoder scheme. The architecture comprises:

- Parallel projection heads for RGB and depth,
- A lightweight “3D awareness” fusion block performing attention between concatenated RGB/depth features and RGB-alone keys/values, with outputs upsampled and linearly separated back into modality-specific streams,
- A ViT-B encoder (pretrained via image MAE objectives),
- A shallow ConvNeXt decoder.

The fusion block pools $[X_i^{\rm rgb}; X_i^{\rm depth}]$ to $k \times k$ grids, computes queries, keys, values via $1 \times 1$ projections, applies attention, and forms modality-updated representations:
\[
\begin{aligned}
Q &= W_q \left(\mathrm{Pool}_{k \times k}\left([X_i^{\rm rgb}, X_i^{\rm depth}]\right)\right),\\
K &= W_k(X_i^{\rm rgb}),\quad V = W_v(X_i^{\rm rgb}),\\
A &= \mathrm{Softmax}\left(\frac{Q^\top K}{\sqrt{C_d}}\right),\\
X_\mathrm{fusion} &= \mathrm{Upsample}_{h, w}(V A),\\
(\widehat{X}_i^{\rm rgb}, \widehat{X}_i^{\rm depth}) &= (\mathrm{FC}_{\rm rgb},\,\mathrm{FC}_{\rm depth})(X_\mathrm{fusion}).
\end{aligned}
\]

The final pixelwise semantic logits are trained using cross-entropy loss:
\[
\mathcal{L}_{\rm seg} = -\frac{1}{HW} \sum_{u=1}^H \sum_{v=1}^W \sum_{c=1}^C y_{u, v, c} \log p_{u, v, c}
\]

## 4. Quantitative Performance and Annotation Efficiency

Performance metrics are consistently reported via mean intersection-over-union (mIoU):
\[
\mathrm{IoU}_c = \frac{\mathrm{TP}_c}{\mathrm{TP}_c + \mathrm{FP}_c + \mathrm{FN}_c}, \quad \mathrm{mIoU} = \frac{1}{C}\sum_{c=1}^C \mathrm{IoU}_c.
\]

In the CholecSeg8k dataset [2512.05529]:

| Method                       | mIoU (%) |
|------------------------------|----------|
| Mask2Former (supervised)      | 78.4     |
| SAM2 + 8D cosine-matching     | 14.7     |
| SAM2 + DINOv3 template-match  | 31.6     |
| DepSeg (depth prompts, full)  | 35.9     |

Annotation efficiency is achieved by template selection: using only $10\%-20\%$ of templates in the bank yields $31.2\%-33.6\%$ mIoU (vs.\ $35.9\%$ with all templates), thus reducing annotation effort by $\sim80\%-90\%$ with minimal quality loss. No further model finetuning is necessary; only template features are precomputed and stored.

In RGB-D fusion paradigms, SurgDepth achieves the following SOTA mIoU values [2407.19714]:

| Dataset              | SurgDepth IoU |
|----------------------|---------------|
| SAR-RARP50           | 0.862         |
| CholecSeg8k          | 0.556         |
| LapI2I               | 0.981         |
| EndoVis2017          | 0.613 (challenge IoU) |

Early fusion outperforms late fusion, and increasing ConvNeXt decoder depth past four blocks does not boost accuracy.

## 5. Analysis of Methodological Differences

DepSeg and SurgDepth represent the two principal strategies in depth-guided segmentation: training-free geometric prompting with template-based labeling versus end-to-end deep RGB-D joint encoding. DepSeg is annotation-efficient, requiring only a template bank and frozen pretrained networks, with no additional model training or finetuning. The procedure is stable, deterministic, and modular, but yields a significantly lower upper-bound mIoU compared to fully-supervised or joint RGB-D training. SurgDepth, by contrast, achieves higher absolute accuracy (near-supervised SOTA) but at the cost of model retraining, complete dataset-specific supervision, and heavier computational resources.

A plausible implication is that DepSeg methods are best suited for rapid, low-resource bootstrapping scenarios or where annotation cost is prohibitive, while RGB-D fusion methods dominate where dense ground truth is available and maximizing segmentation accuracy is critical.

## 6. Limitations and Future Directions

Current depth-guided segmentation faces several critical challenges:

- **Rare class performance**: Both DepSeg and RGB-D models assign near-zero IoU to rare structures (e.g., hepatic veins [2512.05529]).
- **Depth estimation domain gaps**: Monocular depth networks, often pretrained on natural images, degrade under strong specularities, blood, and tissue deformation prevalent in surgical scenes [2407.19714].
- **Temporal consistency**: Existing DepSeg pipelines operate frame-wise; no temporal smoothing or track-based mask refinement is applied.
- **Template selection and coverage**: Uniform template sampling fails for fine-grained or underrepresented anatomical categories.

Potential improvements include smarter active sampling for templates, integration of temporal cues (e.g., recurrent or video-based modules), leveraging metric depth/3D priors if available, and introducing lightweight finetuning to adapt pretrained segmenters to surgical domains. The plug-and-play structure of RGB-D fusion blocks further facilitates extension to multi-modal and multi-view surgical scene parsing.

## 7. Context Within the Broader Field

The introduction of depth-guided prompting and template-based classification [2512.05529] aligns with ongoing efforts to reduce annotation burden in medical imaging and to exploit geometric cues unavailable to RGB-only segmenters. SurgDepth [2407.19714] demonstrates that ViT-based encoder–decoder models with explicit depth fusion can achieve or exceed the state of the art on multiple public benchmarks with modest increases in computational cost. Both methods underscore the importance of robust depth priors for improving mask delineation in challenging scenarios with poor color contrast, specular artifacts, and occlusions.

Depth-guided segmentation continues to evolve, with open challenges including real-time deployment, robustness to domain shift, and construction of comprehensive, calibrated RGB-D surgical datasets for further benchmarking and development.

Source: https://www.emergentmind.com/topics/depth-guided-surgical-scene-segmentation-depseg