EndoUFM: Unsupervised Depth in Endoscopy

• EndoUFM is an unsupervised monocular depth estimation framework for endoscopic scenes that integrates geometric priors from Depth Anything and anatomical segmentation from SAM/MedSAM.
• The method introduces parameter-efficient adaptations via RVLoRA and uses Res-DSC for refined local detail extraction under challenging lighting, occlusions, and tissue deformations.
• Evaluations on SCARED, Hamlyn, SERV-CT, and EndoNeRF datasets demonstrate state-of-the-art performance, enhancing 3D reconstruction and surgical navigation.

Searching arXiv for the specified paper to ground the article and cite it accurately. EndoUFM is an unsupervised monocular depth estimation framework for endoscopic surgical scenes that combines two foundation models—Depth Anything for depth estimation and SAM/MedSAM for image decomposition and anatomy-aware segmentation—to improve robustness under non-uniform illumination, specular reflections, tissue deformation, frequent occlusions, and complex textures. The method is formulated for minimally invasive endoscopic surgery, where true depth is difficult to acquire in vivo and where classical unsupervised monocular depth estimation, with its reliance on photometric consistency and often rigid-scene assumptions, is especially fragile. EndoUFM introduces Random Vector Low-Rank Adaptation (RVLoRA), a Residual block based on Depthwise Separable Convolution (Res-DSC), and a mask-guided smoothness loss, and is reported to achieve state-of-the-art performance on SCARED, Hamlyn, SERV-CT, and EndoNeRF while maintaining an efficient model size (Yao et al., 25 Aug 2025).

1. Endoscopic monocular depth estimation as a domain-shift problem

Monocular depth estimation is a foundational component for 3D reconstruction in minimally invasive endoscopic surgeries, but the assumptions used by many standard self-supervised methods break down in surgical imagery. EndoUFM is motivated by several failure modes that are characteristic of endoscopy: non-uniform illumination caused by light sources attached to the endoscope, specular reflections from wet tissue, tissue deformation in non-rigid internal organs, frequent occlusions by surgical tools, and complex textures and ambiguous surfaces in homogeneous or low-texture tissue regions. The work also emphasizes that traditional CNN-based models often focus on local receptive fields and do not capture broader scene geometry well (Yao et al., 25 Aug 2025).

Within this problem setting, the central methodological claim is that powerful visual foundation models offer a promising solution but exhibit significant domain adaptability limitations and semantic perception deficiencies when transferred directly from natural-image pretraining to endoscopy. This is why EndoUFM is not presented as a simple substitution of a stronger backbone. Instead, it is framed as a domain adaptation and representation integration strategy: global geometric priors are imported from a pretrained depth foundation model, semantic anatomical priors are imported from a segmentation-oriented foundation model, and both are adapted to the surgical domain through parameter-efficient fine-tuning.

A common misconception addressed by the ablation evidence is that inserting a foundation model into a monocular endoscopic pipeline is sufficient on its own. The reported ablations state that adding Depth Anything alone does not help much unless adapted, which positions domain adaptation—not mere model scale—as a central requirement of the framework (Yao et al., 25 Aug 2025).

2. Dual-foundation-model architecture

EndoUFM is built on an unsupervised monocular depth estimation pipeline with a target frame $I_t$, source frames $I_s$, a depth estimation network $\Phi_D$, a pose estimation network $\Phi_T$, an image decomposition module $\Phi_I$, and an overview/reconstruction module $\Phi_L$. The decomposition module produces reflectance $R$ and shading $S$. This arrangement connects geometric estimation, appearance decomposition, and view synthesis in a single training pipeline (Yao et al., 25 Aug 2025).

The depth branch replaces a conventional depth backbone with the Depth Anything transformer encoder-decoder. Features from the last four encoder layers are fed into the frozen reassembly/fusion part and then into a trainable depth head to predict multi-scale depth. In the decomposition branch, SAM/MedSAM is integrated into the decomposition network: the encoder is SAM’s vision transformer, and the decoder is modified from the depth head into a Reflectance head and a Shading head. The stated rationale is complementary specialization. Depth Anything contributes strong geometric priors and long-range context, whereas SAM/MedSAM contributes semantic awareness of anatomical structures, improving tissue-region separation and making reflectance/shading decomposition more reliable under difficult illumination.

This architecture is explicitly tailored to endoscopic scenes rather than being a generic monocular pipeline. The image decomposition branch is relevant because the paper treats illumination variation as a first-order obstacle to self-supervised reconstruction losses. Separating reflectance from shading is therefore not an auxiliary convenience but part of the mechanism by which the model attempts to reduce the mismatch between photometric assumptions and surgical reality.

3. RVLoRA and parameter-efficient adaptation

The main adaptation mechanism is Random Vector Low-Rank Adaptation (RVLoRA), introduced to bridge the domain gap between pretrained foundation models and endoscopic data. The starting point is standard LoRA. For a pretrained matrix $W_0 \in \mathbb{R}^{m \times n}$, LoRA updates it as

$W_0 + \Delta W = W_0 + BA$

and the forward pass becomes

$I_s$0

where $I_s$1, $I_s$2, and $I_s$3. Only $I_s$4 and $I_s$5 are trainable; $I_s$6 remains frozen (Yao et al., 25 Aug 2025).

EndoUFM extends this by introducing random frozen scaling vectors $I_s$7 and $I_s$8, converted into diagonal matrices $I_s$9 and $\Phi_D$0. The adapted update is

$\Phi_D$1

where $\Phi_D$2, $\Phi_D$3, and $\Phi_D$4, $\Phi_D$5. In this construction, $\Phi_D$6 and $\Phi_D$7 are trainable low-rank matrices, while $\Phi_D$8 and $\Phi_D$9 are randomly initialized frozen vectors. The initialization is Kaiming uniform for $\Phi_T$0, $\Phi_T$1, and $\Phi_T$2, while $\Phi_T$3 is initialized to zeros.

The paper argues that this design exploits the low intrinsic dimensionality of task-specific adaptation while improving scaling flexibility across layers. It further argues that the mechanism helps especially with depth scale ambiguity in monocular estimation, enhances robustness to different scene contexts and scales, and remains parameter-efficient. The fine-tuning scope is asymmetric across the two foundation models: for Depth Anything, all transformer blocks in the encoder are fine-tuned with RVLoRA; for MedSAM, only the bottom three transformer blocks are fine-tuned, reducing cost while preserving performance.

The empirical claims are specific. Ablations show that RVLoRA outperforms standard LoRA, VeRA, and DV-LoRA, and that Kaiming uniform initialization for $\Phi_T$4 performs best. A related misconception is therefore that all low-rank adaptation variants are interchangeable in this setting; the reported evidence does not support that equivalence.

4. Res-DSC and mask-guided anatomical regularization

Transformers provide global context but are described as weak at local detail extraction. Endoscopic depth estimation, by contrast, requires both global anatomical structure and fine boundaries or local texture cues. To bridge this gap, EndoUFM inserts Residual blocks based on Depthwise Separable Convolution (Res-DSC) into the depth network (Yao et al., 25 Aug 2025).

Res-DSC contains six elements: a $\Phi_T$5 convolution for channel reduction, a $\Phi_T$6 depthwise separable convolution, another $\Phi_T$7 convolution for channel restoration, channel attention, spatial attention, and a residual connection. The stated purpose of depthwise separable convolution is to reduce computation and parameters compared to standard convolution while still capturing local patterns efficiently. The residual design preserves original transformer features while adding local refinement. To limit parameter overhead, Res-DSC is inserted only after the 3rd, 6th, 9th, and 12th transformer blocks in the depth network. Its reported effects are improved boundary sharpness, local tissue detail, and fine-grained depth accuracy.

Ablation is important here because the paper does not present Res-DSC as universally beneficial. The module improves results when applied to the depth branch, though not in the SAM branch, where it can hurt performance. This clarifies that local refinement is branch-specific rather than a generic gain mechanism.

A second novelty is the mask-guided depth smoothness loss, which uses segmentation masks from SAM/MedSAM to enforce anatomically meaningful smoothness:

$\Phi_T$8

where $\Phi_T$9 is the predicted depth map, $\Phi_I$0 is the input image, $\Phi_I$1 is the auto-generated segmentation mask, and $\Phi_I$2 are spatial gradients.

The paper interprets this as a mask-weighted edge-aware smoothness term. It encourages smooth depth inside the same anatomical region, preserves discontinuities at image edges, reduces depth noise in homogeneous tissue regions, and respects organ and tissue boundaries inferred by SAM masks. In endoscopy, where depth should be consistent across the same tissue surface but should change at true anatomical boundaries, this term is presented as a way to reduce semantic ambiguity and improve intra-structure consistency.

5. Training objective, datasets, and evaluation protocol

The full training objective is

$\Phi_I$3

where $\Phi_I$4 is the reflectance consistency loss, $\Phi_I$5 is the decomposing-synthesis loss, $\Phi_I$6 is the mapping-synthesis loss, and $\Phi_I$7 is the mask-guided smoothness loss (Yao et al., 25 Aug 2025).

The constituent terms are:

$\Phi_I$8

$\Phi_I$9

$\Phi_L$0

$\Phi_L$1

with $\Phi_L$2.

Evaluation is carried out on four public datasets. The training and transfer regime matters because the paper reports both in-domain and cross-dataset behavior.

Dataset	Characteristics	Split / resolution
SCARED	MICCAI 2019 stereo correspondence challenge; porcine cadaver data from da Vinci robot; structured light depth ground truth	15,351 train, 1,705 val, 551 test; $\Phi_L$3
Hamlyn	In vivo laparoscopic dataset; rectified version used; ground truth depth for left-view frames	12,796 train, 2,076 val, 2,319 test; $\Phi_L$4
SERV-CT	Ex vivo porcine stereo images; depth from aligned 3D CT models and stereo endoscopic images	16 stereo pairs, 32 test frames; $\Phi_L$5
EndoNeRF	Two endoscopic scenes; stereo depth from STTR-Light; includes tool masks	219 left-view test images; $\Phi_L$6

For depth evaluation, the paper uses Abs Rel, Sq Rel, RMSE, RMSE log, and $\Phi_L$7, $\Phi_L$8, $\Phi_L$9:

$R$0

$R$1

$R$2

$R$3

$R$4

Because depth is monocular and scale-ambiguous, predictions are scaled using

$R$5

and the scaled depth is capped at 150 mm. Camera pose is evaluated using ATE over 5-frame segments.

6. Quantitative performance, ablations, and implications

On SCARED, EndoUFM reports Abs Rel 0.050, Sq Rel 0.317, RMSE 4.141, RMSE log 0.070, $R$6 0.982, $R$7 0.999, and $R$8 1.000. Relative to MonoPCC, Sq Rel is reduced by 9.17% and RMSE by 7.73%; relative to EndoDAC, Sq Rel is reduced by 17.02% and RMSE by 11.04%. The model size is reported as approximately 99.10M total parameters with only 1.67M trainable, about 1.7% of the depth network parameters (Yao et al., 25 Aug 2025).

On Hamlyn, EndoUFM again outperforms all competitors, with Abs Rel 0.071, Sq Rel 0.445, RMSE 4.280, RMSE log 0.089, $R$9 0.973, $S$0 0.998, and $S$1 0.999. The result is described as especially notable because Hamlyn is more difficult due to lower resolution, complex lighting, and larger camera motion. On SERV-CT, where the model is trained on SCARED and directly tested without fine-tuning, the reported results are Abs Rel 0.071, Sq Rel 0.791, RMSE 8.025, RMSE log 0.092, $S$2 0.965, $S$3 0.999, and $S$4 1.000, achieving the best or near-best performance across all metrics. On EndoNeRF, again transferred without fine-tuning, the reported results are Abs Rel 0.163, Sq Rel 2.926, RMSE 15.818, RMSE log 0.207, $S$5 0.728, $S$6 0.976, and $S$7 0.997; the method is best on most metrics and remains highly competitive.

The ablation study attributes the final result to the full combination of Depth Anything + RVLoRA + Res-DSC, MedSAM + RVLoRA, and the mask-guided smoothness loss. The best configuration yields Abs Rel 0.050, Sq Rel 0.317, RMSE 4.141, RMSE log 0.070, and $S$8 0.982 on SCARED. The same study reports that SAM/MedSAM improves decomposition, that RVLoRA significantly improves performance, that Res-DSC further improves the depth branch, and that applying Res-DSC to SAM hurts performance. For pose estimation, EndoUFM also achieves the best ATE on three SCARED sequences: 0.0424 on Seq 1, 0.0537 on Seq 2, and 0.0516 on Seq 3. The paper interprets this as improved geometric consistency beyond depth maps alone.

The practical significance identified by the work is direct relevance to minimally invasive surgery, 3D reconstruction, augmented reality, navigation systems, and robotic surgery. It states that the method contributes to augmenting surgeons’ spatial perception during minimally invasive procedures, thereby enhancing surgical precision and safety, with implications for augmented reality and navigation systems. It also suggests possible support for point cloud initialization, downstream NeRF reconstruction, 3D Gaussian Splatting, and future surgical scene understanding. A plausible implication is that EndoUFM’s main contribution is not only improved benchmark accuracy but a concrete template for adapting general-purpose foundation models to photometrically unstable, semantically specialized medical imaging domains without fully retraining them.