DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model (2408.17433v2)

Abstract: Robotic-assisted surgery (RAS) relies on accurate depth estimation for 3D reconstruction and visualization. While foundation models like Depth Anything Models (DAM) show promise, directly applying them to surgery often yields suboptimal results. Fully fine-tuning on limited surgical data can cause overfitting and catastrophic forgetting, compromising model robustness and generalization. Although Low-Rank Adaptation (LoRA) addresses some adaptation issues, its uniform parameter distribution neglects the inherent feature hierarchy, where earlier layers, learning more general features, require more parameters than later ones. To tackle this issue, we introduce Depth Anything in Robotic Endoscopic Surgery (DARES), a novel approach that employs a new adaptation technique, Vector Low-Rank Adaptation (Vector-LoRA) on the DAM V2 to perform self-supervised monocular depth estimation in RAS scenes. To enhance learning efficiency, we introduce Vector-LoRA by integrating more parameters in earlier layers and gradually decreasing parameters in later layers. We also design a reprojection loss based on the multi-scale SSIM error to enhance depth perception by better tailoring the foundation model to the specific requirements of the surgical environment. The proposed method is validated on the SCARED dataset and demonstrates superior performance over recent state-of-the-art self-supervised monocular depth estimation techniques, achieving an improvement of 13.3% in the absolute relative error metric. The code and pre-trained weights are available at https://github.com/mobarakol/DARES.

• The paper introduces Vector-LoRA, a novel approach that allocates more parameters to earlier network layers to better adapt foundation models for robotic-assisted surgery.
• It implements a multi-scale SSIM-based reprojection loss that significantly improves depth and pose estimation in complex endoscopic scenes.
• DARES outperforms state-of-the-art methods by reducing the absolute relative error by 13.3%, demonstrating enhanced robustness and generalizability.

Overview of DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model

The paper "DARES: Depth Anything in Robotic Endoscopic Surgery with Self-supervised Vector-LoRA of the Foundation Model" introduces an advanced framework for monocular self-supervised depth estimation in robotic-assisted surgery (RAS). Depth estimation in RAS is crucial for 3D reconstruction and precise visualization, which are essential for surgical navigation and improved clinical outcomes.

Background and Main Contributions

The paper critiques the utilization of foundation models like Depth Anything Models (DAM) in surgical scenes, noting that fully fine-tuning these models can lead to overfitting and catastrophic forgetting. This negatively impacts the robustness and generalizability of the models. Although Low-Rank Adaptation (LoRA) has addressed some adaptation issues, its uniform parameter distribution is suboptimal because it overlooks the feature hierarchy, where earlier layers need more parameters compared to later layers.

To address these issues, the authors propose DARES, which adapts DAM V2 via a novel technique called Vector-LoRA. Vector-LoRA integrates more parameters in earlier layers of the network and progressively decreases parameters in later layers. This approach aligns with the inherent feature hierarchy of deep networks. Additionally, the paper introduces a new reprojection loss based on multi-scale Structural Similarity Index (SSIM) to enhance depth perception in RAS scenes.

The main contributions of this work are fourfold:

1. Adapting the complete architecture of DAM V2 for RAS in a self-supervised learning (SSL) manner to enhance depth estimation without extensive labeled data.
2. Introducing Vector-LoRA, which efficiently adapts foundation models by accounting for feature hierarchy and gradient flow dynamics.
3. Designing a multi-scale SSIM-based reprojection loss function to enhance the performance of depth estimation.
4. Demonstrating the superior performance of their model over state-of-the-art (SOTA) methods with a 13.3% improvement in the absolute relative error metric.

Methodology

DAM V2 and Vector-LoRA

The paper uses DAM V2, which features a DINOv2 transformer-based encoder for feature extraction and a Dense Prediction Transformer (DPT) decoder for depth regression. The encoder consists of 12 multi-headed self-attention blocks interspersed with multi-layer perceptron (MLP) blocks and normalization layers.

Vector-LoRA improves upon traditional LoRA by allocating a unique rank to each layer, giving more parameters to earlier layers. This method enhances the adaptation capability of the model in feature hierarchies and makes it more effective for RAS-specific tasks.

Multi-scale SSIM Reprojection Loss

To further fine-tune the model for the complex RAS environment, the authors implement a multi-scale SSIM-based reprojection loss. This loss function is designed to handle the specific challenges in RAS scenes, such as intricate tissue textures and varying lighting conditions. By processing image pairs through iterative filtering and downsampling, the multi-scale SSIM provides a robust measure of image similarity, significantly improving the depth perception.

Experimental Results

The proposed DARES model was evaluated on the SCARED dataset, which comprises endoscopic sequences from porcine cadavers. The model exhibited a marked improvement in depth estimation over several baseline methods, such as DeFeat-Net, SC-SfMLearner, Monodepth2, Endo-SfM, and AF-SfMLearner. The results, shown in Table 1, demonstrate that DARES outperforms these methods, achieving superior performance metrics across multiple evaluation criteria, including absolute relative error (Abs Rel), squared relative error (Sq Rel), Root Mean Square Error (RMSE), and more.

The fully fine-tuned DAM V2 performed worse than DARES, emphasizing the necessity of a strategic approach to foundation model adaptation. The paper's experimentation also included a qualitative assessment of depth estimation and pose estimation, where DARES showed fewer artifacts and more accurate ego-motion estimation compared to SOTA methods.

Conclusion and Future Directions

DARES effectively addresses the key challenges of adapting foundation models to the RAS domain. The proposed Vector-LoRA technique ensures efficient parameter distribution across network layers, enhancing the model's robustness and generalization capabilities. The multi-scale SSIM reprojection loss further tailors the model to the specific requirements of RAS scenes, achieving state-of-the-art results in both depth and pose estimation tasks.

Future research could focus on refining these models for even greater robustness and reliability in endoscopic environments. Integrating additional strategies such as GaLore and MoRA could potentially lead to further improvements, making these models more suitable for a wider range of surgical applications. The insights and methodologies presented in this paper could serve as a solid foundation for ongoing advancements in the field of robotic-assisted surgery and medical imaging.