SAGE: SLAM with Appearance and Geometry Prior for Endoscopy (2202.09487v2)

Abstract: In endoscopy, many applications (e.g., surgical navigation) would benefit from a real-time method that can simultaneously track the endoscope and reconstruct the dense 3D geometry of the observed anatomy from a monocular endoscopic video. To this end, we develop a Simultaneous Localization and Mapping system by combining the learning-based appearance and optimizable geometry priors and factor graph optimization. The appearance and geometry priors are explicitly learned in an end-to-end differentiable training pipeline to master the task of pair-wise image alignment, one of the core components of the SLAM system. In our experiments, the proposed SLAM system is shown to robustly handle the challenges of texture scarceness and illumination variation that are commonly seen in endoscopy. The system generalizes well to unseen endoscopes and subjects and performs favorably compared with a state-of-the-art feature-based SLAM system. The code repository is available at https://github.com/lppllppl920/SAGE-SLAM.git.

• The paper introduces SAGE, a novel SLAM system for endoscopy that integrates deep learning-based appearance and geometry priors to tackle imaging challenges.
• It leverages a dual-network approach that refines depth estimates and generates robust feature descriptors to handle texture scarcity and illumination variations.
• Experimental results demonstrate significant improvements over ORB-SLAM3, reducing average translation error to 1.6 mm and rotation error to 22.2° for enhanced surgical navigation.

SAGE: SLAM with Appearance and Geometry Prior for Endoscopy

The paper introduces a Simultaneous Localization and Mapping (SLAM) system named SAGE that is specifically designed for use in the endoscopic environment. This system combines learning-based appearance and optimizable geometry priors with factor graph optimization to address challenges unique to endoscopic video analysis, such as texture scarcity and significant illumination variation. Developed for monocular endoscopic videos, this system offers potential improvements in surgical navigation by enabling real-time tracking and reconstruction of the 3D geometry of observed anatomical structures.

Key Innovations and Methodology

The SAGE system integrates deep learning techniques with traditional non-linear optimization to enhance the robustness and accuracy of SLAM in challenging conditions. The system utilizes two neural networks: one for geometry and another for appearance. Key innovations include:

• Geometry Representation: A depth network produces both average depth estimates and depth variations (bases), enabling refinement during SLAM optimization.
• Appearance Representation: A feature network generates descriptor and feature maps, facilitating correspondence matching and robustness to lighting changes.
• Differentiable Optimization: The use of the Levenberg-Marquardt algorithm supports a learning process that incorporates complex non-linear error metrics.
• Comprehensive Training Pipeline: The pipeline involves separate and joint training phases for the networks using innovative losses, such as adversarial losses and triplet histogram loss, to optimize for image alignment tasks.

The system is tested in cross-subject experiments to ensure generalization to unseen endoscopes and anatomical subjects, demonstrating superior performance compared to ORB-SLAM3 across several metrics, such as trajectory estimation accuracy and robustness to diverse endoscopic conditions.

Numerical Results

Crucially, the proposed method outperformed ORB-SLAM3 with significant improvements across key performance indicators:

• Translation and Rotation Errors: Average translation error (ATE) improved to 1.6 mm, while average rotation error (ATE) reached 22.2 degrees.
• Relative Pose Error: Better consistency in motion estimation, suggesting higher resilience to textured-scarce scenes.
• Depth Accuracy: The absolute relative difference was notably reduced, indicating better trajectory-scaled depth reproduction.

Practical and Theoretical Implications

The proposed system holds potential for advancing surgical navigation by providing enhanced precision and reliability in reconstructing the complex topology of the sinus anatomy during endoscopic procedures. The robust integration of learned priors supports a range of endoscopic applications, promising non-intrusive, real-time navigation solutions that align pre-operative models with intra-operative findings.

Theoretically, SAGE exemplifies the effective amalgamation of deep learning models with traditional SLAM optimization, paving the way for similar applications across varied domains. The demonstrated success in adapting neural networks for real-time SLAM applications suggests broader implications for robotics and autonomous systems, potentially influencing factors such as hardware requirement relaxation and sensor versatility.

Future Developments

The system's performance might further improve by addressing current limitations, such as strict global loop detection to prevent erroneous keyframe associations. Future work could involve extending the SLAM framework to accommodate dynamic and deformable environments typical of laparoscopic surgeries using deformation modeling techniques.

The research presented in this paper establishes a strong foundation for ongoing developments in visual SLAM applications leveraging deep learning advancements, particularly within the medical imaging domain. The possibility of refining the precision and operation of endoscopic navigation systems heralds improved patient outcomes in minimally invasive surgeries.