Multi-modal Loop Closure Detection with Foundation Models in Severely Unstructured Environments (2511.05404v1)

Abstract: Robust loop closure detection is a critical component of Simultaneous Localization and Mapping (SLAM) algorithms in GNSS-denied environments, such as in the context of planetary exploration. In these settings, visual place recognition often fails due to aliasing and weak textures, while LiDAR-based methods suffer from sparsity and ambiguity. This paper presents MPRF, a multimodal pipeline that leverages transformer-based foundation models for both vision and LiDAR modalities to achieve robust loop closure in severely unstructured environments. Unlike prior work limited to retrieval, MPRF integrates a two-stage visual retrieval strategy with explicit 6-DoF pose estimation, combining DINOv2 features with SALAD aggregation for efficient candidate screening and SONATA-based LiDAR descriptors for geometric verification. Experiments on the S3LI dataset and S3LI Vulcano dataset show that MPRF outperforms state-of-the-art retrieval methods in precision while enhancing pose estimation robustness in low-texture regions. By providing interpretable correspondences suitable for SLAM back-ends, MPRF achieves a favorable trade-off between accuracy, efficiency, and reliability, demonstrating the potential of foundation models to unify place recognition and pose estimation. Code and models will be released at github.com/DLR-RM/MPRF.

• The paper presents MPRF which integrates fine-tuned DINOv2 and transformer-based SONATA to achieve robust loop closure detection in GNSS-denied, unstructured environments.
• It employs a two-stage pipeline where efficient visual retrieval using SALAD aggregation is followed by geometric verification via RANSAC for reliable 6-DoF pose estimation.
• Experimental results on S3LI datasets show significant improvements in precision and pose accuracy over unimodal methods, enabling practical SLAM integration.

Multi-modal Loop Closure Detection Leveraging Foundation Models in Unstructured Environments

Introduction

Robust loop closure detection is a pivotal challenge in SLAM, especially in GNSS-denied and extremely unstructured environments encountered in planetary exploration. Conventional visual place recognition is hindered by aliasing and low-texture scenes, while LiDAR-based methods suffer from data sparsity and scene ambiguity. The paper presents MPRF, a multimodal pipeline that leverages transformer-based foundation models for vision and LiDAR to facilitate robust loop closure detection, integrating both efficient retrieval and explicit 6-DoF pose estimation.

Methodology

Pipeline Architecture

MPRF operates via two principal stages:

1. Visual Retrieval: DINOv2, a self-supervised ViT, extracts patch-level descriptors, which are globally aggregated using SALAD (Sinkhorn-based optimal transport aggregation). Candidate frames are retrieved efficiently using FAISS approximate nearest neighbor search, with refinement performed by aggregating multi-layer patch embeddings and recalculating cosine similarity.
2. Geometric Verification & Pose Estimation: SONATA, a transformer-based point cloud encoder, generates LiDAR descriptors. Image patches are projected into 3D using LiDAR depth and camera intrinsics, pairing visual and geometric descriptors. Correspondence matching utilizes cosine similarity and Hungarian assignment, followed by RANSAC (Open3D) for SE(3) pose estimation.

This architecture ensures that candidates are filtered via visual discrimination before explicit geometric validation using multimodal correspondences.

Visual and LiDAR Feature Extraction

• DINOv2 Extraction/Fine-Tuning: Images are resized to $224 \times 224$ and passed through ViT-B/14 DINOv2. For planetary-like domains, DINOv2 is fine-tuned using triplet loss (anchor-positive-negative) on S3LI datasets, with margin $m=0.2$ and extensive online augmentation.
• SALAD Aggregation: Patch embeddings are aggregated into 8192-dim global descriptors via learnable VLAD clustering with entropy regularization.
• SONATA: LiDAR scans are encoded into 512-dim descriptors with voxel-based attention, suitable for sparse noisy point clouds.

Multimodal Fusion and Pose Estimation

Fusion is performed by projecting image patches into 3D and concatenating normalized visual and LiDAR descriptors. Matches are established with a similarity threshold (0.90). Pose estimation employs RANSAC on point correspondences, optimizing rigid transformations with $n=3$ minimal set correspondences and a threshold of 0.05 m.

Loop Closure Decision

Loop closure is confirmed if geometric consistency is achieved after RANSAC, with candidates further ranked by estimated pose distance, enabling direct integration into SLAM back-ends.

Experimental Analysis

Datasets and Evaluation Metrics

Experiments are conducted on S3LI and S3LI Vulcano datasets, which feature traversals in true planetary analog sites with challenging visual and geometric properties (low texture, aliased features). Evaluation focuses on Precision@k for retrieval and mean errors in yaw ($\Delta \theta$) and planar translation ($\Delta x, \Delta y$) for pose estimation.

Retrieval Performance

• Retrieval Accuracy: Fine-tuned DINOv2 + pretrained SALAD (MPRF-PF) achieves 75.7% Precision@1 on S3LI, substantially surpassing unimodal and classical baselines. On S3LI Vulcano, Precision@1 reaches 78.3%.
• Runtime: End-to-end retrieval time is below 500 ms per query, achieving real-time capabilities for planetary rovers and field robots.
• LiDAR-only Methods: PointNetVLAD underperforms (7.9% Precision@1, >4s per query), confirming the limited utility of geometry-only retrieval in planetary terrains.

Pose Estimation

• Accuracy and Reliability: MPRF yields mean yaw error of 8.2°, with 69.9% of estimates within 10°, matching regression-based (Reloc3r) and dense-matching (LoFTR) methods in mean accuracy, but with superior interpretability due to explicit correspondence generation.
• Translation Error: Translation accuracy is modest (~8.4 m), attributed to inherent scene structure and sensor noise.
• Robustness: Valid SE(3) poses are computed for all candidate pairs, with reduced catastrophic failures compared to unimodal approaches.

Ablation and Design Insights

• Fine-Tuning & Aggregation: Domain adaptation via DINOv2 fine-tuning increases retrieval precision by >4%. Multi-layer patch aggregation trims 5% more accuracy versus CLS-only descriptors.
• SALAD Training: Retraining aggregation on limited data degrades performance, highlighting generalization strength of pretrained clustering.
• Feature Fusion: Independent use of DINOv2 or SONATA for pose prediction is insufficient; fusion harnesses complementary cues for reliable loop closure.
• Interpretability: Unlike regression-based methods, MPRF outputs verifiable correspondences, crucial for SLAM systems requiring validation.

Implications and Future Directions

The unification of foundation models across modalities marks a notable advance in bridging place recognition and geometric SLAM. By extracting transferable descriptors from self-supervised large-scale pretraining, the requirement for domain-specific annotation is substantially mitigated. The pipeline’s integration of interpretable geometric verification facilitates downstream usage in SLAM graphs and trajectory optimization.

Potential future work includes:

• Accelerating geometric verification for online applications with parallelization and sparse sampling strategies.
• Exploring SLAM back-end architectures that exploit multimodal correspondences for joint optimization.
• Investigating additional sensor modalities (e.g., radar, thermal) to further improve robustness in harsh conditions.
• Leveraging foundation models for lifelong adaptation and unsupervised domain transfer in long-term deployments.

Conclusion

MPRF demonstrates a robust approach for multi-modal loop closure detection by integrating visually discriminative self-supervised descriptors with geometric verification from transformer-based LiDAR features. Fine-tuned visual features coupled with pretrained aggregation deliver state-of-the-art retrieval, while multimodal fusion enables accurate and explainable 6-DoF pose estimation. Experimental results confirm that foundation models unify appearance and geometry cues, improving SLAM reliability in unstructured environments. Future work will address computational efficiency and full integration within end-to-end multimodal SLAM frameworks.