Cross-Modal Benchmarking for Robotic Perception in Natural Environments

Abstract: Natural environments present a complex challenge to robotics perception systems. Current models, particularly vision foundation models, are largely trained on structured, urban environments leading to weaknesses in their perception for field robotics tasks. We showcase the limitations of current models using our recently released WildCross benchmark, a new cross-modal benchmark for place recognition and metric depth estimation in large-scale natural environments. WildCross comprises over 476K sequential RGB frames with semi-dense depth and surface normal annotations, each aligned with accurate 6DoF pose and synchronized dense lidar submaps. In this work, we provide an expanded analysis of the benchmark results from the recent WildCross benchmark, with particular emphasis on expanded metric depth estimation experiments. Access to the code repository and dataset for this work can be found at https://csiro-robotics.github.io/WildCross.

• The paper introduces WildCross, a comprehensive benchmark leveraging over 476K RGB frames and dense lidar submaps to evaluate cross-modal robotic perception in natural environments.
• It rigorously tests visual place recognition, cross-modal localization, and metric depth estimation under challenging reverse revisit and occlusion conditions.
• Results expose significant domain gaps compared with urban datasets, highlighting the need for advanced multi-modal fusion and viewpoint-robust feature modeling.

Cross-Modal Benchmarking for Robotic Perception in Natural Environments: A Detailed Analysis

Introduction and Benchmark Motivation

The gap between robotics perception in structured and natural environments remains highly significant. The paper "Cross-Modal Benchmarking for Robotic Perception in Natural Environments" (2606.11563) addresses this by introducing and analyzing the WildCross benchmark, specifically targeted at evaluating visual and cross-modal perception systems in unstructured outdoor scenes, such as bushwalking trails with dense vegetation, irregular terrain, and pronounced occlusions. The main goal is to provide a rigorous and high-fidelity testbed for large-scale place recognition and metric depth estimation, explicitly extending beyond the biases of urban-focused datasets like KITTI or Oxford RobotCar.

WildCross leverages over 476K sequential RGB frames, semi-dense depth and normal annotations, precisely aligned 6-DoF poses, and dense lidar submaps collected across multiple traversals of two forest sites near Brisbane, Australia. This diversity and complexity exposes real-world weaknesses in foundation models and state-of-the-art (SOTA) robotic perception systems that are rarely tested on such terrain.

Figure 1: WildCross data overview, encapsulating the modalities and spatial diversity across complex natural habitats.

Dataset Construction and Cross-Modal Ground Truth

WildCross is constructed by reprocessing the raw traversals from the earlier Wild-Places dataset, augmenting them with:

• Rectified, sequential RGB frames (15Hz)
• Synchronized lidar-inertial SLAM-derived submaps
• High-precision 6-DoF ground truth
• Automated, semi-dense metric depth and normal mapping for every frame

Annotations are meticulously generated using visibility-filtered, point cloud projections from aggregated lidar, explicitly occlusion-aware. Notably, the dataset supports joint evaluation and training of visual (VPR), cross-modal (CMPR), and geometric (depth) tasks under consistent splits, facilitating multi-task and transfer learning studies.

Experimental Protocols and Task Definitions

Split Protocol: The dataset follows a four-fold cross-validation schema, with each fold holding out complete forward and reverse traversals across both sites, thus maximizing cross-domain challenge, including forward/reverse revisit scenarios with minimal image overlap.

Visual Place Recognition (VPR): Four SOTA models are evaluated—NetVLAD, MixVPR, DINOv2-SALAD, and BoQ—using both zero-shot (OOD) and fine-tuned (in-domain) settings. Positive/negative pairs are carefully defined based on both spatial proximity and heading, countering false positives inherent to forested, viewpoint-divergent sequences.

Cross-Modal Place Recognition (CMPR): Image-to-lidar localization is assessed via a modified LIP-Loc architecture, comparing ResNet, DINOv2, and DINOv3 backbone effects within the WildCross split regime.

Metric Depth Estimation: The evaluation adopts DepthAnythingV2 as a baseline, testing both zero-shot transfer from urban-trained models and multiple fine-tuning regimes. Sparse GT, RANSAC-rescaled outputs, metric-aligned outputs, and PriorDA pseudo-ground-truth are comprehensively benchmarked, reflecting real-world annotation sparsity and modality fusion challenges.

Visual Place Recognition Results

Quantitative performance of SOTA VPR models is drastically lower in WildCross compared to urban datasets. BoQ, the top-performing architecture, achieves 63.17% R1 for intra-sequence and 61.87% for inter-sequence (after in-domain fine-tuning), versus >90% on Pittsburgh or MSLS (2606.11563).

• Zero-shot evaluation exposes substantial domain gaps; fine-tuning yields marked improvement, but absolute accuracy remains limited.
• Reverse revisit scenarios dominate the error landscape, reinforcing the need for viewpoint-invariant and context-aware feature aggregation in natural settings.
• The strong correlation between trajectory structure (presence of reverse revisits) and accuracy collapse is highlighted as a unique challenge not present in prior benchmarks.

Cross-Modal Place Recognition Analysis

LIP-Loc equipped with DINOv3 backbone achieves the best cross-modal performance at 52.35% average R1, with all backbone variants performing well below urban benchmarks. This underscores the compounded difficulty of aligning cross-modal features when faced with dense occlusions, low ground texture diversity, and viewpoint divergence prevalent in natural landscapes.

• Backbone improvements alone are not sufficient; the domain gap between 2D image features and 3D structural/point cloud representations requires explicit bridging methodologies.
• The authors advocate for hybrid spatial-structural learning and domain-adversarial adaptation approaches as directions for future work.

Metric Depth Estimation Benchmarking

Monocular depth estimation in WildCross is characterized by large absolute errors and significant performance degradation versus results observed on synthetic or urban indoor datasets. Notable findings include:

• Zero-shot DepthAnythingV2 yields $\delta_1$ as low as 0.284, with AbsRel and RMSE remaining high, underscoring insufficient domain generalization.
• Fine-tuning on WildCross sparse GT drastically improves results ($\delta_1$ up to 0.789 for ViT-L), but at the expense of fine-grained detail (as confirmed by qualitative visualizations).
• Pseudo-GT (PGT) fine-tuning using RANSAC-rescaled, metric, or PriorDA methods does not surpass direct GT fine-tuning but does improve sharpness and generalization of fine scene details.

  Figure 2: DepthAnythingV2 outputs (zero-shot and fine-tuned), compared against dense GT; fine detail loss and domain-specific artifacts are visually apparent.

• PriorDA-based pseudo-GT demonstrates the best trade-off, balancing quantitative accuracy and preservation of fine scene details, yet still falls short of robust, fully-dense supervised models.

Implications, Limitations, and Future Directions

Practical and Theoretical Implications

• The WildCross benchmark reveals persistent and significant limitations in both VPR and depth models when transitioning to unstructured natural environments.
• The results suggest that monocular depth models are currently inadequate for forested robotics deployments where high-fidelity metric ground truth is unobtainable. Reliance solely on such models cannot supplant lidar or stereo systems for precision autonomy.
• Reverse revisit failure cases identify the importance of developing models with enhanced viewpoint invariance, potentially via trajectory-consistent learning or scene graph reasoning.

Recommendations for Future Research

• Novel architectures explicitly integrating multi-modal spatial context, global place priors, and 3D geometric constraints may be required to address the full complexity of natural terrains.
• Domain-adaptive and semi-supervised approaches able to leverage pseudo or sparse GT, while maintaining scale and fine boundary accuracy, are a critical avenue.
• Expanding the breadth and diversity of natural scene datasets to include more ecological, seasonal, and weather variability will allow for a more generalizable assessment of progress on these tasks.

Conclusion

WildCross (2606.11563) establishes a new difficulty standard for perception in natural environments by providing a comprehensive, multi-modal, and rigorously annotated benchmark. Evaluation of SOTA algorithms across VPR, CMPR, and depth estimation reveals concrete gaps in domain transfer and model reliability. While fine-tuning and pseudo-GT generation yield improvements, there is a clear need for foundational advances in representation learning, multi-modal fusion, and viewpoint-robust feature modeling. WildCross thus constitutes a critical resource driving progress towards robust, reliable field robotic perception under true environmental complexity.