M2H-MX: Multi-Task Dense Visual Perception for Real-Time Monocular Spatial Understanding

Abstract: Monocular cameras are attractive for robotic perception due to their low cost and ease of deployment, yet achieving reliable real-time spatial understanding from a single image stream remains challenging. While recent multi-task dense prediction models have improved per-pixel depth and semantic estimation, translating these advances into stable monocular mapping systems is still non-trivial. This paper presents M2H-MX, a real-time multi-task perception model for monocular spatial understanding. The model preserves multi-scale feature representations while introducing register-gated global context and controlled cross-task interaction in a lightweight decoder, enabling depth and semantic predictions to reinforce each other under strict latency constraints. Its outputs integrate directly into an unmodified monocular SLAM pipeline through a compact perception-to-mapping interface. We evaluate both dense prediction accuracy and in-the-loop system performance. On NYUDv2, M2H-MX-L achieves state-of-the-art results, improving semantic mIoU by 6.6% and reducing depth RMSE by 9.4% over representative multi-task baselines. When deployed in a real-time monocular mapping system on ScanNet, M2H-MX reduces average trajectory error by 60.7% compared to a strong monocular SLAM baseline while producing cleaner metric-semantic maps. These results demonstrate that modern multi-task dense prediction can be reliably deployed for real-time monocular spatial perception in robotic systems.

• The paper presents a unified framework that leverages a DINOv3 backbone with LoRA adaptation to achieve real-time dense depth and semantic predictions from a monocular RGB image.
• The architecture uses hierarchical feature adaptation, register-gated Mamba blocks, and cross-task mixing to deliver improved mIoU and RMSE results on benchmarks like NYUDv2 and Cityscapes.
• Integration with fixed SLAM systems demonstrates significant tracking accuracy improvements, reducing Absolute Trajectory Error by over 60% on ScanNet.

M2H-MX: Multi-Task Dense Visual Perception for Real-Time Monocular Spatial Understanding

Introduction and Motivation

M2H-MX introduces a monocular dense perception approach aimed explicitly at practical, real-time robotic systems where a single RGB camera forms the sensory backbone. The primary focus is on enabling accurate, low-latency, and stable geometry and semantic inference without modification to downstream SLAM or mapping components. This is motivated by the necessity for perception modules in real-world robotic deployments to conform to strict interface and latency budgets, while maximizing the quality of dense predictions for robust camera tracking and 3D metric-semantic mapping.

Contemporary research in multi-task dense scene understanding has demonstrated clear benefits from unified depth and semantic segmentation models, particularly through advanced architectures that exploit shared and cross-task representations. However, most models either operate outside real-time constraints, lack frame-to-frame temporal stability, or rely on modalities not commonly available in low-cost mobile platforms. M2H-MX addresses this confluence of requirements through an architecture that leverages foundation visual models, efficient adaptation, and custom decoding modules designed to operate synergistically for dense, multi-task prediction.

Architecture Overview

M2H-MX's core pipeline accepts a monocular RGB image and predicts dense per-pixel depth and semantics, with optional auxiliary outputs (normals, edges). The model comprises a frozen DINOv3 ViT backbone with parameter-efficient Low-Rank Adaptation (LoRA), a hierarchical feature adapter that builds a multi-scale representation, and a lightweight decoder characterized by register-gated Mamba blocks, task adaptors, cross-task mixing, and convolutional attention refinement.

Figure 1: The M2H-MX architecture, where DINOv3 foundation features are adapted via LoRA and reassembled into a multi-scale decoding pyramid incorporating register-gated Mamba, cross-task mixers, and multi-scale attention.

LoRA is specifically applied only to the backbone's last layers—targeting adaptation with minimal learned parameters and enabling domain transfer while preserving generalization. The hierarchical feature adapter organizes DINOv3 features into a four-level spatial pyramid for coarse-to-fine information flow. Global scene context is encoded via register tokens, further distilled to a compact scene vector used for channel-wise gating in the decoder.

The decoder performs efficient long-range feature propagation at each scale through register-gated Mamba blocks—this mechanism injects global context while ensuring that architectural complexity remains bounded relative to transformer-based alternatives. Each pyramid level produces task-specific intermediate features, with dedicated adaptors and task heads. Task representations are further processed through controlled cross-task mixing (gated asymmetric fusion) and multi-scale convolutional attention, allowing depth and semantics to reinforce each other within spatially localized neighborhoods.

Figure 2: The Register-Gated Mamba (RGM) module efficiently integrates global scene context at each scale, modulating features by register-driven channel gates prior to state-space Mamba and feed-forward operations.

Figure 3: (a) Cross-Task Mixing (CTM) module enables gated asymmetric injection of auxiliary features; (b) Multi-Scale Convolutional Attention (MSCA) hierarchically refines spatial correlations.

Prediction heads are lightweight, with the depth branch employing learnable bin-based decoders (as in DPT-like models) and the semantic segmentation branch using a minimal convolutional stack. Loss balancing across tasks and optional geometric consistency regularization is managed via uncertainty-based weighting.

System Integration

A key practical aspect of M2H-MX is the explicit design of the perception-to-mapping interface. Rather than modifying mapping or SLAM algorithms, M2H-MX emits RGB-aligned dense depth and semantic predictions, which are ingested by a fixed, real-time SLAM system (Mono-Hydra pipeline). Perception and inference execute asynchronously on a GPU, delivering predictions with stable latency, while mapping, state estimation, and scene graph construction execute on the CPU—mirroring practical deployment conditions.

Figure 4: M2H-MX deployed as a perception module in a monocular SLAM pipeline, delivering dense geometric and semantic cues to fixed mapping and odometry components.

Experimental Results

Dense Prediction Benchmarks

On NYUDv2, M2H-MX-L outperforms prior SOTA multi-task methods in both semantic segmentation (mIoU, $+4.06$ points over M2H) and depth estimation (relative RMSE reduction of $9.4\%$), demonstrating that the model design induces significant improvements in per-frame scene understanding. On Cityscapes, the gains persist, with mIoU improved by $+3.15$ points over MTMamba++ and disparity RMSE reduced to 3.89. These consistent improvements underscore the architectural merits of register-gated decoding, backbone adaptation via LoRA, and carefully modulated cross-task interaction.

System-Level Real-Time SLAM Evaluation

Direct system integration tests—measured on ScanNet with a fixed monocular SLAM—establish the practical impact of improved dense perception. The average Absolute Trajectory Error (ATE) is reduced from $17.59\,\mathrm{cm}$ (using the prior Go-SLAM monocular front-end) to $6.91\,\mathrm{cm}$ with M2H-MX, a $60.7\%$ improvement in tracking accuracy. These results are obtained without modification to downstream mapping components, validating the effective translation of dense prediction quality improvements into global system behavior and robustness.

Figure 5: Qualitative mapping comparison reveals that M2H-MX produces cleaner surface geometry and improved semantic consistency versus prior SLAM models.

Ablation and Architectural Analysis

Ablation studies confirm that the majority of performance gains arise from synergistic design choices rather than simply increasing model capacity. Removing CTM and MSCA degrades mIoU by $2.07$ points and increases depth RMSE, indicating that both cross-task fusion and spatial refinement are essential in balancing accuracy and resource constraints. Register-gated decoding—anchoring each scale to a global context—delivers a further $1.16$ point boost in mIoU. Most critically, substituting the DINOv3 foundation with weaker alternatives results in a catastrophic drop in semantic performance (up to $-26.8$ points in mIoU for ConvNeXt-L), underscoring the dependency on globally stable, high-capacity visual features.

Theoretical and Practical Implications

M2H-MX reifies the hypothesis that advances in dense multi-task learning translate directly into quantifiable improvements in SLAM and spatial scene understanding—provided the perception architecture is explicitly constrained by integration and latency requirements. The results suggest that ongoing progress in vision foundation models, efficient adaptation techniques, and architectural innovations in decoder design (such as register-gated state-space models) enable monocular perception stacks ready for real-world robotics.

The strong coupling between backbone representation quality (e.g., DINOv3 with Gram Anchoring) and decoding efficiency highlights a promising direction wherein minimal adaptation via LoRA suffices to unlock substantial downstream benefits—challenging the necessity of using heavy, attention-dominated decoders in these settings.

Future Directions

Extensions to M2H-MX may involve integrating additional spatial or geometric tasks (e.g., instance segmentation, panoptic mapping, 3D object detection), cross-dataset generalization, self-supervised adaptation during deployment, and leveraging richer cross-modal (e.g., language or tactile) cues. Real-time constraints, explicit perception-to-action loop integration, and closed-loop feedback between mapping uncertainty and perception remain compelling open directions for further aligning multi-task learning with long-term, robust embodied reasoning.

Conclusion

M2H-MX demonstrates the potential for foundation model-driven, multi-task perception stacks to enable real-time, stable, and accurate monocular spatial understanding in robotic applications. The careful integration of backbone adaptation, register-gated decoding, efficient cross-task mixing, and system-aware design yields significant improvements in both dense predictive performance and real-world system-level SLAM metrics. These findings highlight the burgeoning intersection of large-scale visual pretraining and practical, latency-bounded robotics, inviting further synthesis of vision and spatial AI research in scenarios constrained by cost, deployment complexity, and integration requirements.

(2603.29236)