M2H-MX Model for Multi-Task Visual Perception

• The M2H-MX model is a state-of-the-art multi-task visual perception system that delivers real-time monocular spatial understanding using a frozen DINOv3 backbone and lightweight decoders.
• It employs advanced components such as the Register-Gated Mamba decoder and cross-task fusion modules to integrate global context with localized features for accurate depth and semantic estimation.
• M2H-MX achieves competitive benchmarks on datasets like NYUDv2 and ScanNet while maintaining low computational latency, supporting efficient deployment in resource-constrained robotic SLAM systems.

The M2H-MX model is a state-of-the-art multi-task dense visual perception architecture designed for real-time monocular spatial understanding, explicitly tailored for resource-constrained robotic applications. Leveraging a frozen DINOv3 Vision Transformer as its backbone and advanced lightweight decoding modules, M2H-MX delivers competitive accuracy in per-pixel depth and semantic estimation while maintaining low computational latency. The model is integral to the Mono-Hydra++ pipeline for indoor 3D mapping and scene graph construction, providing reliable metric-semantic maps from monocular RGB and IMU inputs—thereby obviating the need for active depth sensors (Udugama et al., 31 Mar 2026, Udugama et al., 17 May 2026).

1. Model Architecture and Design

M2H-MX adopts a large, pretrained DINOv3 Vision Transformer (ViT-B or ViT-L) as the backbone. Only the high-level transformer layers (blocks 13–24 out of 24) are adapted via Low-Rank Adaptation (LoRA) modules, ensuring efficient fine-tuning for task-specific features while freezing the bulk of the backbone parameters. This approach controls memory footprint and training cost, and supports deployment on embedded platforms (Udugama et al., 17 May 2026).

Feature extraction proceeds via a Hierarchical Feature Adapter (HFA), which taps intermediate transformer layers to form multiscale spatial maps. These are assembled into a top-down decoder feature pyramid, facilitating hierarchical context aggregation. Register tokens from the final ViT layer are average-pooled to yield a global context vector used throughout the decoder.

The decoder is structured around the Register-Gated Mamba (RGM) module, wherein each decoder scale incorporates global information via channel-wise gating (derived from the global context vector) and localized self-attention (Mamba blocks). Per-task decoding branches (depth, semantic segmentation, normals, edges) are constructed using lightweight convolutional adaptors operating at each scale, fused in a coarse-to-fine manner.

Cross-task interaction is enabled through the Cross-Task Mixer (CTM), which applies gated context fusion for each primary task. The fused representations are further refined by a Multi-Scale Convolutional Attention (MSCA) block, leveraging depthwise convolutions for computational efficiency and spatial attention for localization precision (Udugama et al., 31 Mar 2026, Udugama et al., 17 May 2026).

Task-specific heads include:

• Depth estimation: AdaBins-style “BinDepthHead,” which predicts per-image bin widths, per-pixel bin probabilities, and residual offsets, producing dense and precise depth predictions.
• Semantic segmentation: Lightweight convolutional heads produce per-pixel class logits.
• Auxiliary tasks: Optional heads for surface normal and edge estimation used exclusively during training for regularization.

2. Multi-Task Learning Strategy

M2H-MX optimizes for depth, semantics, and optional auxiliary cues (normals, edges) using a composite loss function:

• Depth: Scale-invariant logarithmic RMSE (SILog)
• Semantics: Categorical cross-entropy (CE)
• Normals: Cosine similarity
• Edges: Binary cross-entropy (BCE)

Cross-task consistency is enforced via penalties on the deviation between predicted surface normals and those inferred from depth, and between edge logits and semantic boundaries. Aggregation of per-task losses employs learned uncertainty weighting (Kendall et al.), balancing contributions adaptively during training (Udugama et al., 31 Mar 2026, Udugama et al., 17 May 2026).

Training leverages NYUDv2 for indoor scenes, Cityscapes for outdoor scenes, and ScanNet-25k for system-level evaluation. Only LoRA adapters and decoder parameters are updated, keeping the foundation backbone frozen to prevent overfitting and facilitate transfer across data regimes.

3. Perception-to-Mapping Integration

M2H-MX serves as the perception front-end in systems such as Mono-Hydra++. The model operates on single RGB images (optionally augmented with IMU for odometry), producing dense depth maps and per-pixel semantic logits. Outputs are transferred via zero-copy buffers to the mapping backend, where:

• Depth estimates inform point-to-plane residuals in (modified) visual-inertial odometry frontends
• Semantic masks drive surfel annotation and scene graph node labeling

The perception-mapping interface adheres to standard RGB-D frame conventions, enabling drop-in integration with established SLAM pipelines without requiring backend changes. M2H-MX predictions are also utilized for dynamic object masking and as sparse depth constraints in pose graph optimization (Udugama et al., 17 May 2026).

4. Quantitative Performance and Ablation

M2H-MX establishes new benchmarks in both dense prediction tasks and overall spatial mapping:

Dataset	Model	Semseg mIoU (↑)	Depth RMSE (↓)	Disp RMSE (↓)	Avg ATE (cm, ↓)
NYUDv2	M2H-MX-L	65.60	0.380	—	—
Cityscapes	M2H-MX-L	82.28	—	3.89	—
ScanNet (SLAM)	M2H-MX+Mono-H	—	—	—	6.91
7-Scenes (SLAM)	M2H-MX+Mono-H	—	—	—	— (29.8% ↓ ATE)

In NYUDv2, M2H-MX-L improves semantic mIoU by 6.6% and depth RMSE by 9.4% relative to representative multi-task baselines such as MTMamba++. On ScanNet, when deployed within Mono-Hydra++, the system achieves average ATE of 6.91 cm—nearly matching RGB-D Go-SLAM performance. In challenging real-time dynamic scenes (uHumans2), M2H-MX-based predictions reduce ATE by 25% and improve object-node F1 scores by 12% in complex environments (Udugama et al., 17 May 2026).

5. Real-Time Inference, Resource Profile, and Deployment

Inference efficiency is a central attribute of M2H-MX. On an NVIDIA RTX 4080 Super, the ViT-L model at 640×480 resolution achieves 25–30 Hz in the perception–mapping loop. On Jetson Orin NX 16GB using ONNX/TensorRT FP16, M2H-MX-L reaches 25.53 FPS at 256×192 input, with competitive GPU memory usage (200–400 MB for model weights, depending on backbone size).

The overall end-to-end SLAM loop incorporating M2H-MX sustains 15–20 Hz real-time throughput on standard workstation hardware, with perception dominating GPU load (≈45 ms/frame, 6 GB VRAM, 40% SM utilization) and low CPU requirements for odometry/mapping (Udugama et al., 31 Mar 2026).

6. Comparison to Prior Approaches and Limitations

M2H-MX supersedes prior state-of-the-art multi-task perception models (e.g., MTMamba++, SwinMTL, InvPT, MTI-Net) in key benchmarks. Ablation studies demonstrate that the register-gated Mamba decoder, cross-task fusion, and uncertainty-weighted loss all contribute meaningfully to performance gains. Furthermore, system-level improvements (e.g., trajectory error in SLAM) reveal the utility of tightly integrated, multi-task perception for spatial robotics.

Current limitations include reliance on only depth and semantic outputs for mapping (with other cues present but not yet used online), absence of learned temporal smoothing, and open questions on generalization to unconventional camera rigs. Proposed directions involve expanding the task set (flow, detection), cross-dataset adaptation without backbone finetuning, and deeper coupling with learned SLAM backends (e.g., NeRF (Udugama et al., 31 Mar 2026, Udugama et al., 17 May 2026)).

7. Impact and Research Significance

M2H-MX demonstrates that the combination of a large frozen transformer backbone, lightweight register-gated decoding, and principled cross-task design enables multi-task dense prediction at a pace and accuracy suitable for real-world robotics. Its pragmatic engineering—enforcing compatibility with monocular SLAM systems and supporting embedded deployment—underscores the growing feasibility of deploying high-capacity vision models beyond data centers, towards agile autonomous platforms operating in unstructured environments (Udugama et al., 31 Mar 2026, Udugama et al., 17 May 2026).