M2H-MX: Real-Time Dense Spatial Perception

• M2H-MX is a real-time multi-task dense visual perception model that infers both metric depth and semantic structure from a single RGB image stream to enhance robotic mapping.
• It uses an encoder-decoder design featuring a frozen DINOv3 backbone adapted via LoRA, multi-scale feature fusion, and a Register-Gated Mamba decoder for efficient dense prediction.
• The model interfaces directly with SLAM and 3D reconstruction pipelines, yielding significant gains in mapping accuracy and stability under real-time latency constraints.

Searching arXiv for papers on M2H-MX and closely related Mono-Hydra++ context. M2H-MX is a real-time multi-task dense visual perception model for monocular spatial understanding. It is designed to infer metric depth and semantic structure from RGB imagery under latency constraints compatible with robotic mapping, while preserving an interface simple enough to connect directly to monocular SLAM, volumetric fusion, and 3D scene-graph construction pipelines. In the robotics literature, the term most prominently denotes the 2026 model introduced for monocular spatial perception and later used as the dense-prediction backbone in Mono-Hydra++ (Udugama et al., 31 Mar 2026).

1. Problem setting and system role

M2H-MX addresses a specific limitation of conventional monocular robotic perception: metric maps support localization and collision avoidance, but do not provide object-level, room-level, and relational structure. The associated systems literature frames this gap in terms of search, inspection, exploration, and human robot interaction, where a robot must reason not only about geometry but also about objects, rooms, places, and spatial relations. Within that setting, M2H-MX serves as the unified multi-task dense-prediction backbone that produces frame-wise depth and semantic estimates used by downstream visual-inertial odometry, temporal fusion, volumetric mapping, and hierarchical 3D scene-graph construction (Udugama et al., 17 May 2026).

The standalone M2H-MX paper presents the model as a response to the difficulty of achieving reliable real-time spatial understanding from a single image stream. Its central claim is not merely that multi-task dense prediction can improve per-pixel depth and semantic estimation, but that these predictions can be translated into stable monocular mapping behavior when the perception front-end is engineered around a compact perception-to-mapping interface and a latency-aware decoder design (Udugama et al., 31 Mar 2026).

This role places M2H-MX between dense prediction and embodied mapping. It is neither a generic semantic segmentation network nor a complete SLAM system. Rather, it is the component that transforms monocular RGB input into metric-semantic fields sufficiently stable to be consumed by odometry and mapping modules.

2. Architectural formulation

M2H-MX follows an encoder-decoder design with three recurring motifs: a frozen foundation backbone with LoRA adaptation, explicit preservation of multi-scale features, and controlled cross-task interaction. The backbone is a frozen DINOv3 Vision Transformer, reported in ViT-B and ViT-L variants, pre-trained with self-supervision. Adaptation is performed through LoRA modules injected into the later transformer blocks; the standalone formulation writes the adapted linear map as

$W' = W + \alpha\,B\,A,\quad \alpha=32,\;r=16,$

while the Mono-Hydra++ instantiation reports LoRA on blocks 13–24 and a training setting with rank $4$ (Udugama et al., 31 Mar 2026).

Multi-scale structure is recovered by the Hierarchical Feature Adapter. Selected token streams from the ViT are reassembled into a spatial pyramid $P2$–$P5$. In the standalone description, patch-token feature maps are projected to a common channel dimension and then fused top-down: $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$ A pooled register-token representation provides global context,

$$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$

which is injected into the decoder through channel gating (Udugama et al., 31 Mar 2026).

That decoder is the Register-Gated Mamba module. At each pyramid level, the current feature state is reshaped into a sequence, modulated by a gate derived from the global register vector, and refined by a Mamba block and FFN: $$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$ This mechanism is intended to supply long-range scene context without the cost of heavier all-to-all attention (Udugama et al., 31 Mar 2026).

Task specialization occurs in several stages. Task Adaptors are per-task $3\times 3$ conv $\rightarrow$ GN $\rightarrow$ GELU modules that generate task-specific features at each scale, followed by top-down fusion. The Cross-Task Mixer then selectively injects complementary cues from related tasks by gated concatenation and $4$0 convolutional mixing, instead of full symmetric attention. The resulting features are refined by Multi-Scale Convolutional Attention, implemented through multi-branch depthwise convolutions and a learned spatial attention gate. The output heads are lightweight: a bin-based metric depth head and a convolutional semantic head (Udugama et al., 17 May 2026).

The depth head is central to the model’s dense metric prediction. In the Mono-Hydra++ description, the refined depth feature $4$1 is used to predict adaptive bin widths,

$4$2

bin intervals and centers,

$4$3

and a per-pixel bin classifier

$4$4

The coarse depth is then

$4$5

followed by a residual offset and clamping,

$4$6

The semantic head is a two-layer convolutional predictor,

$4$7

This combination makes depth and semantics the principal active outputs in deployment (Udugama et al., 17 May 2026).

3. Objectives, supervision, and training protocol

M2H-MX is formulated as a multi-task model with head-specific supervision, auxiliary cross-task consistency terms, and learned uncertainty balancing. The Mono-Hydra++ system description writes the primary losses as

$4$8

with additional losses for normals and edges,

$4$9

Auxiliary consistency penalties couple tasks: $P2$0 and the overall objective uses learned uncertainty weights,

$P2$1

The standalone M2H-MX paper presents a closely related but not numerically identical formulation, writing depth supervision as a per-pixel $P2$2 loss with reported metric $P2$3, while retaining cross-entropy semantics, normals and edges, consistency losses, and Kendall-style uncertainty weighting. This suggests that the precise loss realization differs across paper versions or deployment settings, while the multi-task coupling principle remains unchanged (Udugama et al., 17 May 2026).

The reported training protocol in Mono-Hydra++ uses NYUDv2 train split $P2$4 images) for indoor depth and semantics, Cityscapes train split $P2$5 images) for outdoor disparity and semantics, and ScanNet25k for system-level fine-tuning. Optimization uses AdamW with learning rate $P2$6, weight decay $P2$7, batch size $P2$8, and a $P2$9–$P5$0-epoch schedule with a learning-rate step of $P5$1 at epoch $P5$2. The same description reports LoRA rank $P5$3 and $P5$4 depth bins. By contrast, the standalone paper lists recommended hyperparameters for M2H-MX-L as decoder width $P5$5, Mamba state size $P5$6, register tokens $P5$7, depth bins $P5$8, and LoRA on blocks $P5$9–$$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$0 with rank $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$1, $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$2, and dropout $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$3. The coexistence of these settings indicates that the architecture is parameterized differently across experimental contexts (Udugama et al., 31 Mar 2026).

Input preprocessing is minimal. RGB images are normalized by ImageNet mean and standard deviation, IMU is not used in perception, depth is clamped to $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$4, and semantic logits are consumed through argmax or softmax depending on downstream use (Udugama et al., 17 May 2026).

4. Perception-to-mapping interface

A defining characteristic of M2H-MX is that its outputs are exposed through a deliberately compact interface. In the standalone formulation, perception consumes $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$5 and returns a metric depth map $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$6 and semantic logits $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$7. The depth map is then back-projected pixelwise,

$$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$8

so that the SLAM front end can treat $$\bar p_4 = \phi_4(\widetilde F^{\ell_4}),\quad \bar p_s = \phi_s\bigl(\widetilde F^{\ell_s} + \mathrm{Up}(\bar p_{s+1})\bigr)\;(s=3,2,1).$$9 as if they originated from an RGB-D sensor. Semantic fusion uses $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$0, and the downstream mapping, pose-graph optimization, and scene-graph pipelines are described as remaining exactly as in the original Mono-Hydra (Udugama et al., 31 Mar 2026).

Mono-Hydra++ makes this interface more explicit and more tightly integrated. At the VIO front end, sparse predicted depth is sampled at SuperPoint keypoints $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$1 to form depth-factor residuals,

$$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$2

with gating by motion and reprojection thresholds. Semantic masks robustify VIO by zero-weighting keypoints in dynamic classes, for example when $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$3. Before volumetric fusion, the last $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$4 frames of depth and semantics are pose-warped into the current view, gated by depth consistency and static classes, and fused by averaging depth and majority-vote semantics. The temporally stabilized outputs then drive TSDF-based volumetric fusion and 3D scene-graph node extraction for rooms, objects, and places via the Hydra graph pipeline (Udugama et al., 17 May 2026).

The practical significance of this interface is that M2H-MX is not described as an isolated perception benchmark model. It is specified as an operational module in a metric-semantic mapping stack, and its evaluation therefore includes both per-pixel prediction quality and in-the-loop odometric and mapping behavior.

5. Quantitative performance and runtime

The reported empirical profile of M2H-MX combines dense-prediction benchmarks, closed-loop mapping ablations, and runtime measurements. On NYUDv2 at $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$5 input, the standalone paper reports that M2H-MX-L with only depth and semantics active has $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$6 M parameters and $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$7 GFLOPs, compared with $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$8 M parameters and $$r = W_r\Bigl(\tfrac1R\sum_{j=1}^R h^{\ell_5}_{\mathrm{reg},j}\Bigr)\in\mathbb{R}^C,$$9 GFLOPs for the earlier M2H and $$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$0 M and $$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$1 GFLOPs for MTMamba++ (Udugama et al., 31 Mar 2026).

Context	Metric	Reported result
NYUDv2	Semseg mIoU / depth RMSE	$$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$2; $$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$3
Cityscapes	Semseg mIoU / disparity RMSE	$$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$4; $$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$5
ScanNet SLAM	Average ATE	$$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$6
Go-SLAM ScanNet evaluation subset	Average trajectory error	$$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$7 lower than the strongest RGB-D baseline
Calibrated 7-Scenes	Average ATE	$$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$8 improvement over the strongest competing calibrated baseline
Jetson Orin NX 16GB	Embedded inference	ONNX/TensorRT FP16 M2H-MX-L at $$g_k = \sigma\bigl(\mathcal{A}_k(r)\bigr)\in[0,1]^C,\qquad \bar q_k = q_k \odot g_k.$$9 FPS

Within the standalone M2H-MX paper, the ScanNet in-the-loop result is emphasized as a system-level gain: Go-SLAM as a monocular baseline reports $3\times 3$0 average absolute trajectory error, whereas “M2H-MX front-end + Mono-Hydra” reports $3\times 3$1, corresponding to a $3\times 3$2 reduction. The same paper reports an asynchronous execution profile with M2H-MX-L inference at approximately $3\times 3$3, odometry at approximately $3\times 3$4, mapping backend at approximately $3\times 3$5, and end-to-end stable $3\times 3$6–$3\times 3$7 Hz (Udugama et al., 31 Mar 2026).

Mono-Hydra++ extends these performance claims into a more complete monocular RGB+IMU mapping system. On its ScanNet ablation, replacing MTMamba++ with M2H-MX raises global mesh mIoU from $3\times 3$8 to $3\times 3$9, Radius F1@$\rightarrow$0 from $\rightarrow$1 to $\rightarrow$2, and Box F1@$\rightarrow$3 from $\rightarrow$4 to $\rightarrow$5. The same system paper reports $\rightarrow$6 lower average trajectory error than the strongest RGB-D baseline in its comparison on the Go-SLAM ScanNet evaluation subset, $\rightarrow$7 improvement in average ATE on calibrated 7-Scenes over the strongest competing calibrated baseline, and embedded deployment at $\rightarrow$8 FPS on a Jetson Orin NX $\rightarrow$9GB using an ONNX/TensorRT FP16 M2H-MX-L model (Udugama et al., 17 May 2026).

The benchmark tables reported across the standalone and system papers are not numerically identical, particularly in their NYUDv2 and Cityscapes comparison sets. This suggests that the results should be interpreted in the exact experimental context of each paper rather than collapsed into a single leaderboard narrative.

6. Interpretation, limitations, and nomenclature

The main technical interpretation offered by the M2H-MX literature is that real-time monocular spatial understanding benefits from preserving multi-scale transformer features, injecting global context through cheap register-token gates, and constraining cross-task interaction so that depth and semantics reinforce one another without incurring the cost of heavy attention. The architecture is presented as a latency-aware alternative to larger or denser multi-task decoders, and the system papers repeatedly stress that dense prediction quality matters primarily insofar as it improves odometry, mapping stability, and scene-graph quality (Udugama et al., 31 Mar 2026).

At the same time, the model is not depicted as universally robust. The standalone paper states that failure modes tend to occur on glossy or heavily occluded regions where depth bins have low confidence or semantics are ambiguous. This is consistent with the broader monocular-depth problem setting: ambiguities in reflectance, visibility, and scale remain difficult even when multi-task coupling is strong (Udugama et al., 31 Mar 2026).

The name “M2H-MX” also has a second usage in the arXiv record that is unrelated to robotic perception. A 2024 hardware paper uses “M2H-MX” to denote an FP32-to-microscaling conversion process for reduced-precision arithmetic. In that context, the term refers to a memory-free hardware model that converts $\rightarrow$0 single-precision floating-point values into one of six MX-format types—E5M2, E4M3, E3M2, E2M3, E2M1, and INT8—through three purely combinational stages: max-finder, scale-generator, and quantizer (Gorodecky et al., 2024). Because the same string therefore appears in both robotic perception and MX-format hardware conversion, citations require contextual disambiguation.

In current robotic mapping usage, however, M2H-MX most commonly denotes the monocular dense-perception backbone built around a frozen DINOv3 encoder, LoRA adaptation, Register-Gated Mamba decoding, controlled cross-task mixing, and lightweight depth and semantic heads. Its significance lies less in any single per-pixel metric than in demonstrating that modern multi-task dense prediction can be coupled to real-time monocular mapping and scene-graph construction under resource-constrained deployment conditions (Udugama et al., 17 May 2026).