Mono-Hydra++: Real-Time 3D Scene Graph

• The paper introduces Mono-Hydra++, a system that enables real-time 3D scene graph construction from monocular RGB+IMU inputs using multi-task deep learning.
• It leverages robust visual-inertial odometry, dynamic object rejection, and pose-aware volumetric fusion to achieve dense metric-semantic mapping.
• Extensive evaluations demonstrate enhanced accuracy and efficiency over RGB-D baselines, supporting deployment on resource-constrained robotic platforms.

Mono-Hydra++ is a real-time monocular RGB+IMU 3D scene graph construction pipeline that leverages multi-task deep learning, tightly integrated visual-inertial odometry, and pose-aware volumetric fusion to enable dense metric-semantic mapping in resource-constrained robotic platforms. Mono-Hydra++ advances the original Mono-Hydra framework by introducing a DINOv3-based multi-task perception model (M2H-MX), robust pose estimation with dynamic object rejection, temporal aggregation for semantic/depth stability, and a hierarchical 3D scene graph, all without reliance on active depth sensors (Udugama et al., 2023, Udugama et al., 17 May 2026).

1. System Overview and Architecture

Mono-Hydra++ comprises three primary modules, each optimized for real-time parallel execution:

1. Front-End Visual-Inertial Odometry (VIO): Implements a robocentric, square-root information filter using SuperPoint features, IMU preintegrations, and depth constraints from the perception network. Keyframes, odometry, and sparse depths are fused in a pose graph.
2. GPU-Based Dense Perception and Pose-Aware Alignment: Utilizes the multi-task model M2H-MX, which infers dense metric depth and per-pixel semantics at high throughput. VIO poses temporally align network outputs across frames, reducing semantic/depth flicker and improving fusion quality.
3. CPU-Based Volumetric Fusion and Hierarchical Scene Graph Construction: Integrates stabilized depth/semantic maps into a Truncated Signed Distance Function (TSDF) volume via the Mono-Hydra/Hydra backend. Marching cubes produce a colored mesh, from which semantic and topological clusters yield a layered 3D scene graph organizing buildings, rooms, places, and objects (Udugama et al., 2023, Udugama et al., 17 May 2026).

The architecture prioritizes data flow from the camera/IMU through concurrent feature extraction, dense perception, odometry, temporally-stabilized fusion, and hierarchical graph construction. This supports modular upgrades and embedded deployment (e.g., on Jetson Orin NX) (Udugama et al., 17 May 2026).

2. Multi-Task Perception Model (M2H-MX)

M2H-MX is the multi-task backbone responsible for joint depth and semantic prediction, enabling shared spatiotemporal features for both tasks:

• Backbone:

Frozen DINOv3 Vision Transformers (ViT-B/ViT-L) with LoRA adapters in layers 13–24 for adaptable finetuning.

• Hierarchical Feature Aggregation (HFA):

Converts transformer tokens to a four-scale pyramid $\{p_2, ..., p_5\}$ using token reassembly and top-down fusion.

• Register-Gated Mamba (RGM) Blocks:

Inject global scene context (register tokens) into local spatial sequences for each scale.

• Depth Head:

AdaBins-style adaptive binning $N_b$, with softmax weighting, coarse bins $c_i$, and per-pixel probabilities to estimate depth:

$$D_c(\mathbf{x}) = \sum_{i=1}^{N_b}p_{b,i}(\mathbf{x})\,c_i; \qquad \hat D(\mathbf{x}) = \mathrm{clamp}\left(D_c(\mathbf{x}) + W_o(\tilde h^d(\mathbf{x})),\,d_{\min},\,d_{\max}\right)$$

• Semantic Head:

Lightweight 3×3 → GELU → 1×1 conv for semantic logits $\hat S$.

• Loss Function:

Multi-task with learned-uncertainty weighting:

$$L_{\mathrm{total}} = \sum_{a \in \{\mathrm{depth},\,\mathrm{sem},\,\dots\}}\left(\tfrac{1}{2\sigma_a^2}L_a + \log\sigma_a\right)$$

Tasks: $\mathrm{SILog}$ for depth, cross-entropy for semantics, auxiliary normal/edge, cross-task consistency.

Training:

Supervised on NYUDv2, Cityscapes, and ScanNet-25k, with random flip, color jitter, and resize/crop augmentations (Udugama et al., 17 May 2026). The result is a unified high-throughput network validated on RTX 4080 Super and Jetson Orin (30 Hz and 25.53 FPS, respectively, at real-time resolutions).

3. Visual-Inertial Odometry and Robustification

The VIO front end provides robust, metric 6-DoF motion estimates by integrating IMU and visual measurements:

• State Variable:

$$[\mathbf{R}_t, \mathbf{p}_t, \mathbf{v}_t, \mathbf{b}_{\omega,t}, \mathbf{b}_{a,t}]$$ in robocentric frame.

• Preintegration:

Computes $\Delta R, \Delta v, \Delta p$ between frames.

• SuperPoint Features:

Keypoints and descriptors extracted from $I_t$; dynamic points masked by semantic output $N_b$0.

• Sparse Depth Constraints:

Depth factors sampled from M2H-MX output $N_b$1, weighted by semantic class and measurement quality.

• Square-Root Information Form:

Updates via linear system $N_b$2 and QR factorization.

• Pose Graph Integration:

Each new keyframe and loop closure produces relative pose and depth constraints for incremental backend smoothing.

Semantic Robustification:

Dynamic classes (e.g., persons) are removed from both visual and depth measurements via $N_b$3 This increases resilience in dynamic environments and suppresses mesh/scene corruption (Udugama et al., 17 May 2026).

4. Pose-Aware Temporal Alignment and Volumetric Fusion

Frame-level predictions are subject to temporal flicker, addressed by pose-aligned temporal fusion:

• Temporal Warping:

For previous $N_b$4 frames, predictions are projected into the newest keyframe’s coordinate system using VIO transformations:

$N_b$5, then $N_b$6

• Consistency Gating:

Only points satisfying $N_b$7 and non-dynamic $N_b$8 are fused.

• Aggregation:

$N_b$9

$c_i$0

These stabilized, pose-aligned maps drive a TSDF-based volumetric integrator (voxel size $c_i$15 cm, truncation $c_i$2), producing a manifold mesh and per-voxel semantic labels (Udugama et al., 17 May 2026).

5. Hierarchical 3D Scene Graph Construction

Volumetric maps are incrementally partitioned into semantic and topological entities:

• Mesh Clustering:

Segments the mesh by semantic class and spatial adjacency.

• Node Hierarchy:
  • Building (root)
  • Rooms (partitioned by continuity)
  • Objects (clusters within rooms)
• Edge Types:
  • Containment (object→room, room→building)
  • Adjacency (shared boundaries between rooms)
  • Support (vertical/stable relationships, e.g., lamp on table)
• Graph Optimization:

Node poses $c_i$3 are refined by minimizing reprojection and adjacency terms via incremental smoothing. A plausible implication is that learned extensions could replace plane+cluster rules with graph neural networks or differentiable SLAM layers (Udugama et al., 2023).

6. Experimental Results and Benchmarks

Mono-Hydra++ has been evaluated on synthetic, real, and embedded testbeds:

Environment	Metric	Mono-Hydra++ (RGB+IMU)	Strongest Baseline	Relative Improvement
ScanNet (Go-SLAM subset)	Average ATE	6.91 cm	7.02 cm (Go-SLAM, RGB-D)	1.6% better
7-Scenes (calibrated)	ATE (RMSE)	0.033 m	0.047 m (MASt3R-SLAM)	29.8% better
ScanNet-v2	Global mIoU	44.96%	—	+3–4% vs earlier
Embedded (Jetson Orin NX 16GB)	M2H-MX-L Perception FPS	25.53	(see (Udugama et al., 17 May 2026))	Real-time capability

Further, Mono-Hydra++ reconstructs 200 m real-world loops with mean mesh error $c_i$47–8 cm, achieves object recall +6% higher than two-stream baselines, and semantic masking improves Object-Node F1 by up to 12% (Udugama et al., 17 May 2026).

7. Limitations and Prospects

• Mono-Hydra++ approaches RGB-D scene graph performance, but still leverages monocular cues; large open spaces and non-planar geometries can be more challenging.
• Remaining bottlenecks include semantic segmentation throughput and CPU-based backend multithread contention.
• Scene graph accuracy can benefit from learned, rather than hand-crafted, clustering and relation modules; differentiable optimization, and additional sensory modalities (e.g., stereo, event cameras) can further enhance robustness and fidelity (Udugama et al., 2023, Udugama et al., 17 May 2026).

The system architecture and modularity suggest extensibility for outdoor domains, mobile robotics, and embedded platforms sensitive to power and compute constraints. Ongoing innovations in transformer-based perception, adaptive multi-task learning, robust SLAM, and learned scene understanding are expected to further advance this research direction.