OmniMap: Unified Online Robotic Mapping

Updated 10 July 2026

OmniMap is an online robotic mapping framework that integrates optical appearance, geometric layout, and semantic understanding into a unified representation.
It employs a hybrid 3DGS-Voxel architecture combining TSDF-Fusion and Gaussian splatting to achieve high-quality rendering and precise geometric reconstruction.
The framework demonstrates state-of-the-art performance in rendering, geometry, and semantics while supporting diverse tasks like scene Q&A, interactive editing, and map-based navigation.

OmniMap is an online robotic mapping framework that jointly models three scene attributes that are often separated in prior systems: photo-realistic appearance, geometric structure, and open-vocabulary semantics. Introduced as “a general mapping framework integrating optics, geometry, and semantics,” it combines a tightly coupled 3DGS-Voxel representation with adaptive camera modeling, hybrid incremental reconstruction, and probabilistic instance-level semantic fusion. The framework is positioned as the first online mapping system that simultaneously captures optical, geometric, and semantic scene attributes while maintaining real-time performance and model compactness, and it is evaluated through rendering, reconstruction, and zero-shot semantic segmentation as well as downstream embodied tasks such as scene Q&A, interactive editing, perception-guided manipulation, and map-assisted navigation (Deng et al., 9 Sep 2025).

1. Definition and problem setting

OmniMap addresses a recurrent systems problem in robotic perception: high-fidelity rendering, stable geometry, and semantically meaningful scene understanding are usually optimized by different representations and different learning objectives. The paper frames this as a deficiency of existing mapping methods, which typically satisfy only part of the requirement set and exhibit optical blurring, geometric irregularities, or semantic ambiguities. OmniMap therefore defines mapping not as pure reconstruction, nor as pure semantics, but as a unified online estimation problem over appearance, shape, and language-grounded scene content (Deng et al., 9 Sep 2025).

Within this formulation, “optics” refers to photo-realistic appearance and view synthesis; “geometry” refers to accurate layout shape and mesh reconstruction; and “semantics” refers to open-vocabulary, instance-level scene understanding. The framework’s stated goal is not merely to maintain separate outputs for these modalities, but to make them mutually reinforcing. Voxels provide regularized spatial structure and a substrate for probabilistic fusion; 3D Gaussian Splatting provides high-quality rendering and explicit geometric detail; and language embeddings provide instance retrieval and open-set recognition. This coupling is central to the framework’s claim of generality (Deng et al., 9 Sep 2025).

A common misconception is to treat OmniMap as only a rendering system augmented with labels. The architecture and experiments argue against that reading. The paper evaluates rendering fidelity, geometric accuracy, and zero-shot semantic segmentation as distinct criteria, and its downstream tasks rely on the coexistence of all three modalities rather than on any single one. This suggests that OmniMap is best understood as a multimodal world representation rather than as a narrow NeRF- or 3DGS-style renderer.

2. Hybrid 3DGS-Voxel architecture

The core of OmniMap is a tightly coupled hybrid representation composed of a voxel map, a global embedding codebook, and a 3D Gaussian field. The voxel map $M_t$ is incrementally built from depth and color images using TSDF-Fusion at configurable resolution, with $0.03\,\text{m}$ given as an example. Each voxel stores a probability vector over instance identities. The global codebook $B_t$ maps instance IDs to fused textual and visual embeddings. The 3DGS field $G_t$ stores Gaussian splats parameterized by position, orientation, size, color, and opacity (Deng et al., 9 Sep 2025).

The paper organizes the system into three modules:

Module	Role	Details
2D Language Embeddings Extractor	Per-frame semantic front-end	YOLO-World, TAP model, SBERT
Probabilistic Voxel Reconstructor	3D semantic fusion	Voxel probability distribution and global instance-level embedding codebook
Motion-Robust 3DGS Incremental Reconstructor	Optical and geometric back-end	New Gaussian initialization, normal constraints, camera optimization

The interaction between voxel allocation and Gaussian allocation is explicit. New Gaussian primitives are created only for newly allocated voxels,

$\mathcal{V}_{t}^{\text{new}} = \left\{ v^j \mid \alpha_{t-1}^j = 0 \ \text{and}\ \alpha_t^j > 0 \right\},$

where $\alpha_t^j$ denotes the instance count in voxel $v^j$ . This incremental rule is important for both compactness and stability: unlike batch-style 3DGS initialization, OmniMap grows the Gaussian field only where the voxel reconstruction indicates genuinely new support (Deng et al., 9 Sep 2025).

This architecture uses the different strengths of the constituent representations in a deliberately asymmetric way. Voxels serve as the regularized backbone for semantics and spatial bookkeeping, while Gaussians carry the burden of efficient high-quality rendering and mesh extraction. The paper argues that this division avoids the fragility of purely Gaussian systems for semantic accumulation and the optical limitations of purely voxel-based systems.

3. Motion-robust optical and geometric reconstruction

OmniMap introduces an adaptive camera model to compensate for motion blur and exposure variation during online reconstruction. Each frame is modeled as a combination of a sharp image and a blurred or shifted component:

$C_t \approx \omega_{\text{trans}}\, T_{x_{\text{trans}}, y_{\text{trans}}}(\bar{C}_t) + \omega_{\text{raw}}\, \bar{C}_t,$

where $\omega_{\text{trans}}$ and $\omega_{\text{raw}}$ are blend weights, and $0.03\,\text{m}$ 0 model 2D shift induced by camera movement during exposure. These four parameters are differentiably optimized together with scene parameters. The stated purpose is robustness against sensor effects that otherwise degrade both rendering fidelity and geometric consistency (Deng et al., 9 Sep 2025).

Geometric reconstruction is handled by a hybrid incremental strategy. TSDF-Fusion allocates voxels, and Gaussian splats are initialized only at newly allocated voxel centers. Keyframe selection provides multi-view constraints, and optimization uses RGB loss, depth loss, and a normal regularizer,

$0.03\,\text{m}$ 1

where $0.03\,\text{m}$ 2 are normals derived from rendered depth and $0.03\,\text{m}$ 3 are normals from ground-truth depth. The normal term is reported to improve both smoothness and detail beyond depth-only supervision (Deng et al., 9 Sep 2025).

This design is notable because it merges two reconstruction logics that are often separated in practice. TSDF allocation contributes structural stability and explicit occupancy support, while 3DGS contributes view-dependent appearance and fine-scale reconstruction. A plausible implication is that OmniMap is attempting to preserve the update discipline of classical volumetric fusion while adopting the rendering efficiency of modern Gaussian splatting.

4. Probabilistic semantic fusion and open-vocabulary understanding

Semantic mapping in OmniMap is instance-centric and probabilistic. The 2D front-end uses a sequential multi-model pipeline for open-vocabulary object detection, fine segmentation with descriptive captions, and language embedding. The paper names YOLO-World for detection, the TAP model for segmentation and captions, and SBERT for language embeddings. These 2D predictions are then fused into 3D rather than being treated as frame-local annotations (Deng et al., 9 Sep 2025).

Each voxel maintains a probability vector over instance labels, updated through a Bayesian rule described as a Dirichlet-Categorical conjugate update:

$0.03\,\text{m}$ 4

Here $0.03\,\text{m}$ 5 is $0.03\,\text{m}$ 6 when the current observation assigns voxel $0.03\,\text{m}$ 7 to instance $0.03\,\text{m}$ 8. The expected probability $0.03\,\text{m}$ 9 becomes the semantic state stored for that voxel (Deng et al., 9 Sep 2025).

Instance association relies on both geometric overlap and embedding similarity to a global instance-level codebook. The paper states that this improves robustness to over-segmentation, under-segmentation, and fragmented segmentation across frames. Codebook updates use weighted averaging, with weights defined by the product of instance association probability and visibility ratio. In effect, the semantic system is designed not only to recognize categories, but to maintain persistent object identities under noisy framewise masks (Deng et al., 9 Sep 2025).

This semantic design also explains the framework’s downstream flexibility. Because the codebook stores fused textual and visual embeddings, arbitrary language queries can retrieve object instances, and the same instance-aware structure can support editing, retrieval, and manipulation. The semantic map is therefore not merely descriptive metadata attached to geometry; it is an operational interface for embodied interaction.

5. Empirical performance and downstream uses

OmniMap is evaluated on Replica and ScanNet for rendering, on ScanNet for geometry, and on Replica and ScanNet for zero-shot semantic segmentation. The reported results place the method at or above the best baselines in the supplied comparisons, while maintaining compact models and online throughput (Deng et al., 9 Sep 2025).

Dimension	Dataset / metric	OmniMap result
Rendering	Replica: PSNR / SSIM / LPIPS	39.48 / 0.982 / 0.021
Rendering	ScanNet: PSNR	26.04
Geometry	ScanNet: Acc. / Comp. / Comp. Rate / F-score	9.70 / 2.47 / 91.24 / 79.10
Semantics	Replica: mIoU / fIoU	29.06 / 64.42
Semantics	ScanNet: mIoU / fIoU	25.42 / 50.86
Efficiency	Frame rate	5.43 FPS on Replica; 5.55 FPS on ScanNet
Compactness	Model size	14.2 MB on Replica; 40.5 MB on ScanNet

For rendering, the paper reports on Replica that OmniMap reaches PSNR $B_t$ 0, SSIM $B_t$ 1, and LPIPS $B_t$ 2, exceeding RTG-SLAM, Gaussian-SLAM, MonoGS, and GSFusion in the provided comparison. On ScanNet it reports the best PSNR and SSIM, with PSNR $B_t$ 3 versus $B_t$ 4 for HI-SLAM2 in the cited example. For geometry, the ScanNet table gives Accuracy $B_t$ 5, Completeness $B_t$ 6, Completion Rate $B_t$ 7, and F-score $B_t$ 8. For zero-shot semantics, OmniMap reports $B_t$ 9 mIoU/fIoU on Replica and $G_t$ 0 on ScanNet, exceeding OpenGaussian, ConceptFusion, ConceptGraph, and, on ScanNet, HOV-SG in the tabulated averages (Deng et al., 9 Sep 2025).

The downstream applications are unusually broad for a mapping paper. The representation supports multi-domain scene Q&A through object-centric retrieval and multimodal LLM integration; interactive editing such as coloring, deletion, duplication, and movement at the instance level; perception-guided manipulation with a Franka robot arm using end-effector RGB-D reconstruction and LLM-generated task logic; and map-assisted navigation for wheeled and quadruped robots in cluttered or reflective environments. These applications are not presented as separate systems detached from the map; rather, they are demonstrations that the same unified representation can serve rendering, semantics, and control pipelines without re-encoding the scene (Deng et al., 9 Sep 2025).

6. Relation to adjacent mapping paradigms

OmniMap sits within a larger research trend toward more unified scene representations, but related work distributes that ambition across distinct technical subproblems rather than a single standardized formulation. Hybrid navigation-oriented SD–HD association is studied in Online Map Association (OMA), whose OMA benchmark contains $G_t$ 1 roads and $G_t$ 2 lane paths and whose Map Association Transformer is designed to align lane-level HD centerlines with road-level SD maps for autonomous-vehicle planning (Wan et al., 10 Jul 2025). Multi-map consistency is treated in systems such as ORBSLAM-Atlas, which introduces an atlas of disconnected sub-maps with place-recognition-based merging (Elvira et al., 2019), and Uni-Mapper, which targets dynamic-aware multi-modal LiDAR map merging with dynamic object removal, DynaSTD loop closure, and centralized anchor-node optimization (Kang et al., 28 Jul 2025).

Other papers address complementary representational layers. osmAG defines an OSM-XML-based hierarchical semantic topometric area graph format for indoor and outdoor robotics, with area and passage abstractions, capability-aware planning, and ROS integration (Feng et al., 2023). OmniVLN constructs a five-layer Dynamic Scene Graph from mesh geometry to building-level structure and then uses an agent-centric 3D octant representation for token-efficient LLM reasoning across aerial and ground robots (Liu et al., 18 Mar 2026). These works emphasize that “unified mapping” may mean graph structure, navigation abstraction, or interoperability rather than joint rendering-semantics fusion.

A separate line emphasizes omnidirectional sensing and immersive reconstruction. OmniLayout uses spherical convolutions for room layout recovery from indoor panoramas (Rao et al., 2021); OmniSyn synthesizes intermediate 360° views between wide-baseline panoramas using spherical cost volumes, mesh rendering, and fusion networks (Li et al., 2022); and OmniSLAM combines omnidirectional depth estimation, loop closing, and TSDF fusion for wide-baseline multi-camera rigs (Won et al., 2020). Beyond 3D reconstruction, Maps from Motion generates 2D semantic maps from sparse uncalibrated multi-view images through graph-based alignment (Toso et al., 2024), OptiMVMap performs select-then-fuse multi-vehicle vectorized map construction via Optimal Vehicle Selection, Cross-Vehicle Attention, and a Semantic-aware Noise Filter (Dan et al., 18 Apr 2026), UniMapGen formulates large-scale map construction as multi-modal autoregressive generation over vector sequences (Yuan et al., 26 Sep 2025), and OmniNWM extends the ambition further into panoramic state-action-reward world modeling with occupancy-grounded rewards (Li et al., 21 Oct 2025).

This landscape suggests that “OmniMap” has two related meanings in current literature. In the strict sense, it denotes the specific 3DGS-Voxel online framework of Wang and collaborators (Deng et al., 9 Sep 2025). In a broader, aspirational sense, neighboring papers use the term to indicate a systems objective: comprehensive, multi-modal, semantically grounded, and operationally useful maps that can support navigation, question answering, manipulation, and simulation. The literature does not yet present a single agreed-upon OmniMap standard. Instead, it presents a family of partial realizations whose integration remains an active research direction.