OpenRoboVox: Real-Time Semantic Voxel Mapping

• OpenRoboVox is a real-time, open-vocabulary voxel mapping system that integrates 2D RGB-D instance segmentation with high-level language processing to build comprehensive 3D maps.
• It employs probabilistic voxel fusion using TSDF and Dirichlet-based instance labeling to achieve stable, low-latency mapping and accurate semantic segmentation.
• Its deployment in RoboAtlas demonstrates robust performance in large-scale environments, enabling active semantic exploration and navigation on mobile robotic platforms.

to=arxiv_search 天天中彩票未json variant perhaps? Let's attempt. to=arxiv_search code 大发快三大小单双ি {"query":"OpenRoboVox OpenVox RoboAtlas", "max_results": 10} OpenRoboVox is a real-time, instance-level, open-vocabulary voxel mapping system for robotics. It is described as an on-robot adaptation of OpenVox, a real-time incremental open-vocabulary probabilistic instance voxel representation, and it fuses 2D RGB-D instance observations and high-level language understanding into a unified 3D map designed end-to-end for stable, low-latency operation on mobile robotic platforms (Deng et al., 23 Feb 2025). In the RoboAtlas framework, OpenRoboVox serves as the real-time 3D semantic-mapping backbone: it maintains a globally consistent, metric-scale volumetric map via a TSDF, assigns and fuses open-vocabulary instance labels into a probabilistic voxel grid, and distills millions of voxels into a compact Scene-Dictionary for high-level reasoning (Schperberg et al., 24 Jun 2026).

1. Lineage, scope, and problem formulation

OpenRoboVox is explicitly motivated by the limitations identified in open-vocabulary mapping with vision-LLMs. OpenVox frames the underlying problem as obtaining rich semantic understanding and robust incremental reconstruction at the instance level while mitigating semantic ambiguity in point-wise feature maps (Deng et al., 23 Feb 2025). OpenRoboVox inherits that emphasis on instance-level reasoning, but reorients the system toward robotic deployment, where stable low-latency operation, sensor noise tolerance, and compatibility with navigation and planning become central requirements.

The system is presented as covering several coupled functions: front-end instance segmentation and caption encoding for language reasoning; back-end probabilistic instance voxel formulation and cross-frame fusion through instance association and live map evolution; vision-language integration enabling zero-shot recognition via LLM-encoded captions; real-time performance and hardware specifications; quantitative zero-shot evaluation; and real robot experiments (Deng et al., 23 Feb 2025). In the later RoboAtlas technical overview, the scope expands further: OpenRoboVox becomes the unified world model used by frontier-based exploration, global semantic-map reasoning, and egocentric VLM reasoning (Schperberg et al., 24 Jun 2026).

A plausible implication is that OpenRoboVox should be understood not merely as a semantic reconstruction module, but as a mapping substrate intended to connect low-level SLAM, probabilistic semantic fusion, and language-conditioned robotic behavior.

2. Architectural organization

One architectural description uses a streaming input tuple of RGB images $C_t$, depth maps $D_t$, and poses $P_t$. Its front-end "Instance Segmentation & Understanding" applies an open-vocabulary detector $\mathrm{Det}(\cdot)$ identified as Yolo-World, a mask-and-caption model $\mathrm{SegCap}(\cdot)$ identified as TAP, and an LLM-based encoder $\mathrm{Enc}(\cdot)$ identified as SBERT. The resulting outputs are 2D masks $m_t^i$ and 384-dimensional caption features $f_t^i$. The back-end "Probabilistic Voxel Reconstruction" performs instance association by MLE and live map evolution by MAP, producing probabilistic voxels $V_t=\{v_t^j\}$ that store instance-ID distributions $\theta_t^j$ and a codebook $D_t$0 mapping each instance $D_t$1 (Deng et al., 23 Feb 2025).

In that front-end pipeline, each incoming RGB frame is processed as

$D_t$2

The report states that all operations run at approximately $D_t$3--$D_t$4 Hz on a single RTX 4090, with approximately $D_t$5 Hz on offline I/O, and that passing captions through a lightweight LLM yields richer language reasoning than CLIP alone with only approximately $D_t$6 ms overhead per instance (Deng et al., 23 Feb 2025).

A second architectural description, introduced in the RoboAtlas setting, places OpenRoboVox between low-level SLAM and high-level expert modules and decomposes it into two asynchronous threads. The geometric thread performs tracking and TSDF integration at approximately $D_t$7 Hz from depth frames and robot poses from SLAM Toolbox, maintains a sparse, block-partitioned TSDF volume on the GPU, and extracts a 2D occupancy-grid projection for navigation and frontier detection. The semantic thread runs at approximately $D_t$8 Hz on RGB frames, depth frames, and the current TSDF; runs open-vocabulary detectors (YOLO-World) and promptable segmenters (TAP/SAM); fuses 2D masks into the 3D voxel grid via Bayesian updates over instance IDs; and updates a Scene-Dictionary in the background for new or modified instances only (Schperberg et al., 24 Jun 2026).

The published outputs of this asynchronous version are an OctoMap or 2D occupancy grid for Nav2 planning, a probabilistic 3D voxel grid with Dirichlet posteriors over instance IDs, a 2D Pillar Map of height-collapsed instance lists per grid cell, and a Scene-Dictionary $D_t$9 of extracted object primitives (Schperberg et al., 24 Jun 2026).

3. Probabilistic map representation and fusion mechanics

At the semantic level, OpenRoboVox represents the world as a sparse voxel grid $P_t$0 with resolution $P_t$1 cm, where each voxel $P_t$2 maintains a Dirichlet-parameterized distribution over instance IDs $P_t$3:

$P_t$4

A separate codebook $P_t$5 stores fused caption embeddings per instance (Deng et al., 23 Feb 2025).

Cross-frame fusion is factored into two subproblems:

$P_t$6

which are identified as instance association and live map evolution. Here the update target is the previous map $P_t$7 under a new observation $P_t$8 (Deng et al., 23 Feb 2025).

For instance association, each mask $P_t$9 is back-projected into voxel space:

$\mathrm{Det}(\cdot)$0

The system then solves for the existing instance $\mathrm{Det}(\cdot)$1 that maximizes the likelihood of the observation. With a uniform prior and voxel independence, geometric similarity is approximated by averaging existing posterior probabilities over the voxels in the projected mask,

$\mathrm{Det}(\cdot)$2

while feature similarity is computed via cosine similarity in the codebook,

$\mathrm{Det}(\cdot)$3

The two are fused, for example by a weighted sum,

$\mathrm{Det}(\cdot)$4

If $\mathrm{Det}(\cdot)$5 exceeds a threshold $\mathrm{Det}(\cdot)$6, the mask is associated to an existing instance; otherwise a new instance $\mathrm{Det}(\cdot)$7 is initialized with $\mathrm{Det}(\cdot)$8 (Deng et al., 23 Feb 2025).

For live map evolution, once the instance identity of each mask is known, voxel-label measurements $\mathrm{Det}(\cdot)$9 are used to update each voxel’s Dirichlet counts:

$\mathrm{SegCap}(\cdot)$0

The posterior expected instance-ID vector is then

$\mathrm{SegCap}(\cdot)$1

Codebook embeddings are fused by a weighted update using

$\mathrm{SegCap}(\cdot)$2

$\mathrm{SegCap}(\cdot)$3

$\mathrm{SegCap}(\cdot)$4

Because the Dirichlet-Categorical conjugacy admits closed-form updates, these local updates are reported to take approximately $\mathrm{SegCap}(\cdot)$5--$\mathrm{SegCap}(\cdot)$6 ms per frame (Deng et al., 23 Feb 2025).

In the RoboAtlas technical overview, this probabilistic representation is complemented by a TSDF volume composed of blocks of size $\mathrm{SegCap}(\cdot)$7 voxels with $\mathrm{SegCap}(\cdot)$8, stored in a hash table and fused by weighted average with signed distance truncated to $\mathrm{SegCap}(\cdot)$9. The Scene-Dictionary stores, for each instance, the instance identifier, caption and SBERT embedding $\mathrm{Enc}(\cdot)$0, 3D centroid $\mathrm{Enc}(\cdot)$1, axis-aligned bounding box $\mathrm{Enc}(\cdot)$2, and occupancy probability $\mathrm{Enc}(\cdot)$3, defined as the fraction of voxels consistently labeled $\mathrm{Enc}(\cdot)$4. The 2D Pillar Map stores, at resolution $\mathrm{Enc}(\cdot)$5 such as $\mathrm{Enc}(\cdot)$6 m, the top-$\mathrm{Enc}(\cdot)$7 instance-count pairs per cell with $\mathrm{Enc}(\cdot)$8 for fast semantic queries (Schperberg et al., 24 Jun 2026).

4. Vision-language grounding and semantic querying

The language interface of OpenRoboVox is built around caption-mediated visual grounding. The system employs an open-vocabulary detector, Yolo-World, trained via natural language prompt engineering; TAP generates captions that capture fine-grained visual descriptions such as "a red plastic cup on wooden table"; and SBERT maps captions into a $\mathrm{Enc}(\cdot)$9-dimensional embedding space (Deng et al., 23 Feb 2025). The report attributes to this design zero-shot category queries such as "find tomatoes," relevance queries such as "objects I can use for painting," and functionality queries such as "objects to sit on" (Deng et al., 23 Feb 2025).

A central claim is that caption encoding provides richer language reasoning than CLIP alone. The stated examples include material properties and functionality queries, and the mechanism is explicitly tied to captions passing through a lightweight LLM with only approximately $m_t^i$0 ms overhead per instance (Deng et al., 23 Feb 2025). Because the embeddings are updated incrementally in the codebook, the map supports retrieval under composite or previously unseen textual descriptions with no further training (Deng et al., 23 Feb 2025).

In RoboAtlas, this language-grounded representation is organized further into the Scene-Dictionary and the 2D Pillar Map. For semantic navigation, the Semantic Map Expert queries the Scene-Dictionary by filtering entries with cosine similarity above thresholds $m_t^i$1 and $m_t^i$2, serializing the top $m_t^i$3 entries into an LLM prompt, and asking the model to rank candidates; the chosen instance centroid $m_t^i$4 becomes a navigation waypoint. In parallel, the Egocentric VLM Expert reasons over the current RGB frame and annotation grid, and its JSON-formatted output either issues a waypoint or an in-place turn. Its textual explanations are appended to $m_t^i$5, which enriches the context for the Semantic Map Expert (Schperberg et al., 24 Jun 2026).

This suggests that OpenRoboVox is designed not only for label assignment, but also for queryable semantic abstraction across multiple spatial levels: voxel, instance, pillar cell, and global dictionary entry.

5. Quantitative performance and robotic deployment

The reported quantitative results on the OpenVox-derived evaluation emphasize zero-shot instance segmentation, semantic segmentation, and open-vocabulary retrieval. For 3D zero-shot instance segmentation on Replica (8 scenes) and ScanNet (6 scenes), metrics are AP, AP50, and AP25. Against ConceptGraphs, the reported Replica averages are AP $m_t^i$6, AP50 $m_t^i$7, and AP25 $m_t^i$8. On ScanNet, the reported averages are AP $m_t^i$9, AP50 $f_t^i$0, and AP25 $f_t^i$1 (Deng et al., 23 Feb 2025).

For 3D zero-shot semantic segmentation, the metrics are mIoU and mAcc, with comparisons to ConceptFusion, ConceptGraphs, and Open-Fusion. On Replica, the reported averages are mIoU: CF $f_t^i$2, CG $f_t^i$3, OF $f_t^i$4, Ours $f_t^i$5; and mAcc: CF $f_t^i$6, CG $f_t^i$7, OF $f_t^i$8, Ours $f_t^i$9. On ScanNet, the reported averages are mIoU: CF $V_t=\{v_t^j\}$0, CG $V_t=\{v_t^j\}$1, OF $V_t=\{v_t^j\}$2, Ours $V_t=\{v_t^j\}$3; and mAcc: CF $V_t=\{v_t^j\}$4, CG $V_t=\{v_t^j\}$5, OF $V_t=\{v_t^j\}$6, Ours $V_t=\{v_t^j\}$7 (Deng et al., 23 Feb 2025).

For open-vocabulary instance retrieval over 21 queries, the reported modes are Ontology, Relevance, and Functionality. The reported Recall@$V_t=\{v_t^j\}$8 values are Ontology $V_t=\{v_t^j\}$9 versus ConceptGraphs $\theta_t^j$0, Relevance $\theta_t^j$1 versus ConceptGraphs $\theta_t^j$2, and Functionality $\theta_t^j$3 versus ConceptGraphs $\theta_t^j$4 (Deng et al., 23 Feb 2025).

The real-time deployment profile is also specified in detail. On an off-board PC with an NVIDIA RTX 4090, OpenRoboVox is reported to achieve an end-to-end frame rate of $\theta_t^j$5--$\theta_t^j$6 Hz for RGB $\theta_t^j$7 with depth, mapping, and queries; front-end latency of approximately $\theta_t^j$8 ms; back-end latency of approximately $\theta_t^j$9 ms; and a memory footprint of approximately $D_t$00 GB for a $D_t$01 scene at $D_t$02 cm resolution. Onboard on an Autolabor M1 with an RTX 3060, sustained real-time operation is reported at $D_t$03--$D_t$04 Hz using an Azure Kinect for RGB-D and Livox MID-360 LiDAR SLAM for pose, with robustness to depth noise, segmentation jitter, and dynamic lighting (Deng et al., 23 Feb 2025).

In the RoboAtlas deployment, OpenRoboVox is reported to map approximately $D_t$05 over two floors in a single hardware run while extracting $D_t$06 unique semantic instances. Update rates are given as TSDF integration at approximately $D_t$07 Hz and perception plus fusion at approximately $D_t$08 Hz, while Scene-Dictionary refresh is stated to be sub-millisecond per instance via batched mask extraction, with a $D_t$09 speedup. The memory design bounds peak VRAM to $D_t$10 with $D_t$11, eliminating out-of-memory failures seen on RTX 4090 under naive resizing and yielding stable allocation over hours of mapping on Jetson Orin. Semantic accuracy is summarized as approximately $D_t$12 of instances maintaining correct MAP labels after incremental updates, with drift handled by Dirichlet smoothing (Schperberg et al., 24 Jun 2026).

6. Role in RoboAtlas, limitations, and proposed extensions

Within RoboAtlas, OpenRoboVox is the semantic mapping system that allows a contextual Active SLAM controller to balance geometric exploration and semantic reasoning. The larger system integrates frontier exploration, global semantic-map reasoning, and egocentric VLM-based reasoning through a contextual multi-armed bandit that transitions from exploration to semantically guided navigation as scene understanding improves. In simulation and on a Unitree Go2 robot, RoboAtlas is reported to operate in large-scale real-world environments exceeding $D_t$13 with approximately $D_t$14 mapped semantic instances and to achieve a $D_t$15 task success rate. On GOAT-Bench "Val Unseen," the reported success rate is $D_t$16 using GPT-4o, improving over the strongest prior baseline by $D_t$17 percentage points, while Qwen2.5-VL-7B achieves $D_t$18 and still outperforms all baselines using GPT-4o in success rate (Schperberg et al., 24 Jun 2026).

These results are used to support a specific interpretive claim in the source: they reveal the importance of the information gained by the semantic mapping framework over simply replacing the underlying foundation model (Schperberg et al., 24 Jun 2026). A common misunderstanding, therefore, is that performance gains in this setting derive primarily from the choice of LLM or VLM; the published comparison instead attributes a substantial part of the gain to the semantic mapping substrate.

The published design trade-offs and limitations are explicit. Asynchronous semantic processing prevents SLAM-chain stalls, but end-to-end decision latency still depends on remote LLM and VLM calls; deployments with unreliable communications may opt for smaller on-board models such as Qwen2.5-VL-7B. Fixed voxel size may under-represent small objects, and adaptive voxel sizing is identified as a possible way to improve small-object recall at the cost of more blocks. Errors in 2D detection or 3D association can propagate into the Scene-Dictionary, occasionally causing the Semantic Map Expert to propose invalid waypoints if an instance is never observed from a good viewpoint. The contextual bandit also requires a hand-tuned reward design combining coverage, backtracking penalty, VLM confidence, and similarity, and suboptimal weights can bias the system too strongly toward exploration or exploitation (Schperberg et al., 24 Jun 2026).

The discussion sections also outline several extensions. One proposal is dynamic object handling by adding temporal occupancy flags and per-instance velocity priors in the Dirichlet counts. Others include multi-robot collaboration by sharing codebook embeddings $D_t$19 and Dirichlet $D_t$20 across agents via sparse map merge, and sensor fusion that integrates LiDAR semantic cues or tactile feedback as additional $D_t$21 measurements. More broadly, future directions are summarized as dynamic scene adaptation, multi-agent collaborative mapping, and on-board language-driven task execution (Deng et al., 23 Feb 2025).

In aggregate, OpenRoboVox is characterized by the combination of TSDF geometry, probabilistic instance voxels, a height-compressed 2D pillar abstraction, and a queryable Scene-Dictionary. This combination is presented as the mechanism by which geometric world modeling, open-vocabulary semantic fusion, and language-conditioned robotic behavior are unified in a single mapping system (Schperberg et al., 24 Jun 2026).