SLIM-VDB: Real-Time 3D Semantic Mapping

• SLIM-VDB is a real-time 3D semantic mapping framework that unifies closed-set and open-set semantic fusion using analytic Bayesian updates.
• It leverages OpenVDB’s sparse voxel grid for efficient TSDF integration, low memory consumption, and rapid GPU-based visualization.
• The framework demonstrates state-of-the-art geometric and semantic accuracy with significant improvements in integration speed over baseline methods.

SLIM-VDB is a real-time, probabilistic 3D semantic mapping framework that unifies closed-set and open-set semantic fusion within the computationally efficient OpenVDB hierarchical sparse voxel grid. It delivers real-time performance, state-of-the-art geometric and semantic accuracy, and a significant reduction in both memory overhead and integration time, supporting both fixed-category and open-language semantic labels (Sheppard et al., 15 Dec 2025).

1. Volumetric Scene Representation with OpenVDB

SLIM-VDB organizes scene geometry and semantics in a sparse, hierarchical voxel grid using the OpenVDB data structure. Each leaf voxel at spatial location $\mathbf{x}^* \subset \mathbb{R}^3$ stores:

• Truncated signed distance value: $D_t(\mathbf{x}^*)$
• Integration weight: $W_t(\mathbf{x}^*)$
• Semantic parameters: either a Dirichlet $\alpha$-vector (closed-set) or $(\mathbf{m}, \lambda, \nu, \beta)$ tuple (open-set)

Key properties of OpenVDB leveraged by SLIM-VDB include:

• Average $O(1)$ complexity for lookup, insertion, and deletion due to the B$^+$-tree hierarchical design
• Memory allocation only in local regions near observed surfaces (sparse allocation)
• No requirement for predefined global map boundaries
• Efficient raycasting with Differential Digital Analyzer for integrating truncated signed distance function (TSDF) updates
• Real-time GPU-based visualization using NanoVDB

TSDF integration is performed through per-voxel incremental updates. For a voxel $\mathbf{x}^*$ and a new signed distance measurement $d_t(\mathbf{x}^*)$ (from a pointcloud or depth image):

$$D_t(\mathbf{x}^*) = \frac{W_{t-1}(\mathbf{x}^*)D_{t-1}(\mathbf{x}^*) + \hat{w}(\mathbf{x}^*)d_t(\mathbf{x}^*)}{W_t(\mathbf{x}^*)}$$

$$W_t(\mathbf{x}^*) = W_{t-1}(\mathbf{x}^*) + \hat{w}(\mathbf{x}^*)$$

Here, $\hat{w}(\mathbf{x}^*)$ is the per-measurement weight.

2. Unified Bayesian Semantic Fusion

SLIM-VDB generalizes semantic fusion by maintaining a conjugate prior at each voxel, enabling analytic posterior updates for new semantic observations. This supports both:

• Closed-set semantics (fixed-category labels) via Dirichlet–Categorical conjugacy
• Open-set (language-based, embedding-based) semantics using Normal–Inverse-Gamma–Normal conjugacy

2.1 Closed-Set (Dirichlet–Categorical) Fusion

For discrete categorical labels $z \in \{1, \ldots, K\}$, a Dirichlet prior $\text{Dir}(\theta|\alpha)$ is placed on the class probabilities $\theta \in \Delta^K$:

$$p(\theta|\alpha) = \text{Dir}(\theta|\alpha) \propto \prod_{i=1}^K \theta_i^{\alpha_i - 1}$$

Upon observing $z_n$, parameters update as $\alpha_i \leftarrow \alpha_i + 1_{[z_n = i]}$ for $i = 1 \ldots K$. The posterior predictive class probability at a voxel is then:

$$p(z=i|\alpha) = \frac{\alpha_i}{\sum_{j=1}^K \alpha_j}$$

During semantic fusion, each voxel accumulates $\alpha$-counts corresponding to frame-wise categorical label assignments.

2.2 Open-Set (Normal–Inverse-Gamma–Normal) Fusion

Open-set segmentation networks yield $l$-dimensional feature embeddings $\mathbf{Z} \in \mathbb{R}^l$. Each dimension $Z_i$ is modeled as Gaussian with mean $\mu_i$ and variance $\sigma_i^2$, equipped with a Normal–Inverse-Gamma prior:

$$p(\mu_i, \sigma_i^2) = \mathcal{N}(\mu_i | m_i, \sigma_i^2/\lambda_i) \cdot \text{Inv-}\Gamma(\sigma_i^2 | \nu_i, \beta_i)$$

Given observation $z_{i,n}$, conjugate updates are:

$\tilde{\lambda}_i = \lambda_i + 1$

$$\tilde{m}_i = \frac{\lambda_i m_i + z_{i,n}}{\tilde{\lambda}_i}$$

$\tilde{\nu}_i = \nu_i + \frac{1}{2}$

$$\tilde{\beta}_i = \beta_i + \frac{1}{2}(z_{i,n} - \bar{z}_i)^2 + \frac{\lambda_i (z_{i,n} - m_i)^2}{2\tilde{\lambda}_i}$$

After $T$ samples, the predictive for $Z_i$ is a Student-t; the mean $m_i$ is used as a semantic feature estimate. The TSDF weight $W_t(\mathbf{x}^*)$ acts as an implicit count to simplify scale parameter updates.

3. Semantic Probability Representation and Uncertainty

At each voxel, SLIM-VDB stores the following:

• Closed-set: $\alpha$-vector $\alpha(\mathbf{x}^*) \in \mathbb{R}^K_{+}$, initialized uniformly. The class probability is $p_i = \alpha_i / \sum_j \alpha_j$.
• Open-set: Mean vector $\mathbf{m}(\mathbf{x}^*) \in \mathbb{R}^l$, scale and variance parameters, prediction derived via Student-t or taking the softmax over $\mathbf{m}$.

SLIM-VDB supports explicit uncertainty quantification by thresholding the predicted semantic probabilities (e.g., discarding voxels with $\max p_i < p_t$ for some $p_t$).

4. Real-Time Mapping and Fusion Pipeline

A high-level pseudocode demonstrates the mapping and fusion process:

Initialize OpenVDB grid G For each incoming frame (RGB-D or LiDAR + semantics): # 1. Pre-process P = backprojectDepthToPointcloud(depth) S = semanticNetwork(images) # categorical labels or CLIP embeddings # 2. TSDF update with raycasting for each point p in P: for each voxel x* along ray to p within truncation band: d = signedDistance(x*, p) w_t = 1 G.D[x*] = (G.W[x*]*G.D[x*] + w_t*d) / (G.W[x*] + w_t) G.W[x*] += w_t # 3. Semantic Fusion if closed_set: for each point p with label z in P: x* = voxelOf(p) G.alpha[x*][z] += 1 else: # open_set for each point p with embedding Z in P: x* = voxelOf(p) update_NI_Gamma(G.m[x*], G.beta[x*], G.W[x*], Z) # 4. (Optional) Visualization renderWithNanoVDB(G)

Real-time performance is achieved by sparse voxel allocation, local incremental updates, $O(1)$ semantic posterior computation per sample, and NanoVDB GPU acceleration.

5. Quantitative Performance Evaluation

SLIM-VDB is benchmarked against state-of-the-art semantic mappers on the SceneNet (indoor) and SemanticKITTI (outdoor) datasets.

Runtime and Memory Usage

Method	SceneNet FPS	Memory [GB]
VoxField	2.49	1.05
SNI-SLAM	0.27	19.43
SLIM-VDB* (non-Bayes)	16.55	0.51
ConvBKI	1.10	14.32
SEE-CSOM	2.67	5.19
SLIM-VDBᶜ (Bayes)	10.84	1.08
LatentBKI (open-set)	1.67	27.68
SLIM-VDBᵒ (open-set)	1.76	3.49

• SLIM-VDB delivers 5–10× faster integration than Bayesian baselines.
• Memory consumption is reduced by 80–90% versus dense approaches.

Semantic Accuracy (mIoU)

Dataset	ConvBKI	SEE-CSOM	SLIM-VDB*	SLIM-VDBᶜ	SLIM-VDBᵒ
KITTI	0.165	0.128	0.216	0.252	–
SceneNet	0.049	0.128	0.087	0.122	0.116

Geometric Chamfer Distance ($\ell_2$)

Dataset	VoxField	ConvBKI	SEE-CSOM	SLIM-VDB
KITTI	6.079	4.000	504.952	0.079
SceneNet	0.008	1.189	0.003	0.004

Semantic and geometric accuracy is on par with or superior to contemporaneous semantic mapping systems (Sheppard et al., 15 Dec 2025).

6. Strengths, Limitations, and Application Domains

Strengths:

• Efficient, sparse, and GPU-accelerated scene representation
• Unified analytic Bayesian updates for both closed-set and open-set semantic fusion
• Real-time integration attainable on both desktop and embedded (Jetson Orin) platforms
• Low memory requirement and no fixed-world bounding constraints
• Supports explicit semantic uncertainty estimates for downstream tasks

Limitations:

• Visualization rendering is presently static per viewpoint
• Open-set semantics require storage of moments (mean/variance) per voxel, increasing overhead for high-dimensional embeddings
• Scalability to extremely large-scale scenes introduces memory and allocation overheads

Applications:

• Semantic-aware mobile robot navigation
• Active perception and planning using semantic uncertainty
• AR/VR 3D scene reconstruction with open vocabulary or language-based labeling
• Multi-robot semantic mapping with efficient, distributed data fusion

In summary, SLIM-VDB demonstrates that integrating OpenVDB with Bayesian conjugate priors enables a unified, efficient, and robust semantic mapping solution applicable to a wide range of robotics and 3D scene understanding scenarios (Sheppard et al., 15 Dec 2025).