---
title: Probabilistic Fusion for Semantic & Geometric Info
url: https://www.emergentmind.com/topics/probabilistic-fusion-for-semantic-and-geometric-information
type: topic
---

# Probabilistic Fusion for Semantic & Geometric Info

Probabilistic fusion for semantic and geometric information refers to a set of techniques that combine uncertainty-aware semantic predictions (e.g., class probabilities from visual perception) and geometric information (e.g., 3D structure from depth, lidar, or SLAM) into coherent representations such as semantic maps, 3D reconstructions, or object-aware environments. These methods are foundational in robotics, autonomous driving, mixed reality, and computer vision, where robust interpretation of complex, noisy, multi-modal sensor data is required for decision making and scene understanding. The core principle is the adoption of Bayesian or probabilistically principled updates to maintain, integrate, and refine semantic and geometric evidence over time, accommodating sensor noise, label ambiguity, occlusion, and dynamics.

## 1. Bayesian Formulation for Semantic-Geometric Fusion

Probabilistic fusion is commonly grounded in recursive Bayesian estimation, where semantic and geometric evidence are modeled as latent random variables updated as new measurements arrive. The canonical update, as applied to semantic mapping tasks, utilizes the Bayes filter:

\[
p(\ell_t | z_{1:t}) \propto p(z_t | \ell_t)\,p(\ell_{t} | z_{1:t-1})
\]

where $\ell_t$ is the latent semantic label (or class-distribution) at a location or instance, and $z_t$ comprises the semantic evidence (e.g., segmentation scores, open-set tags) and geometric evidence (e.g., depth, pose, 3D association) at time $t$.

This approach underlies per-instance, per-voxel, or per-surfel fusion in dense mapping pipelines as well as landmark-based probabilistic SLAM [2402.04555][1609.05130][2202.12802]. Measurement likelihoods are factorized over the available semantic cues, such as open-set foundation-model labels and their corresponding similarity scores, and geometric cues, such as instance associations or 3D overlaps.

## 2. Fusion Architectures and Algorithmic Pipelines

Modern systems implement probabilistic semantic–geometric fusion via diverse algorithmic pipelines:

- **Volumetric and Surfel-based 3D Maps:**  
  Pipelines such as FM-Fusion [2402.04555] and SemanticFusion [1609.05130] maintain a geometric backbone (TSDF volumes, surfel clouds) where each spatial primitive carries a semantic probability mass function (pmf) over closed-set classes:

    - **FM-Fusion:** Maintains both a global TSDF and per-instance TSDF grids, updating their class pmfs using open-set measurements and a learned label-likelihood matrix, via recursive Bayes and running-average smoothing.
    - **SemanticFusion:** Attaches pmfs to surfels, updating via Bayesian multiplication of per-pixel CNN softmax probabilities with the surfel's prior, projected under tracked SLAM correspondences.

- **Sensor-Projection Fusion:**  
  For multi-modal data (e.g., camera-lidar fusion), probabilistic fusion involves projecting semantic probabilities from 2D camera images into 3D (lidar) space, correcting for pose, motion, and occlusion, then registering the resulting class distributions in 3D voxels, octrees, or point clouds [2003.01871][2007.05490][2006.04894]. Uncertainty is propagated through geometric transformations (including using the Unscented Transform for pose and projection models), and information from multiple frames is recursively integrated using cell-wise or voxel-wise Bayesian updates.

- **Probabilistic Data Association in Semantic SLAM:**  
  For SLAM with ambiguous object correspondence, frameworks compute marginal assignment probabilities over measurement-landmark pairs using k-best assignment enumeration, then use association-weighted measurement likelihoods combining semantic (e.g., class softmax) and geometric (e.g., ellipsoid or bounding-box overlap) factors in the joint inference objective [2202.12802].

- **Text-Guided and Multistream Attention Fusion:**  
  In neural approaches such as TextSplat, semantic–geometric fusion occurs at feature level: depth, semantic, and cross-view features are aligned, aggregated, and fused using attention mechanisms guided by textual embeddings, producing probabilistic Gaussian splats in 3D whose parameters (mean, covariance, opacity, color) are inferred via differentiable neural regression [2504.09588].

## 3. Measurement Uncertainty, Label Likelihoods, and Occlusion Handling

Most frameworks explicitly model and propagate measurement uncertainty:

- **Semantic likelihoods:**  
  Measurement likelihoods are extracted from CNN softmax outputs, open-set detection scores (from foundation models), or confusion matrices empirically estimated on held-out data [2402.04555][2006.04894]. For mapping, superpixel-based temperature scaling is often applied to flatten over-confident predictions in low-purity regions [2003.01871][2007.05490].

- **Geometric uncertainty:**  
  In lidar-camera fusion, point-level covariances are propagated through transform chains using the Unscented Transform, yielding uncertainty ellipses for projection into image space [2007.05490].

- **Occlusion and visibility:**  
  Dedicated viewpoint validation algorithms mask out occluded 3D points by simulating projective masks along depth ordering (front-to-back) in the image plane, ensuring label fusion only for visible, unoccluded points [2003.01871][2007.05490].

- **Instance association and merge:**  
  Data association steps exploit IoU in 2D and 3D, per-instance semantic similarity, and temporally linked open-set measurements to robustly associate observations or merge over-segmented regions [2402.04555].

## 4. Map Representation, Update Rules, and Refinement

Semantic–geometric probabilistic fusion produces map representations such as:

- **Dense voxel or surfel grids:**  
  Each unit stores a class pmf, updated recursively as new observations arrive—either by strict Bayes rule or by running-average smoothing to reduce over-confidence. Occupancy probabilities are similarly tracked for geometric reasoning [1609.05130][2007.05490].

- **Instance volumes:**  
  FM-Fusion maintains per-instance TSDFs, performing data-association and implementing merge/prune strategies for over-segmentation and geometric noise. Instance semantic pmfs are updated using a learned (matrix) probability of mapping open-set label and detection scores to closed-set classes [2402.04555].

- **Bird's-Eye-View (BEV) Grids:**  
  In mapping for autonomous driving, 3D point clouds "painted" from camera semantics are collapsed into 2D BEV cells, whose class pmfs are updated via a confusion-matrix likelihood and intensity priors—supporting probabilistic outputs necessary for downstream planners or control modules [2006.04894].

- **Gaussian Splatting:**  
  Representations such as in TextSplat treat each "splat" as a probabilistically inferred Gaussian kernel in $\mathbb{R}^3$, with geometric parameters and RGB attributes fused from multimodal features via neural, attention-based mechanisms [2504.09588].

Refinement modules in these pipelines include:

- **Boundary polishing:**  
  Merge heuristics based on semantic similarity and 3D IoU, deletion of noisy or unsupported voxels based on global geometry, and optional spatial or temporal smoothing [2402.04555][2007.05490].

- **Regularization:**  
  CRF-based spatial smoothing across adjacent surfels or voxels, and propagation of pmf fractions to neighbors for denoising [1609.05130][2007.05490].

## 5. Empirical Performance and Evaluation

The efficacy of probabilistic semantic–geometric fusion is consistently demonstrated across various benchmarks:

- **Instance-aware 3D semantic mapping:**  
  FM-Fusion yields 40.3 mAP@50% on ScanNet—a significant +15.6 point gain over the strong open-set baseline (Kimera + RAM-Grounded-SAM), with discrete ablations indicating that probabilistic likelihood integration, prompt augmentation, and instance merge refinement each contribute 3–10 mAP [2402.04555].

- **Dense RGB-D reconstruction:**  
  SemanticFusion demonstrates a 3–5% increase in class and pixel accuracy over single-frame segmentation on NYUv2 and office scenes, with additional gains from CRF regularization [1609.05130].

- **Sensor fusion benchmarks:**  
  In fused camera–lidar settings, proper motion correction and occlusion masking boost F1 scores by 4–5 points for thin or complex object classes; full probabilistic pipelines further reduce class confusion and improve map consistency [2003.01871][2007.05490].

- **SLAM scalability and accuracy:**  
  Probabilistic data association via k-best enumeration achieves near-exact marginal assignment probabilities in sub-millisecond time even with 50×50 assignment matrices; resulting semantic SLAM backends run in real time and maintain robustness in ambiguous, repetitive environments [2202.12802].

- **Text-guided reconstructions:**  
  TextSplat achieves 0.39 dB PSNR and 0.010 SSIM gains over previous best GGS on RealEstate10K/ACID, with a reported 3.2% IoU improvement for semantic consistency in part labeling [2504.09588].

## 6. Limitations, Assumptions, and Future Directions

Published works discuss several limitations:

- **Dynamic Environments:**  
  Most approaches assume static class probabilities per cell or voxel. Handling persistent dynamic objects or temporal label changes (e.g., FreMEn models, temporal decay) is an open direction [2007.05490].

- **Sparse Sensing and Noise:**  
  Sparse lidar or limited camera views reduce label fusion quality, especially at range or in occluded zones. Early mislabeling may persist in unsensed regions.

- **Computational Load:**  
  Uncertainty propagation via the Unscented Transform and dense per-point Bayesian integration are computationally intensive; hybrid EKF linearizations or parallelization are suggested extensions [2007.05490].

- **End-to-End Learning:**  
  Boundary and instance refinement remain largely heuristic; fully differentiable, loss-driven geometric-semantic polishing could further improve consistency and sharpness [2402.04555][2504.09588].

This suggests that future research will increasingly focus on scalable, uncertainty-aware, differentiable fusion architectures, robust to cross-domain shift, dynamic scenes, and open-set conditions.

## 7. Comparative Summary of Representative Frameworks

| Framework        | Semantic Inputs             | Geometric Representation | Fusion Mechanism                |
|------------------|----------------------------|-------------------------|---------------------------------|
| FM-Fusion [2402.04555]    | Open-set VLM detections      | TSDF (global + per-inst)  | Bayesian update, label-likelihood matrix, instance merge/refinement  |
| SemanticFusion [1609.05130] | CNN softmax, per-pixel        | Surfel cloud                | Per-surfel Bayes rule, CRF reg.           |
| Camera-Lidar Fusion [2007.05490] | 2D semantic with uncertainty | Octree voxels               | Bayes rule via integration over uncertain projections, occlusion-checking |
| BEV Mapping [2006.04894]         | Image segm. with conf. matrix| Grid/BEV                    | Per-cell Bayesian update w/ confusion matrix, intensity prior  |
| Probabilistic SLAM [2202.12802]  | Softmax scores, bbox/ellipsoid| Object landmarks             | Joint semantic-geometric measurement model, k-best data association    |
| TextSplat [2504.09588]           | Depth, semantic, cross-view   | 3D Gaussian splats           | Neural attention, text-guided fusion, end-to-end learning      |

Each of these frameworks embodies the fundamental principles of probabilistic semantic–geometric fusion: explicit modeling of uncertainty, principled Bayesian updates, strong coupling between geometry and semantics, and robust integration over time, sensors, and ambiguous evidence.

Source: https://www.emergentmind.com/topics/probabilistic-fusion-for-semantic-and-geometric-information