VoteSplat: 3D Scene Understanding

• VoteSplat is a 3D scene understanding framework that integrates Hough voting with 3D Gaussian Splatting to enhance instance-level segmentation and semantic analysis.
• It augments standard 3D representations by learning per-Gaussian spatial offset vectors that collaboratively estimate object centers and mitigate depth ambiguity.
• Leveraging SAM for mask generation and open-vocabulary semantic mapping, VoteSplat delivers efficient, real-time 3D object localization and fine-grained scene interpretation.

VoteSplat is a 3D scene understanding framework that fuses Hough voting mechanisms with 3D Gaussian Splatting (3DGS), targeting accurate instance-level segmentation, object localization, and semantic understanding in 3D environments. The framework aims to resolve the limitations of standard 3DGS—which excels at real-time photorealistic rendering but is not designed for deep semantic scene analysis—by embedding spatial voting and learning mechanisms into the representation and rendering pipeline (Jiang et al., 28 Jun 2025).

1. Integration of 3D Gaussian Splatting with Hough Voting

VoteSplat advances traditional 3DGS by attaching a spatial offset vector to each Gaussian primitive in the scene. In standard 3DGS, a scene is represented as a set of anisotropic Gaussians parameterized by position, scale, orientation, opacity, and color. VoteSplat augments this by enabling each Gaussian, centered at position $p_i$, to cast a vote $V^{3d}_i = p_i + \Delta p_i$, where $\Delta p_i$ is a learned offset vector pointing toward the associated object instance center.

Instead of solely relying on differentiable alpha blending for rendering, VoteSplat computes instance center votes using a uniform average over all Gaussian votes associated with an instance: $$V^{3d} = \frac{1}{|\mathcal{M}|} \sum_{i \in \mathcal{M}} V^{3d}_i,$$ where $\mathcal{M}$ is the set of depth-ordered Gaussians contributing to the instance vote. This design overcomes the surface-clustering effect of 3DGS by allowing primitives from object surfaces to collaboratively estimate 3D instance centroids. The voting process is essentially a spatial generalization of the classic Hough voting algorithm, optimized for the continuous and probabilistic nature of Gaussian splats.

2. Instance Segmentation via Segment Anything Model (SAM) and 2D-3D Association

Instance segmentation in VoteSplat is bootstrapped by the Segment Anything Model (SAM), which generates high-quality masks from multiple training views. Each instance's mask, $M_l \in \{0,1\}^{H \times W}$ (for an image of size $H \times W$), yields a 2D vote via its centroid: $$c_x^l = \text{round} \left( \frac{\sum_{x,y} x \cdot M_l(x, y)}{\sum_{x,y} M_l(x, y)} \right), \quad c_y^l = \text{round} \left( \frac{\sum_{x,y} y \cdot M_l(x, y)}{\sum_{x,y} M_l(x, y)} \right).$$ These centroids, $V^{2d} = (c_x^l, c_y^l)$, serve as supervision targets for 3D voting. During optimization, the network learns offset vectors such that the projected 3D votes align with these image-based centers, robustly associating 2D semantic masks to 3D point clouds.

3. Spatial Offset Vectors and Depth Distortion Regularization

VoteSplat’s core mechanism is the optimization of per-Gaussian spatial offset vectors $\Delta p_i$, learned via supervision from 2D centroids. These vectors allow points sampled on object surfaces (which dominate 3DGS representations) to “vote” toward the center of their respective instances in 3D.

Ordinary projection causes depth ambiguity since projecting 3D centers to 2D loses information along the imaging axis. To mitigate this, VoteSplat employs a depth distortion constraint that regularizes the spread of votes in the depth dimension: $L_d^* = \sum_{i, j} |z_i - z_j|$ for all $i, j$ in the relevant set, encouraging votes to coalesce tightly along the z-axis (depth) and reducing variances that could degrade instance localization. Without this constraint, vote clusters are observed to be widely dispersed along depth, leading to imprecise instance assignments.

4. Open-Vocabulary 3D Object Localization and Semantic Mapping

For open-vocabulary tasks, VoteSplat bypasses direct CLIP feature embedding into every Gaussian, which is computationally expensive and can introduce semantic ambiguity. Instead, after clustering 3D votes (e.g., via HDBSCAN), each cluster is given a unique instance ID. A mapping is established from 2D Instance ID maps (rendered via the 3DGS pipeline) to corresponding CLIP features extracted from the semantic segmentation of RGB views.

This process associates each 3D instance cluster with open-vocabulary semantics by connecting the instance’s spatial votes to CLIP-derived representations, reducing supervised learning costs while maintaining semantic discrimination. This methodology supports zero-shot and language-guided object localization directly within the 3D scene.

5. Experimental Evaluation and Comparative Results

VoteSplat demonstrates superior performance in several benchmarks. It achieves improved mean Intersection-over-Union (mIoU) and mean accuracy (mAcc) for open-vocabulary 3D instance localization compared to methods such as LangSplat and OpenGaussian. The system supports robust click-based 3D object localization—where a 2D selection in the image reliably returns the corresponding 3D points—and produces densely clustered, semantically meaningful point groups even under occlusions.

Ablation studies confirm the necessity of key components: removing the depth distortion term results in poor vote consolidation along depth; disabling background filtering causes incorrect clustering. Hierarchical segmentation, enabled through multi-level SAM masks, allows VoteSplat to represent object-part hierarchies and produce granular structural analyses.

6. Hierarchical Segmentation and Extensibility

VoteSplat supports not just instance-level, but also hierarchical segmentation. By leveraging the layered mask outputs from SAM, the framework can construct a hierarchy—from object categories down to sub-parts—within the 3D point cloud. This capacity is especially significant for applications that require object part recognition or fine-grained editing in 3D content creation, robotics, or autonomous agent scene analysis.

The system is designed to be extensible: other segmentation models or annotation sources can be integrated to guide segmentation or voting, and the Gaussian representation can be leveraged for downstream tasks such as object manipulation, editing, or scene graph induction.

7. Practical Implementation and Resource Availability

The VoteSplat source code is publicly available at https://sy-ja.github.io/votesplat/. The implementation is compatible with conventional 3DGS pipelines, with added modules for SAM-based segmentation, offset learning, vote aggregation, clustering, and semantic association. This availability facilitates reproducibility, benchmarking, and further research on 3D scene understanding, particularly for projects requiring real-time performance and high-fidelity semantic segmentation (Jiang et al., 28 Jun 2025).

8. Significance and Impact

VoteSplat introduces a modular, interpretable, and computationally efficient approach for instance and part-level understanding of complex 3D scenes, advancing beyond the capabilities of existing photorealistic renderers by directly embedding semantic reasoning mechanisms. By connecting 2D vision foundation models to 3D point cloud representations via a voting-based pipeline, and regularizing semantic and spatial consistency, VoteSplat concretely bridges rendering and scene understanding in modern computer vision.

The demonstrated reductions in training cost and improvements in segmentation quality make VoteSplat suitable for applications in real-time robotics, digital content editing, autonomous driving perception, and AR/VR scene analysis. Its voting-based core sets a foundation for future research in 3D semantic aggregation and interpretable neural scene representations.