EmbodiedSplat: Online Feed-Forward Semantic 3DGS for Open-Vocabulary 3D Scene Understanding

Abstract: Understanding a 3D scene immediately with its exploration is essential for embodied tasks, where an agent must construct and comprehend the 3D scene in an online and nearly real-time manner. In this study, we propose EmbodiedSplat, an online feed-forward 3DGS for open-vocabulary scene understanding that enables simultaneous online 3D reconstruction and 3D semantic understanding from the streaming images. Unlike existing open-vocabulary 3DGS methods which are typically restricted to either offline or per-scene optimization setting, our objectives are two-fold: 1) Reconstructs the semantic-embedded 3DGS of the entire scene from over 300 streaming images in an online manner. 2) Highly generalizable to novel scenes with feed-forward design and supports nearly real-time 3D semantic reconstruction when combined with real-time 2D models. To achieve these objectives, we propose an Online Sparse Coefficients Field with a CLIP Global Codebook where it binds the 2D CLIP embeddings to each 3D Gaussian while minimizing memory consumption and preserving the full semantic generalizability of CLIP. Furthermore, we generate 3D geometric-aware CLIP features by aggregating the partial point cloud of 3DGS through 3D U-Net to compensate the 3D geometric prior to 2D-oriented language embeddings. Extensive experiments on diverse indoor datasets, including ScanNet, ScanNet++, and Replica, demonstrate both the effectiveness and efficiency of our method. Check out our project page in https://0nandon.github.io/EmbodiedSplat/.

• The paper introduces EmbodiedSplat, a feed-forward framework that adds open-vocabulary semantics to 3D Gaussian Splatting using sparse CLIP codebooks and geometry-aware features without per-scene optimization.
• EmbodiedSplat-fast achieves 43.89, 32.43, and 51.09 mIoU on ScanNet, ScanNet200, and ScanNet++ while processing 363 images in 70 seconds at 5.18 FPS.
• The sparse coefficient field compresses CLIP features by 67×, but performance still depends on feed-forward depth and reconstruction quality, especially for ceilings, transparent surfaces, and out-of-domain scenes.

Motivation and problem setting

Embodied agents require 3D perception that is online (synchronous with exploration), real-time, generalizable to novel scenes, whole-scene in scope, and open-vocabulary. Existing open-vocabulary semantic 3DGS methods satisfy none of these jointly. Rasterization-based distillation approaches such as LangSplat and LEGaussians require per-scene optimization taking roughly six hours; direct feature-lifting methods such as Dr. Splat and Occam's LGS reduce inference cost but still optimize per scene (~2 hours); SLAM-based online methods such as Online-LangSplat achieve only ~1.12 FPS; feed-forward methods such as LSM operate on only a few views and cannot reconstruct a full scene. EmbodiedSplat addresses this gap by endowing the pretrained feed-forward 3DGS FreeSplat++ with open-vocabulary semantics, producing whole-scene language-embedded Gaussians from over 300 streaming images without any per-scene optimization.

Method

The framework builds on FreeSplat++'s online-capable pipeline: a CNN encoder predicts pixel-aligned local Gaussian triplets (positions, confidences, latents) plus depth; overlapping local and global Gaussians are fused via confidence-weighted position updates and GRU-based latent fusion; an MLP decoder produces final Gaussian parameters. EmbodiedSplat augments each Gaussian with two complementary semantic representations.

2D CLIP features via sparse coefficient field. Rather than rasterizing features into 2D (the "3D-to-2D" route of prior work), pixel-level CLIP features are directly unprojected onto Gaussians ("2D-to-3D"). Storing a dense CLIP vector per Gaussian is memory-prohibitive, so the method introduces a CLIP Global Codebook with an Online Sparse Coefficient Field. Instance-level CLIP features, obtained by pooling pixel-wise features over FastSAM masks, are appended to a monotonically growing global codebook $\mathbf{C}^t$. Each Gaussian stores instead two length-$L$ caches: an index cache into the codebook and a weight cache of sparse coefficients. During fusion, indices accumulate across views while weights update under confidence-weighted averaging; pruning to the top $L-1$ entries keeps cache size fixed at 10 numbers per Gaussian ($L=6$) versus a 512–768 dimensional vector. The final per-Gaussian embedding is reconstructed as a sparse linear combination of codebook entries. Because the codebook stores unmodified instance-level CLIP features, no pretraining or per-scene codebook optimization is needed, and full CLIP semantics are preserved — unlike auto-encoder compression (LangSplat), product quantization (Dr. Splat), or per-scene codebooks (OpenGaussian, InstanceGaussian).

Geometry-aware 3D CLIP features. Since image-derived CLIP features lack explicit 3D priors, semantic-aware latents $$\mathbf{g}^t_l = \mathbf{f}^t_l + \mathrm{proj}(\mathbf{s}^t_l)$$ together with Gaussian centers form a feature point cloud processed by a 3D sparse U-Net (Minkowski Res16UNet18A) with a temporal memory adapter that injects previously reconstructed global latents. Fused 3D features are projected back to CLIP space by a lightweight MLP decoder. Training minimizes cosine similarity between reconstructed 2D and decoded 3D features; ground-truth class labels are not used, and the FreeSplat++ backbone remains frozen. At inference, 2D and 3D probabilities are ensembled via a geometric mean.

EmbodiedSplat-fast. A lightweight variant replaces heavy 2D foundation models with real-time ones (FastSAM + Mask-Adapter), drops the 3D U-Net entirely (making it training-free and composable with arbitrary 2D VLMs), and introduces codebook-based cosine similarity. Because inner products are linear for unit vectors, text-to-Gaussian similarity reduces to a weighted sum of precomputed text-to-codebook similarities, lowering complexity from $O(MD)$ to $O(KD + M(L{-}1))$ where $K \ll M$. This yields a measured 14× speedup (1.18 ms vs. 14.35 ms on a 3.2M-Gaussian scene) and enables 5–6 FPS per-frame processing.

Results

On 3D semantic segmentation evaluated against annotated point clouds (Mahalanobis-distance aggregation of per-Gaussian logits), EmbodiedSplat substantially outperforms all baselines while being the only generalizable, online, whole-scene method:

Method	ScanNet 19-class mIoU	ScanNet200 70-class mIoU	ScanNet++ 20-class mIoU	Reconstruction time (363 imgs)
LangSplat	1.34	0.72	2.21	~6 hr
OpenGaussian	22.52	15.15	25.65	~2.5 hr
Occam's LGS	30.49	20.32	34.08	~2 hr
Dr. Splat	28.38	19.29	39.85	~2 hr
InstanceGaussian	26.57	23.20	29.98	~3 hr
Online-LangSplat	3.45	2.45	4.51	5.4 min (1.12 FPS)
EmbodiedSplat-fast (RGB-D)	43.89	32.43	51.09	1 min 10 s (5.18 FPS)

With RGB-D input, EmbodiedSplat reaches 57.41 mIoU on ScanNet 10-class and 52.12 on 19-class, roughly 15–38 points above the strongest offline baselines at reconstruction times of minutes rather than hours. Ablations confirm that combining 2D and 3D features improves mIoU consistently over either alone (e.g., 46.22 vs. 45.09/45.39 on ScanNet 19-class), and cache size $L=6$ suffices (44.38 at $L=2$ vs. 45.09 at $L$0).

Memory efficiency is a central quantitative claim: the sparse coefficient field achieves an average 67× compression versus storing raw 768-d CLIP features per Gaussian (148 MB vs. 2295 MB for Occam's LGS on one ScanNet scene), with no information loss and no pretraining — though this comparison assumes the codebook basis count $L$1 stays small relative to $L$2, which holds empirically (8.7K entries for 3.2M Gaussians).

Cross-domain evaluation reveals a clear asymmetry. ScanNet++ → ScanNet transfer degrades by less than 1 mIoU relative to in-distribution performance, but ScanNet → ScanNet++ drops by 11.28 mIoU, which the authors attribute to poor predicted depth on ceilings largely absent from ScanNet training data; supplying sensor depth restores near-in-distribution performance (44.14 vs. 44.03 mIoU). On the real-to-synthetic ScanNet → Replica transfer, EmbodiedSplat does not beat per-scene optimized baselines, achieving results comparable to clustering methods and, with RGB-D, on par with feature-lifting methods such as Dr. Splat. Supplementary experiments show competitive novel-view synthesis inherited from FreeSplat++ (e.g., 21.54 iPSNR / 0.330 LPIPS on ScanNet++, close to its offline teacher's 22.63 / 0.261) and 2D-rendered segmentation comparable to LangSplat despite no per-scene optimization.

Limitations

The paper concedes several dependencies explicitly. The system inherits FreeSplat++'s failure modes: inaccurate 3DGS reconstruction propagates directly into noisy semantic fields, which manifests in out-of-distribution settings (synthetic Replica scenes) and regions poorly covered by training data (ceilings, transparent surfaces). Sensor depth mitigates but does not eliminate this, and the RGB-D variant still requires hole-filling with predicted depths. The fast variant sacrifices the 3D geometric branch entirely, so its speed advantage comes at some representational cost. The evaluation is confined to indoor datasets with dataset-provided camera poses; performance under estimated poses or outdoor conditions is not assessed. An open question left by the cross-domain results is how to make predicted depth robust to viewpoint distributions absent from training, since this currently gates generalization more than the semantic components do.

Conclusion

EmbodiedSplat demonstrates that a frozen feed-forward 3DGS backbone can be augmented with open-vocabulary semantics through a training-light combination of a growing instance-level CLIP codebook, per-Gaussian sparse coefficient caches, and a geometry-aware 3D feature branch. It attains state-of-the-art indoor 3D semantic segmentation among semantic 3DGS methods while reducing whole-scene reconstruction from hours to minutes, or to 5–6 FPS per frame in its fast configuration — the first feed-forward 3DGS supporting online, whole-scene, open-vocabulary perception. Its principal constraints are inherited depth/reconstruction quality and limited evidence outside real-world indoor domains.