LightSplat: Fast and Memory-Efficient Open-Vocabulary 3D Scene Understanding in Five Seconds
Abstract: Open-vocabulary 3D scene understanding enables users to segment novel objects in complex 3D environments through natural language. However, existing approaches remain slow, memory-intensive, and overly complex due to iterative optimization and dense per-Gaussian feature assignments. To address this, we propose LightSplat, a fast and memory-efficient training-free framework that injects compact 2-byte semantic indices into 3D representations from multi-view images. By assigning semantic indices only to salient regions and managing them with a lightweight index-feature mapping, LightSplat eliminates costly feature optimization and storage overhead. We further ensure semantic consistency and efficient inference via single-step clustering that links geometrically and semantically related masks in 3D. We evaluate our method on LERF-OVS, ScanNet, and DL3DV-OVS across complex indoor-outdoor scenes. As a result, LightSplat achieves state-of-the-art performance with up to 50-400x speedup and 64x lower memory, enabling scalable language-driven 3D understanding. For more details, visit our project page https://vision3d-lab.github.io/lightsplat/.
Summary
- The paper presents a novel framework that employs compact 2-byte semantic indices for rapid, open-vocabulary 3D scene segmentation.
- It uses indexed feature mapping and single-step clustering to achieve a 50-400x speedup and reduce memory usage by 64x compared to traditional methods.
- The approach facilitates practical applications in robotics, AR/VR, and interactive 3D editing by enabling efficient language-driven scene understanding.
Overview of "LightSplat: Fast and Memory-Efficient Open-Vocabulary 3D Scene Understanding in Five Seconds"
The paper "LightSplat: Fast and Memory-Efficient Open-Vocabulary 3D Scene Understanding in Five Seconds" (2603.24146) introduces a novel framework designed to address the challenges of open-vocabulary 3D scene understanding. This framework aims to streamline the identification and segmentation of novel objects within complex 3D environments using natural language input, by circumventing the common pitfalls of high computational cost and memory usage found in existing methods.
Methodology
Semantic Injection and Cluster Formation
LightSplat employs a unique semantic injection method, where compact 2-byte semantic indices are embedded into 3D scene representations derived from multi-view images. This is facilitated by an indexed feature mapping that links 2D image semantics directly to the 3D structure through discrete mask indices, effectively bypassing iterative optimization processes.
In practice, the framework starts by extracting 2D object masks and corresponding CLIP features from multi-view images. Instead of per-Gaussian feature storage, which is memory-intensive, LightSplat assigns semantics to Gaussians that have substantial rendering contributions via these compact indices. Semantic consistency and inference efficiency are achieved via a single-step clustering process that groups semantically and geometrically related masks in 3D Figure 1.
Figure 1: Overall framework of LightSplat. Featuring indexed feature injection and context-aware 3D clustering for efficient open-vocabulary scene understanding.
The final step involves constructing an inter-mask graph and performing context-aware 3D clustering. This enables efficient cluster-level feature management with minimal overhead, facilitating rapid and interpretable 3D scene understanding.
Fast Inference
LightSplat's innovative inference process contrasts starkly with traditional methods. Instead of comparing all Gaussians or pixels, the model conducts fast text query retrieval through cluster-feature mapping Figure 2. This approach drastically reduces the complexity and time required for inference, demonstrating the system's efficiency.
Figure 2: Fast inference via cluster-feature mapping, enabling accelerated text-query retrieval.
Experimental Results
Quantitative Analysis
LightSplat was evaluated on multiple datasets including LERF-OVS, ScanNet, and DL3DV-OVS and demonstrated superior efficiency and accuracy compared to existing methods. Specifically, it achieved a 50-400x speedup in feature distillation with 64x lower memory usage, while surpassing state-of-the-art performance in segmentation tasks Figure 3.
Figure 3: Qualitative comparison for 3D Object Selection on LERF-OVS, showcasing improved object boundaries.
Robustness and Flexibility
The framework's robustness across diverse scene complexities is further validated through qualitative comparisons (Figures 5 and 6). Its application spans simple object selections to handling complex scenes with numerous similar objects and varying environmental attributes.
Figure 4: Model behavior visualization across complex indoor and outdoor scenes in DL3DV-OVS.
Figure 5: Semantic segmentation on ScanNet, evidencing robust performance across varied real-world scenarios.
Implications and Future Directions
The implications of LightSplat's approach are profound for real-world applications such as robotic manipulation, AR/VR, and interactive 3D scene editing. By enabling efficient semantic injection and rapid inference, LightSplat lays the foundation for scalable and practical implementations in dynamic environments.
Future developments could explore refining object feature selection and further increasing specificity in language-driven manipulation tasks, balancing precision with processing speed. Additionally, integrating more advanced semantic filters or adaptive clustering techniques may enhance the system's capability to tackle even more intricate scene compositions.
Conclusion
LightSplat significantly advances the field of 3D scene understanding by offering a fast, memory-efficient, and training-free solution to open-vocabulary segmentation. Its innovative design not only reduces computational demands but also ensures high-quality semantic understanding, driving forward practical applications and research opportunities in dynamic language-driven 3D environments.
Paper Prompts
Sign up for free to create and run prompts on this paper.
Top Community Prompts
Explain it Like I'm 14
LightSplat: A simple explanation for teens
What is this paper about?
This paper introduces LightSplat, a fast and memory‑friendly way for computers to understand 3D scenes using everyday words. Imagine asking, “Where’s the red mug?” and instantly seeing the red mug highlighted in a 3D model of a room. LightSplat makes this happen in about five seconds, without heavy training or huge memory use.
What questions does it try to answer?
- How can we let people find and label things in 3D scenes using any words they choose (open‑vocabulary), not just a fixed list like “chair” or “table”?
- How can we do this quickly, without spending minutes or hours “training” each scene?
- How can we use much less memory, so it works on big scenes and real devices?
How does it work? (Simple explanation with analogies)
LightSplat connects simple 2D understanding to 3D models in a smart, lightweight way.
First, a few tools and ideas:
- 3D Gaussian Splatting (3DGS): Think of a 3D scene built from thousands of tiny, soft bubbles (Gaussians). Together, they make the scene you can view from any angle, and they render very fast.
- SAM (Segment Anything Model): Given a photo, SAM draws outlines (masks) around objects—like a super‑fast digital scissors.
- CLIP: A model that understands image parts and text together. It can tell how similar a picture region is to words like “mug,” “plant,” or “red chair.”
Now, the LightSplat trick:
- Traditional methods store a big description (hundreds of numbers) on every little 3D bubble. That’s slow and eats memory.
- LightSplat does something smaller and smarter: it gives each bubble a tiny code (a 2‑byte index—think of it like a short barcode) instead of a huge description. A separate “lookup table” maps each tiny code to the full meaning from CLIP. This saves tons of memory.
Here’s the pipeline in everyday terms:
- From multiple photos of the scene, SAM cuts out object shapes (masks). CLIP gives each mask a meaning vector (how it relates to words).
- LightSplat projects those 2D masks onto the 3D bubbles and only tags the bubbles that really affect the pixels inside a mask (like marking only the bubbles that actually show up for that object).
- It throws away noisy or unhelpful masks that barely show up in 3D (like ignoring shadows or tiny fragments).
- It then links related masks across different photos if they overlap in 3D and mean similar things (like realizing “this mug from the left photo” and “that mug from the right photo” are the same actual mug).
- Finally, it groups those linked masks into object‑level clusters in 3D and assigns each cluster a single, clean meaning. During search (e.g., the text “red mug”), it only compares your words to these clusters—not to every bubble—so it’s very fast.
Why this is efficient:
- Storing a tiny code on each bubble is far lighter than storing a high‑dimensional feature.
- Grouping bubbles into object clusters means the system compares your query to ~100 clusters instead of ~100,000 bubbles or millions of pixels.
What did they find?
In tests on several benchmarks—LERF‑OVS (tabletop scenes), ScanNet (indoor scenes), and DL3DV‑OVS (large indoor and outdoor scenes)—LightSplat:
- Set up the 3D scene semantics in about 5 seconds (often 50–400× faster than previous top methods).
- Used about 64× less memory because it stores tiny codes per bubble instead of large feature vectors.
- Matched or beat the best previous accuracy on finding and segmenting objects by text, even in challenging cases (many similar objects, small objects, or big, complex scenes).
- Ran very fast at query time because it compares your text to a small set of object clusters.
Why this matters:
- Speed: No waiting around—near instant setup and fast responses to text queries.
- Scale: Memory savings make it practical for large scenes and real devices.
- Accuracy: Clean boundaries and consistent results across different camera views.
What’s the impact?
LightSplat makes language‑driven 3D understanding practical:
- Robots can quickly find objects to pick up (“pick up the blue bottle near the sink”).
- AR/VR apps can highlight or edit objects on the fly (“recolor the sofa,” “show me all the chairs”).
- 3D content creators can select and modify scene parts faster and with less hassle.
In short, LightSplat shows that you don’t need heavy training or huge memory to connect words to 3D scenes. By using tiny codes plus a smart lookup table, and by grouping related parts into object clusters, it delivers fast, accurate, and scalable 3D understanding that responds to natural language in just a few seconds.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a concise list of the key missing pieces, uncertainties, and unexplored aspects that future work could address.
- Dependence on 2D foundation models: No analysis of how errors in SAM masks or CLIP embeddings propagate to 3D semantics; robustness to domain shift, poor segmentation, or ambiguous/long-tail categories remains unquantified.
- Index capacity and management: The 2-byte index caps at 65,536 unique mask/cluster IDs; scalability to very large or long-duration scenes (e.g., city-scale scans) with more masks is not discussed, nor are index collisions or reuse policies across scenes.
- Threshold sensitivity: The method relies on several thresholds ($\tau_{\text{contrib}$, $\tau_{\text{noise}$, $\tau_{\mathrm{IoU}$, $\tau_{\mathrm{feat}$); there is no sensitivity analysis, auto-tuning strategy, or guidance for setting them across different datasets or scene types.
- Cluster feature aggregation: Averaging CLIP features for cluster representation is assumed; the impact of alternative aggregations (e.g., weighted by Gaussian contribution, robust/medoid pooling, attention-based fusion) on accuracy and stability is unexplored.
- Overlapping/ambiguous assignment: Each Gaussian is assigned the “most influential” mask index, but the handling of overlapping masks, multi-part objects, or multi-instance categories within a single cluster is unclear and not evaluated.
- Graph construction complexity: Pairwise IoU and feature similarity among masks are computed; scalability of the graph building to thousands of masks (quadratic growth) and possible approximate schemes (e.g., locality-sensitive hashing, sparsification) are not analyzed.
- Dynamic scenes: The pipeline assumes static 3DGS scenes; robustness to moving objects, changing illumination, or streaming video with incremental updates (e.g., online re-indexing and re-clustering) is not studied.
- Geometry quality dependence: Performance may hinge on high-quality 3DGS reconstructions; no experiments quantify robustness to geometry noise (e.g., mis-registered cameras, sparse views, reflective/translucent surfaces) or degraded Gaussian distributions.
- Mask filtering criteria: 3D-aware mask filtering uses $|\mathcal{G}_k| \ge \tau_{\text{noise}$; its failure modes (e.g., small but salient objects, thin structures) and how to balance recall vs. precision are not investigated.
- Open-vocabulary breadth: Query types are limited to category-level terms; compositional queries (e.g., “red chair”), relational queries (“chair left of table”), attributes (size, material), counting, and multi-object queries are not tested.
- Text encoder choices: Only CLIP (512-d) is used; the effect of different text-vision encoders (e.g., SigLIP, EVA-CLIP, multilingual models) or ensemble methods on retrieval and segmentation is not explored.
- Cross-lingual queries: Support for non-English queries and the impact of multilingual text embeddings on 3D retrieval is not addressed.
- Confidence and calibration: There is no uncertainty estimation or confidence calibration for retrieval scores; thresholds for selecting clusters given a query are not discussed.
- Incremental/interactive updates: How to insert, delete, or refine semantics online without re-running the entire pipeline (e.g., user edits, new masks or views) is not specified.
- Instance vs. category semantics: Averaging features may conflate distinct instances of the same category (e.g., multiple chairs); mechanisms for instance-level differentiation and tracking are not presented.
- Memory accounting: The 2-byte per-Gaussian claim excludes the size of the index-feature mapping and cluster structures; total memory (per-scene) vs. number of masks/clusters is not reported, particularly under worst-case mask counts.
- Hardware/reporting details: Speed claims (5 s distillation, ~500 FPS retrieval) lack hardware specs, parallelization details, and batch sizes; reproducibility of timing and scalability across GPUs/CPUs is uncertain.
- Fairness of comparisons: Some baselines (e.g., Dr.Splat) use reported numbers rather than re-runs; differences in preprocessing (e.g., SAM/CLIP versions, masking protocols) may affect fairness and are not fully controlled.
- DL3DV-OVS annotations: The open-vocabulary extension is self-annotated; annotation protocols, quality control, inter-annotator agreement, and release plans (licenses, coverage) are not detailed, raising questions about bias and generalizability.
- Evaluation scope on ScanNet: Results are reported on 10 scenes and subset class settings (19/15/10); performance on the full dataset, broader class coverage, and cross-scene generalization are not included.
- Large-scene cluster counts: Retrieval speed is tied to number of clusters; behavior when cluster counts are very high (e.g., dense urban scenes) and strategies to cap/merge clusters without hurting accuracy are not studied.
- Occlusion/transparency handling: The alpha/transmittance-based contribution may be sensitive to occlusions, transparency, and rendering order; systematic evaluation under these conditions is missing.
- Alternative clustering methods: The single-step connected-component approach is simple; comparisons to stronger graph/community-detection or learned clustering (even lightweight) are absent.
- Robustness to prompt phrasing and synonyms: Variability in text queries (synonyms, phrasing, granularity) and its effect on retrieval scores is not analyzed; query normalization strategies are not proposed.
- Extension beyond 3DGS: Applicability to other explicit or hybrid 3D representations (e.g., meshes, point clouds with radiance fields) is not explored; portability of the index-injection and clustering to those settings is unknown.
Practical Applications
Immediate Applications
Below are concrete, deployable-now use cases that leverage LightSplat’s training-free, 5-second semantic distillation, compact 2-byte indices, and cluster-level querying for real-time, open-vocabulary 3D understanding.
- Robotics and automation (robotics, manufacturing, logistics)
- Text-referred grasping and pick-and-place in static setups (e.g., “pick up the red mug on the left”)
- Rapid “teach-by-showing” stations where technicians scan a cell, then issue natural-language tasks referencing objects
- Tools/workflows: ROS node exposing a queryable 3D Gaussian map; cluster-ID to grasp-pose selector; warehouse bin-picking with natural-language targets
- Assumptions/dependencies: prebuilt 3DGS of the scene; mostly static environment; calibrated multi-view captures; GPU for real-time retrieval; CLIP/SAM models available
- AR/VR content authoring and editing (media, gaming, software)
- Natural-language scene editing (select/replace/stylize “all chairs”, “remove the vase”)
- On-the-fly labeling and asset organization of 3D captures for level design and virtual production
- Tools/workflows: plugin for GaussianEditor/GaussCtrl that maps text queries to 3D clusters; “select-by-text” brush; batch relabeling via cluster IDs
- Assumptions/dependencies: pre-optimized 3DGS; reliable SAM masks; static or quasi-static scenes
- Digital twin search and asset management (AEC, facilities, manufacturing)
- Queryable 3D facility models for maintenance (“show fire extinguishers”, “locate all ladders”), inventory, and space planning
- Tools/workflows: cloud or on-prem pipeline that ingests multi-view video, builds 3DGS, injects indices, and exposes a search API
- Assumptions/dependencies: sufficient coverage and lighting in capture; accurate camera poses; data governance for scanned spaces
- Retail and warehousing (retail, logistics)
- Planogram compliance and shelf audits via natural-language queries (“all cereal boxes facing backward”)
- AR-assisted item finders for workers (“highlight the blue bin labeled bolts”)
- Tools/workflows: handheld scan → 3DGS → LightSplat index injection → store-wide query dashboard
- Assumptions/dependencies: repetitive objects can be disambiguated by spatial context; frequent rescans for dynamic shelves
- Construction and safety (AEC, EHS)
- As-built verification and safety audits by querying assets and signage (“find missing guardrails”, “show fire exits”)
- Tools/workflows: site capture → queryable 3D twin → compliance report generation
- Assumptions/dependencies: periodic updates for evolving sites; occlusion handling; policy-compliant data retention
- Security and forensics (public safety)
- Natural-language forensic search in reconstructed scenes (“spent casings”, “broken glass near window”)
- Tools/workflows: scan-to-search pipeline; chain-of-custody metadata with cluster-level provenance
- Assumptions/dependencies: careful thresholding to avoid false positives; audit trails; controlled access to sensitive data
- Education and training (education, e-learning)
- Interactive 3D labs and classrooms where learners query objects (“highlight capacitors”, “locate emergency shutoff”)
- Tools/workflows: queryable viewer for class-specific 3D scenes with cluster-level annotations
- Assumptions/dependencies: prepared scenes; accessible GPU or cloud backend
- Real estate, tourism, and customer support (real estate, services)
- Voice-searchable virtual tours (“show the master closet”, “where’s the washer/dryer?”)
- Home support apps to help users locate items in their own scans (“find the spare batteries”)
- Tools/workflows: mobile capture → queryable tour viewer → shareable links
- Assumptions/dependencies: consent and privacy for residential captures; adequate scan completeness
- Insurance claims and assessment (insurance, finance)
- Adjusters search 3D captures for damaged items (“cracked tiles”, “water-damaged baseboards”)
- Tools/workflows: LightSplat-enabled claims intake viewer; export of query hits as claim evidence
- Assumptions/dependencies: variable scene conditions; explainability needs; secure data handling
- Research acceleration (academia, R&D)
- Rapid baselining and ablation for open-vocabulary 3D tasks; dataset curation; class-agnostic object mining
- Tools/workflows: drop-in LightSplat module for 3DGS pipelines enabling 5-second distillation and 500 FPS query retrieval
- Assumptions/dependencies: standard CLIP/SAM models; quality of multi-view coverage drives mask reliability
Long-Term Applications
The following use cases become practical as LightSplat’s methods are adapted to dynamic scenes, scaled environments, and embedded/edge hardware, and as additional engineering and validation are completed.
- Always-on AR glasses with on-device 3D understanding (consumer electronics, accessibility)
- Continuous, privacy-preserving “talk-to-your-environment” assistance (“where did I leave my keys?”)
- Tools/workflows: on-device 3DGS/SLAM + indexed semantics; low-power similarity search
- Assumptions/dependencies: efficient/online 3DGS or alternative mapping; dynamic-scene handling; battery/thermal constraints; on-device SAM/CLIP or distilled equivalents
- Autonomous mobile robots with closed-loop semantic planning (robotics, logistics, manufacturing)
- Real-time, open-vocabulary scene updates for navigation and manipulation in changing environments
- Tools/workflows: SLAM integration that streams updated indices; planners consuming cluster-level semantics
- Assumptions/dependencies: dynamic-object robustness; safety certification; multi-robot coordination; strong failure recovery
- City- and campus-scale digital twins (smart cities, urban planning)
- Open-vocabulary search over large public assets (“all damaged streetlights in Zone 3”)
- Tools/workflows: distributed capture → hierarchical, sharded index-feature maps; multi-tenant query services
- Assumptions/dependencies: privacy and data-minimization policies; scalability beyond 2-byte index limits; storage and governance at municipal scale
- Healthcare environments (healthcare, medtech)
- OR/ICU mapping for equipment localization; assistive robots referencing open-vocab targets
- Tools/workflows: hospital-compliant pipelines with access controls and audit logs; validated prompts
- Assumptions/dependencies: rigorous validation, bias and reliability studies; regulatory clearance; PHI-safe data handling
- Industrial inspection and energy infrastructure (energy, utilities)
- Open-vocabulary defect/asset search in plants and wind farms (“corroded flange”, “missing bolt on tower 12”)
- Tools/workflows: drone capture → 3DGS → indexed semantics; inspector consoles with query bookmarks and reports
- Assumptions/dependencies: robustness to outdoor lighting, weather, and scale; domain-adapted visual-LLMs
- Agriculture and environmental monitoring (agriculture, climate)
- Drone-based orchard/turf scans with open-vocabulary queries (“ripe apples”, “leaf blight patches”)
- Tools/workflows: seasonal mapping with temporal change detection on cluster groups
- Assumptions/dependencies: generalization of vision-language features to agricultural domains; large-area mapping efficiency
- Accessibility-first home/office assistants (consumer, enterprise)
- Contextual guidance for low-vision users in scanned environments (“guide me to the stove knobs”)
- Tools/workflows: privacy-preserving local inference; safe voice interfaces; persistent but updateable maps
- Assumptions/dependencies: continuous re-indexing as objects move; strong privacy controls; robust speech understanding
- Insurance underwriting and risk analytics (finance, insurtech)
- Property risk assessments from queryable 3D scans (“knob-and-tube wiring”, “blocked egress”)
- Tools/workflows: standardized scan-to-risk pipelines; audit-ready reports with cluster provenance
- Assumptions/dependencies: regulatory acceptance; explainability of open-vocab matches; curated taxonomies
- Standards and policy frameworks (public policy, compliance)
- Privacy-by-design guidance for 3D capture; energy/compute reporting standards favoring compact representations
- Tools/workflows: best-practice toolkits for minimizing stored data via indexed semantics; consent and masking protocols
- Assumptions/dependencies: multi-stakeholder alignment; procurement requirements in public sector; audit mechanisms
- Hardware and platform ecosystem (semiconductors, cloud)
- Accelerators for index-feature lookups and clustering; standardized APIs for 3D open-vocab search-as-a-service
- Tools/workflows: SDKs exposing cluster-level query endpoints; edge-cloud split inference; interoperable index formats
- Assumptions/dependencies: community adoption; licensing for foundation models; long-term support and benchmarks
Cross-cutting assumptions and dependencies
- Pre-requisite 3D reconstruction: LightSplat’s 5-second figure is for semantic injection; building a high-quality 3DGS from multi-view imagery still requires its own capture and optimization time.
- Scene properties: Best performance in static or slowly changing scenes with adequate multi-view coverage, lighting, and accurate camera poses.
- Foundation models: Availability and licensing of SAM/CLIP (or domain-adapted variants). Open-vocabulary quality depends on CLIP’s semantic coverage.
- Scale and memory: 2-byte indices imply up to 65,536 masks per scene; very large scenes may need hierarchical partitioning or extended index schemes.
- Safety-critical use: For robotics/healthcare/inspection, require formal validation, fallbacks, and human-in-the-loop oversight.
- Privacy and compliance: Scanning real spaces raises consent, retention, and access-control issues; index-only semantics can help minimize stored PII but do not eliminate legal obligations.
Glossary
- 3D-aware mask filtering: A preprocessing step that removes masks with weak 3D support to improve cross-view semantic consistency. "To ensure semantic consistency, we perform 3D-aware mask filtering"
- 3D Gaussian Splatting (3DGS): An explicit 3D scene representation using Gaussian primitives that enables real-time rendering and efficient access to scene elements. "To overcome these computational bottlenecks, 3DGS has emerged as a promising alternative, offering real-time rendering and explicit scene representation."
- Alpha-blending weights: Weights from the rendering equation that determine each Gaussian’s contribution to a pixel by compositing along a ray. "we compute their pixel-wise contributions using alpha-blending weights from the rendering equation"
- Autoencoder: A neural compression model used to reduce the dimensionality of language features. "compress them using an autoencoder and codebooks."
- CLIP: A vision-LLM that provides semantic embeddings aligning images and text in a shared space. "semantic embeddings from CLIP"
- Cluster-feature mapping: A mapping from 3D clusters to their aggregated language features for compact inference. "the cluster-feature mapping can be constructed immediately in this single step."
- Codebook: A discrete set of vectors used to quantize high-dimensional features. "codebook-based feature quantization struggles to fully cover the scene"
- Contrastive learning: A training strategy that separates semantic features by pulling similar pairs together and pushing dissimilar ones apart. "contrastive learning for semantic separation"
- Context-aware 3D clustering: Object grouping in 3D guided by both geometric overlap and semantic similarity. "which guides context-aware 3D clustering in a single step."
- Cosine similarity: A similarity measure between two feature vectors based on the cosine of the angle between them. "we compute the semantic similarity using the cosine similarity between their CLIP features"
- Differentiable rasterization: A rendering technique that allows gradients to flow through rasterization, enabling optimization or analysis of 3D primitives. "which can be efficiently rendered through differentiable rasterization."
- Feature distillation: The process of transferring semantic information from 2D foundation models into a 3D representation. "feature distillation is bottlenecked by iterative optimization that repeatedly aligns rendered views with CLIP embeddings."
- Feature quantization: Compression of high-dimensional features into a compact discrete representation. "feature quantization to compress language features."
- Gaussian primitives: The basic scene elements in 3DGS that model geometry and appearance using 3D Gaussians. "It models a scene using a collection of 3D Gaussian primitives"
- Graph diffusion: A propagation process on graphs used to spread or refine features/labels across connected nodes. "LUDVIG employs graph diffusion"
- Index-feature mapping: A lookup structure tying compact indices stored on Gaussians or clusters to their corresponding language features. "we propose an index-feature mapping that associates each 2-byte index to its corresponding CLIP feature."
- Intersection over Union (IoU): A measure of overlap between two sets, used here to quantify mask agreement via shared Gaussians. "we compute the geometric overlap using the Intersection over Union (IoU) of their corresponding Gaussian sets"
- Mask-feature mapping: A mapping from each 2D mask’s index to its CLIP feature for efficient semantic injection. "These indices are linked to CLIP features via a mask-feature mapping"
- [email protected]: Mean accuracy under a thresholded overlap criterion (0.25), used to evaluate selection/segmentation quality. "The rendered results are evaluated using mIoU and [email protected] against the ground-truth masks."
- mIoU: Mean Intersection over Union, an average overlap metric widely used to assess segmentation performance. "The rendered results are evaluated using mIoU and [email protected] against the ground-truth masks."
- NeRF: Neural Radiance Fields; an implicit volumetric representation for high-quality view synthesis. "NeRF provides high-quality rendering via MLP-based volumetric fields"
- Open-vocabulary: A setting where models must handle arbitrary user-specified categories without a fixed label set. "Open-vocabulary 3D scene understanding enables users to segment novel objects in complex 3D environments through natural language."
- Opacity: The per-Gaussian alpha parameter controlling how much a primitive contributes to the rendered color. "where is the opacity of the -th Gaussian."
- Per-Gaussian feature assignments: Attaching a semantic feature vector to each Gaussian primitive individually. "dense per-Gaussian feature assignments."
- Product Quantization: A vector quantization technique that splits features into subvectors and quantizes each to reduce storage and compute. "requires large-scale Product Quantization training"
- Radiance field: A function describing scene appearance (color/density) as a function of position and view direction. "distilling semantic embeddings from 2D foundation models like CLIP, DINO, and SAM to the radiance field."
- Single-step clustering: A one-pass procedure to form 3D object clusters by linking masks with geometric and semantic agreement. "We further ensure semantic consistency and efficient inference via single-step clustering"
- Transmittance: The fraction of light that reaches a given Gaussian along a ray after accounting for occlusions by preceding Gaussians. "The transmittance quantifies how much light reaches the -th Gaussian at pixel in the -th view."
- Volumetric rendering: Rendering by integrating densities and colors along camera rays through a volume, often computationally intensive. "slow and computationally intensive volumetric rendering."