QueryGaussian: Scalable and Training-Free Open-Vocabulary 3D Instance Retrieval

Abstract: Efficiently retrieving specific 3D instances from large-scale scenes via natural language prompts remains a formidable challenge in multimedia analysis. Existing approaches predominantly follow a "scene-level embedding" paradigm, which requires distilling high-dimensional semantic features into every 3D primitive. This strategy suffers from a fundamental architectural bottleneck: memory and computational costs scale linearly with scene complexity, inevitably triggering out-of-memory (OOM) failures in city-scale environments. To address this barrier, we propose QueryGaussian, a training-free framework for expeditious and scalable open-vocabulary 3D instance retrieval. Unlike holistic semantic distillation, QueryGaussian employs an instance-level query mechanism that decouples semantic understanding from geometric representation. Specifically, we leverage pre-trained 2D vision models to interpret user prompts and lift segmentation masks into 3D via a concurrent maximum-weight association strategy, ensuring semantic-visual consistency. To mitigate projection ambiguity, we introduce a temporal fusion module with multi-stage adaptive density clustering. Experimental results demonstrate that QueryGaussian not only matches the accuracy of state-of-the-art methods but also delivers a decisive efficiency leap, reducing GPU memory usage by over 70% and accelerating inference by 180x. Crucially, QueryGaussian enables expeditious instance retrieval on city-scale scenes containing tens of millions of Gaussians using consumer-grade hardware.

• The paper introduces a training-free method that decouples semantic segmentation from geometric representation to enable efficient 3D instance retrieval.
• It leverages pre-trained 2D segmentation and adaptive clustering to achieve high retrieval accuracy with reduced memory and computation.
• The approach scales from indoor scenes to city-scale environments without expensive scene-level embeddings or retraining.

QueryGaussian: Training-Free, Scalable Open-Vocabulary 3D Instance Retrieval

Motivation and Context

The rapid evolution of 3D Gaussian Splatting (3DGS) has enabled real-time, high-fidelity renderings of multimedia content across small indoor and large city-scale scenes. However, open-vocabulary 3D instance retrieval—identifying arbitrary objects via natural language in these large environments—remains limited due to the computational and memory overheads of scene-level semantic embedding paradigms. Prior methods rely on precomputing dense semantic features for each 3D primitive, causing severe scaling bottlenecks and frequent Out-Of-Memory (OOM) failures in urban-scale settings. QueryGaussian introduces an alternative: instance-level, training-free retrieval that strictly decouples semantic understanding from geometric representation. It leverages pre-trained 2D vision models for prompt-conditioned segmentation and lifts the results to 3DGS via an efficient, maximum-weight association and adaptive clustering.

Figure 1: QueryGaussian framework performs multi-view rendering, prompt grounded segmentation, and 2D-to-3D lifting, culminating in refined instance retrieval via adaptive clustering.

Methodology

Instance-Level Query Mechanism

QueryGaussian operates in three stages:

1. Render-and-Map: Multi-view images are rendered from the 3DGS scene, and for each pixel, the associated Gaussian with the maximum rendering contribution (weight) is recorded.
2. 2D Semantic Parsing and Lifting: An open-vocabulary 2D segmentation model (e.g., Grounded-SAM) interprets user prompts to generate masks for relevant regions in each view. Masked pixels are then mapped to their dominant 3D Gaussians using the maximum-weight index buffer, forming an initial candidate set.
3. Adaptive Instance Clustering and Temporal Fusion: Cross-view consistency filtering and multi-stage DBSCAN-based clustering suppress outliers and structural noise. Temporal fusion ensures incremental robustness, while tightness-driven refinement extracts geometrically cohesive clusters—critical for separating semantically relevant objects from floaters and reconstruction artifacts.

This pipeline avoids scene-wide semantic distillation or retraining—semantic association costs depend on the number of rendered pixels, not Gaussians—directly enabling scalable operation.

Experimental Evaluation

Retrieval Performance

QueryGaussian’s accuracy and efficiency are validated across two regimes: small-scale indoor scenes (≈$10^5$ Gaussians) and large-scale outdoor city scenes (≈$10^7$ Gaussians).

• On indoor datasets, mean IoU reaches 0.6380—superior to Gaussian Grouping (0.5582) and OpenGaussian (0.5095). Visual comparison demonstrates notably reduced floaters and precise boundary adherence.

  Figure 2: QueryGaussian generates cleaner segmentations (less noise, sharper boundaries) than embedding-based baselines on indoor scenes.

• On city-scale benchmarks, scene-level embedding baselines universally fail with OOM on consumer GPUs. QueryGaussian sustains a mean IoU of 0.7676 with no preprocessing or retraining, requiring only 7GB VRAM and query time in the order of 60 seconds.

  Figure 3: QueryGaussian reliably retrieves objects (e.g., buildings, vehicles) in large scenes where prior methods cannot operate due to memory constraints.

Efficiency Metrics

QueryGaussian reduces both memory and compute overhead:

• GPU memory usage is cut by over 70% relative to embedding-based methods.
• End-to-end query is accelerated by 180$\times$, complete in under a minute even for dense urban scenes.

Analyses of its clustering module reveal that multi-stage, tightness-driven refinement is essential for large-scale robustness—single-stage clustering degrades sharply, undermining both precision and recall.

Robustness and Limitations

Ablations confirm robust behavior under imperfect 2D masks: the refinement pipeline retains core object structures even when segmentation inputs are noisy or incomplete. Retrieval performance also scales with underlying 3DGS reconstruction quality (mIoU improves from 0.4810 to 0.6317 with better geometry). The upper bound remains dictated by base scene fidelity, not semantic processing.

Downstream Integration: 3D Spatial Reasoning Agent

QueryGaussian is integrated with a LLM to enable spatial Q&A. The LLM converts user queries into retrieval instructions; QueryGaussian returns instance masks and centroids, facilitating geometric reasoning. This approach is modular—zero-shot retrieval supports dynamic, multi-instance queries and visually grounded spatial logic.

Figure 4: QueryGaussian used within a spatial reasoning agent: LLM orchestrates retrieval; mask and coordinate outputs are used for spatial Q&A.

Implications and Future Directions

QueryGaussian materially expands practical instance retrieval in 3D media:

• Scalability: Its on-demand, image-space semantic querying bypasses memory constraints, supporting city-scale use on consumer GPUs.
• Flexibility: Training-free operation enables retrieval from any pre-existing 3DGS asset, supporting cold-start and dynamic environments.
• Modularity: It seamlessly integrates with higher-level reasoning frameworks, setting the stage for visually grounded spatial Q&A, indoor navigation, and urban data mining.

Theoretically, QueryGaussian exemplifies a paradigm shift from global, embedding-heavy semantics toward localized, prompt-conditioned association. This decoupling may inspire further work in efficient multimodal retrieval, physics-aware 3D reasoning, and scalable scene editing. Its limitation—reliance on the fidelity of the underlying 3DGS—highlights ongoing needs for improved base reconstruction, and the refinement modules may evolve to better handle ambiguous or complex segmentation prompts.

Conclusion

QueryGaussian establishes a new standard for open-vocabulary 3D instance retrieval—training-free, efficient, and fully scalable. By separating semantics from geometry and leveraging maximum-weight association with adaptive clustering, it achieves competitive or superior accuracy, resource efficiency, and extensibility. Its demonstrated integration with spatial reasoning agents positions it as a foundational building block for advanced multimodal and interactive 3D systems.