3D Queryable Scene Representation

• 3D QSR is a family of 3D scene representations that enables querying of spatial, semantic, and relational features using continuous neural fields, sparse primitives, and graph structures.
• These approaches support queries from coordinate lookups and open-vocabulary retrieval to explicit symbolic operations, benefiting applications in robotics, autonomous driving, and indoor scene understanding.
• Robust queryability is achieved through advanced supervision and alignment strategies that integrate geometric cues with language models for scalable and interpretable scene analysis.

Searching arXiv for papers on 3D Queryable Scene Representation and closely related scene-graph, occupancy, and multimodal scene-representation work. arXiv search results reviewed for recent relevant literature on 3D queryable scene representations, including scene graphs, occupancy fields, and multimodal 3D reasoning interfaces. 3D Queryable Scene Representation (3D QSR) is best understood as a family of 3D scene representations designed to make scene structure computationally accessible through queries. In the current literature, those queries range from continuous coordinate lookups of occupancy or local scene features, to open-vocabulary text-conditioned retrieval, to explicit graph or database operations over objects, parts, rooms, and relations. The field therefore spans continuous neural fields, sparse primitive-based occupancy models, open-vocabulary semantic maps, semantic-topological maps, 3D scene graphs, and multimodal scene packages tailored to large vision-LLMs (VLMs) or multimodal LLMs (MLLMs) (Sitzmann et al., 2019, Yamazaki et al., 2023, Mehan et al., 2024, Koch et al., 2024, Li et al., 1 Feb 2025, Li et al., 24 Sep 2025).

1. Scope and defining properties

A useful generic formalization appears in the 3D Scene Question Answering survey, which writes scene querying as

$\mathcal{F} : (S, Q) \mapsto (T, B),$

where a scene representation $S$ and a query $Q$ produce an answer $T$ and sometimes grounded regions $B$ (Li et al., 1 Feb 2025). Within that framing, a 3D QSR is not restricted to one storage format. The representation may be a point cloud, a set of multiview images, a scene graph, a semantic field, or a hybrid object-centric memory, provided that it preserves enough geometry, semantics, and cross-view consistency to support querying.

The surveyed query space is broad. It includes object-centric queries, attribute queries, spatial queries, relational and compositional queries, grounding-oriented queries, semantic and functional queries, embodied or agent-centric queries, navigation and planning queries, dialogue and instruction-following, and emerging temporal or episodic queries (Li et al., 1 Feb 2025). This breadth explains why 3D QSR has developed along several distinct technical lines rather than converging on a single canonical representation.

A recurrent point in the literature is that “queryable” does not denote a single interface. In some systems, queryability means continuous evaluation at arbitrary 3D coordinates, as in coordinate-indexed neural fields (Sitzmann et al., 2019). In others, it means open-vocabulary retrieval in a shared vision-language embedding space (Yamazaki et al., 2023). In yet others, it means explicit symbolic-geometric retrieval through a graph database and a formal query language such as Cypher (Ray et al., 18 Oct 2025). This distinction is important because a representation may be highly queryable in one sense while lacking capabilities expected in another.

2. Major representation families

The literature now supports a fairly clear taxonomy of 3D QSR designs.

Family	Typical stored structure	Representative examples
Continuous neural or occupancy fields	Coordinate-indexed local features, occupancy, semantics	SRNs (Sitzmann et al., 2019), QueryOcc (Lilja et al., 21 Nov 2025)
Sparse primitive-based scene fields	Superquadrics, Gaussians, TSDF-linked region embeddings	QuadricFormer (Zuo et al., 12 Jun 2025), Open-Fusion (Yamazaki et al., 2023), Query3D (Chahe et al., 2024)
Graph and topological representations	Nodes, edges, containment, connectivity, room topology	QueSTMaps (Mehan et al., 2024), Open3DSG (Koch et al., 2024), FreeQ-Graph (Zhan et al., 16 Jun 2025), Cypher-backed 3DSGs (Ray et al., 18 Oct 2025)
Multimodal VLM-native scene packages	Object-coordinate text, multiview images, hierarchical visual tokens, part-aware superpoints, point maps	HMR3D (Li et al., 28 Nov 2025), PAR3D (Dai et al., 4 Jun 2026), POMA-3D (Mao et al., 20 Nov 2025), multimodal 3D QSR (Li et al., 24 Sep 2025)

Continuous field methods treat the scene as a function over 3D coordinates. Scene Representation Networks define

$\Phi : \mathbb{R}^3 \to \mathbb{R}^n,$

so that a world-space coordinate returns a learned local scene descriptor (Sitzmann et al., 2019). QueryOcc follows the same query-centric principle in an occupancy setting, but conditions the field on current multi-view images and a 4D query $\mathbf q=[x,y,z,t]^\top$, predicting occupancy and semantics on demand rather than preclassifying a dense voxel tensor (Lilja et al., 21 Nov 2025).

Primitive-based methods replace dense scene volumes with explicit scene elements. QuadricFormer models a scene as a set of learnable superquadrics,

$$\mathcal{Q}=\{(\mathbf{x}_i,\mathbf{s}_i,\mathbf{r}_i,a_i,\epsilon_{1;i},\epsilon_{2;i},\mathbf{c}_i)\}_{i=1}^{P},$$

so geometry, semantics, and occupancy are all attached to sparse primitives (Zuo et al., 12 Jun 2025). Open-Fusion couples a sparse voxel-block TSDF with an embedding dictionary, allowing open-vocabulary 3D retrieval through region-level language-aligned embeddings (Yamazaki et al., 2023). Query3D stores compressed language semantics in a Language Embedded 3D Gaussian representation and answers natural-language queries through rendered relevance maps (Chahe et al., 2024).

Graph and topological systems make queryability explicit at the symbolic level. QueSTMaps constructs a semantic topological map in which rooms and transition regions form nodes and connectivity is encoded through edges, then aligns room embeddings to language for room-level querying such as “place to cook” (Mehan et al., 2024). Open3DSG predicts open-vocabulary object classes and open-set relationships from point clouds, producing an explicit 3D scene graph over object instances (Koch et al., 2024). FreeQ-Graph similarly builds an explicit graph

$\mathbf{G}=(\mathbf{V},\mathbf{E}),$

but emphasizes semantic consistency through superpoint-derived 3D alignment and an LLM-based two-stage reasoning procedure for free-form queries (Zhan et al., 16 Jun 2025). A particularly strong operational version appears in the database-backed 3DSG work, where scene graphs are stored in Neo4j as property graphs and exposed to an LLM through Cypher queries (Ray et al., 18 Oct 2025).

Multimodal VLM-native representations adopt a different strategy: instead of forcing a VLM to decode an opaque 3D latent, they convert the scene into modalities already native to the model. HMR3D serializes scenes into object-coordinate text and five rendered views, then augments these with hierarchical patch-, view-, and scene-level tokens (Li et al., 28 Nov 2025). PAR3D makes the representation part-aware by enriching superpoint features with object-part supervision and grounding language through hierarchical $[{\rm OBJ}]$ and $S$0 tokens (Dai et al., 4 Jun 2026). POMA-3D uses point maps, dense $S$1 grids of world-space coordinates, as a geometry-preserving but 2D-model-compatible scene substrate (Mao et al., 20 Nov 2025). The multimodal 3D QSR framework for robotics makes the object itself the cross-modal anchor that links panoptic radiance fields, segmented point clouds, captions, and scene-graph nodes (Li et al., 24 Sep 2025).

3. Query interfaces and mathematical mechanisms

The query mechanism depends directly on the chosen representation.

For coordinate-queryable fields, the interface is literal evaluation at a spatial point. In SRNs, the scene is queried by computing $S$2 for any $S$3, and a differentiable ray-marching renderer uses those local features to synthesize images (Sitzmann et al., 2019). QueryOcc extends this idea to spatio-temporal semantic occupancy. Given current images $S$4 and a 4D query $S$5, it predicts

$S$6

where $S$7 is occupancy probability and $S$8 is the semantic class distribution conditioned on occupancy (Lilja et al., 21 Nov 2025).

QuadricFormer makes queryability explicit through a sparse primitive mixture. A pointwise occupancy query is answered by a noisy-or composition,

$S$9

and semantic probability is obtained by occupancy-weighted aggregation of primitive semantics,

$Q$0

This yields continuous occupancy and semantic fields induced by a sparse set of superquadrics rather than a dense 3D tensor (Zuo et al., 12 Jun 2025).

Open-vocabulary semantic field methods replace explicit symbolic queries with embedding-space retrieval. Open-Fusion stores only semantic keys and confidence values in voxels, while the actual embeddings are maintained in a dictionary. A query vector $Q$1, typically obtained from a text encoder, is compared to the stored embeddings by cosine similarity, and the corresponding 3D region is recovered through the voxel keys and TSDF geometry (Yamazaki et al., 2023). Query3D uses a similar principle but renders a language feature map from the Gaussian scene representation and scores it against CLIP text embeddings; its contribution is to replace fixed canonical phrases with LLM-generated main positives, helping positives, and canonical negatives (Chahe et al., 2024).

Graph-based systems support more structured query operators. In the Cypher-backed 3DSG framework, the graph has labeled ontology layers such as Object, MeshPlace, Place, and Room, typed edges such as CONTAINS, and support for transitive containment through patterns such as $Q$6 This allows aggregation, multi-hop containment reasoning, geometric filtering with point.distance, connectivity queries, and compositional references such as “the box closest to the bicycle” or “poles within 5 meters of a fence” (Ray et al., 18 Oct 2025). Open3DSG and FreeQ-Graph do not expose a formal query language, but they still support graph-structured retrieval because nodes have captions, boxes, IDs, and relation labels, and edges encode inter-object structure (Koch et al., 2024, Zhan et al., 16 Jun 2025).

In VLM-native systems, the scene is queried by language over a persistent multimodal package. HMR3D concatenates object-coordinate text, five scene views, and hierarchical view/scene tokens, then answers the question autoregressively (Li et al., 28 Nov 2025). PAR3D goes further by separating object-level from part-level grounding. Object references trigger $Q$2, while part references trigger $Q$3 followed by $Q$4, and the corresponding hidden states are converted into 3D segmentation queries (Dai et al., 4 Jun 2026). This turns the query interface itself into a hierarchy aligned with object-part structure.

4. Construction, supervision, and alignment

A central theme across 3D QSR research is that queryability depends as much on supervision and alignment strategy as on the representation format itself.

One line of work learns a queryable representation from geometric image formation. SRNs require only posed RGB images and camera parameters; geometry emerges because the coordinate field must support differentiable ray-marching and novel-view synthesis (Sitzmann et al., 2019). QueryOcc rejects both dense voxel supervision and purely rendering-based self-supervision, instead learning from independent positive and negative 4D spatio-temporal point queries sampled from pseudo point clouds or lidar. Its main claim is that direct 4D query supervision yields stronger self-supervised occupancy learning than rendering consistency alone (Lilja et al., 21 Nov 2025).

Another line relies on sparse explicit scene elements but trains them through image-conditioned refinement. QuadricFormer starts from a random 3D initialization of superquadrics, refines them through quadric-encoder blocks with sparse convolution and deformable attention, and then reallocates capacity with a pruning-and-splitting module that removes small-scale primitives in empty regions and splits large ones in occupied regions (Zuo et al., 12 Jun 2025). Open-Fusion uses a region-level vision-LLM, SEEM, and fuses its confidence maps and embeddings into a TSDF through rendered-region matching and a modified Jonker-Volgenant assignment procedure (Yamazaki et al., 2023).

Language alignment is equally varied. Model2Scene aligns pointwise CAD-derived features with static class-name embeddings and uses Deep Convex-hull Regularization to reduce the gap between CAD models and real scene scans, yielding a dense 3D representation whose points can be semantically addressed by language anchors (Chen et al., 2023). POMA-3D aligns point maps to images, view captions, and scene captions, and supplements this with POMA-JEPA, a joint embedding-predictive objective that enforces multi-view geometric consistency across masked point-map patches (Mao et al., 20 Nov 2025). HMR3D avoids latent-space projection as the primary mechanism and instead aligns to the VLM “at the input space” by providing object-coordinate text plus carefully chosen multiview renderings (Li et al., 28 Nov 2025).

Several papers argue that language supervision alone is insufficient for robust 3D queryability. The 3DRS work measures 3D-awareness through multi-view correspondence and reports a strong positive correlation between correspondence quality and downstream task performance. It then aligns final MLLM visual features to features from pretrained 3D foundation models such as FLARE and VGGT, explicitly supervising the internal scene representation rather than only the answer text (Huang et al., 2 Jun 2025). This is one of the clearest statements in the literature that better language outputs do not by themselves guarantee a good 3D query substrate.

Graph construction papers make a related point in symbolic form. FreeQ-Graph argues that free-form scene graphs generated only by LLMs or LVLMs can be semantically inconsistent with actual 3D regions, and therefore aligns graph nodes with 3D semantic features derived from merged superpoints (Zhan et al., 16 Jun 2025). The Cypher-backed 3DSG work reaches the same conclusion from a systems perspective: once the graph is large enough, scalability depends not on longer prompts but on making the scene representation explicitly retrievable through a tool interface and a schema that the model can use correctly (Ray et al., 18 Oct 2025).

5. Empirical evidence and application domains

The empirical record shows that 3D QSR is not a niche formulation tied to one benchmark. It appears in autonomous driving, robotics, indoor scene understanding, embodied QA, and part-aware grounding.

System	Reported result	Implication
QuadricFormer	31.22 IoU and 20.12 mIoU on nuScenes; 1600 primitives at 162 ms and 2554 MB	Sparse superquadrics can outperform Gaussian primitives with fewer scene elements (Zuo et al., 12 Jun 2025)
Cypher-backed 3DSGs	Large-graph QA 0.77 vs 0.33 for context-window baseline; 582,202 input tokens vs 2,395 plus tool-return tokens on large-graph PDDL	Database-backed graph querying scales far better than prompt serialization (Ray et al., 18 Oct 2025)
Query3D	Accuracy 0.92, Precision 0.58, mIoU 0.37, mAP 0.54 vs 0.65, 0.33, 0.23, 0.49 for predefined canonical phrases	LLM-mediated query expansion improves language-conditioned 3D Gaussian retrieval (Chahe et al., 2024)
Open-Fusion	50 FPS geometry-only, 4.5 FPS semantic reconstruction, mAcc 0.62, f-mIoU 0.59 on ScanNet	Real-time open-vocabulary 3D mapping is feasible without additional 3D training (Yamazaki et al., 2023)
QueSTMaps	Matterport3D room AP 88.47, transition AP 60.92; complete pipeline 74.02 [email protected] IoU	Room-level semantic-topological querying is practical and robust in cluttered indoor scans (Mehan et al., 2024)
HMR3D	62.1 EM@1 on SQA3D with ground-truth location; 44.3 BLEU-1, 13.5 BLEU-4, 43.9 ROUGE, 17.8 METEOR, 89.0 CIDEr on ScanQA	Input-space multimodal packaging supports strong situated and general 3D QA (Li et al., 28 Nov 2025)
PAR3D	ScenePart-Seg overall mIoU 60.7 vs 51.8 for the ScenePart-trained 3D-LLaVA baseline	Part-aware latent structure and hierarchical grounding improve fine-grained scene querying (Dai et al., 4 Jun 2026)

The robotics-oriented graph database work is particularly strong evidence for 3D QSR as an operational systems pattern rather than a descriptive label. It evaluates instruction grounding to PDDL goals and scene QA, shows that Cypher tool use scales substantially better than scene-graph serialization or a Python API baseline, and demonstrates a Boston Dynamics Spot deployment in which natural-language corrections update the graph and regenerate the grounded plan (Ray et al., 18 Oct 2025).

Autonomous driving contributes a different form of evidence. QuadricFormer shows that a sparse primitive set can define a continuous occupancy and semantic field while remaining efficient at inference (Zuo et al., 12 Jun 2025). QueryOcc shows that self-supervised occupancy need not be limited to dense voxel lattices: it reaches 23.6 semantic RayIoU and 45.2 occupancy RayIoU, while QueryOcc+ reaches 25.8 and 47.4 respectively, and the method runs at 11.6 FPS (Lilja et al., 21 Nov 2025). Query3D shows that even when the underlying representation is a language-embedded 3D Gaussian scene, practical queryability depends strongly on how the language prompt is expanded and disambiguated (Chahe et al., 2024).

The indoor robotics and embodied reasoning literature adds still another layer. The multimodal 3D QSR framework links a panoptic radiance field, segmented point cloud, and 3D scene graph through shared object identity, then uses representation-specific querying for planning. In its reported averages, point clouds are strongest on descriptive precision and recall, while the scene graph is strongest on affordance and negation queries; a two-step scene-graph-first strategy improves affordance queries by 7% and negation queries by 50% (Li et al., 24 Sep 2025). A related systems paper studies online maintenance rather than querying directly and shows that timeliness-fidelity scheduling matters for real-time 3D scene representation; its $Q$5-wait policy reached PSNR 30.05, SSIM 0.793, and LPIPS 0.248 in the reported setting (Xu et al., 21 Sep 2025). This suggests that queryability in deployed settings depends not only on the stored representation but also on how fresh and how complete the representation remains under communication constraints.

6. Limitations, misconceptions, and open directions

A persistent misconception is that any language-conditioned 3D model is automatically a full 3D QSR. The literature is more precise. QuadricFormer is queryable because occupancy and semantics can be evaluated at arbitrary 3D points, but the paper explicitly states that it is not a full scene-graph or relational QSR and does not provide object IDs or a query API beyond the induced continuous field (Zuo et al., 12 Jun 2025). Query3D similarly supports open-vocabulary text-conditioned relevance maps, but the output remains segmentation or retrieval through rendered language features rather than symbolic decomposition or formal relational querying (Chahe et al., 2024).

A second misconception is that scene graphs by themselves solve queryability. Explicit graphs do make semantics and relations addressable, but their usefulness depends on graph quality, schema design, and interface competence. Open3DSG shows that open-vocabulary object nodes and open-set relations can be distilled from point clouds without scene-graph labels, yet relation prediction remains weaker than strong supervised closed-set systems and can suffer from hallucinated or inconsistent edge descriptions (Koch et al., 2024). FreeQ-Graph improves graph completeness and semantic consistency through superpoint-based alignment, but still depends on multiple pretrained LLM and LVLM components and remains discrete and object-centric rather than continuously geometric (Zhan et al., 16 Jun 2025). The Cypher-backed 3DSG paper further notes that even with a formal interface, models can struggle with ontology changes or fail to use transitive containment syntax correctly (Ray et al., 18 Oct 2025).

A third issue is scope. Many systems are static, room-scale, or domain-specific. QueSTMaps is indoor-only and room-centric (Mehan et al., 2024). POMA-3D is room-scale and uses only geometric inputs, which constrains appearance-dependent querying (Mao et al., 20 Nov 2025). The multimodal 3D QSR framework assumes a static environment and explicitly identifies dynamic-scene updates as future work (Li et al., 24 Sep 2025). QueryOcc introduces 4D spatio-temporal queries, but its inference still conditions on current images rather than maintaining a persistent dynamic world memory (Lilja et al., 21 Nov 2025).

The representation-learning literature adds a deeper warning: answer quality can mask representational weakness. The 3DRS study shows that MLLMs need explicit 3D-aware representation supervision and that multi-view correspondence quality is strongly correlated with downstream grounding, captioning, and QA performance (Huang et al., 2 Jun 2025). This suggests that future 3D QSR systems will benefit from explicit internal representation objectives rather than relying only on end-task language loss.

A plausible direction, implied jointly by current results, is a hybrid architecture that combines continuous geometry-aware fields, explicit object- or room-level structure, language-aligned embeddings, and dynamic update mechanisms. The survey literature already points toward such a layered design: geometric substrates for localization, persistent object instances for grounding, relational graphs for compositional reasoning, language-aligned embeddings for open-vocabulary access, and embodied state or memory for situated and temporal queries (Li et al., 1 Feb 2025). The main unresolved problem is not whether 3D scenes can be queried, but how to integrate these layers so that queryability remains scalable, interpretable, temporally consistent, and useful for downstream action.