Open3DSG: Open-Vocabulary 3D Scene Graphs

• Open3DSG is an open-vocabulary 3D scene graph method that represents point cloud scenes as nodes with free-form language-generated edges.
• It features an end-to-end pipeline combining 2D and 3D streams using PointNet backbones, GNN message passing, and multimodal feature distillation.
• The system achieves competitive zero-shot performance on benchmarks by effectively modeling long-tail object and predicate distributions.

Open3DSG is an open-vocabulary 3D scene graph method for point clouds that predicts queryable object nodes and open-set inter-object relationships without requiring labeled 3D scene graph data. It co-embeds features from a 3D scene graph prediction backbone with the feature space of open-world 2D vision-language foundation models, enabling zero-shot object querying from an arbitrary text vocabulary and relationship generation through a grounded LLM. In the original formulation, a 3D scene graph is represented as $G=(V,E)$, with object instances as nodes and relationships as edges, and the method is explicitly designed to express rare as well as specific objects and relationships in a predicted 3D scene graph (Koch et al., 2024).

1. Problem formulation and representational scope

Open3DSG defines a 3D scene graph as a relational representation of a 3D scene, with $V=\{o_i\}$ denoting object nodes and $E=\{(o_i,o_j)\}$ denoting edges that carry relationship text and associated attributes (Koch et al., 2024). Each node is associated with geometry and instance-mask-derived descriptors such as a bounding box $B_i$, centroid, size, and language-aligned embeddings $f_i \in \mathbb{R}^d$ distilled from 2D vision-LLMs. Each edge carries relationship text generated in free-form natural language and can later be mapped to fixed taxonomies for evaluation.

The method is built around two claims. First, object recognition should be open-vocabulary: object classes are queried zero-shot from arbitrary text prompts rather than selected from a fixed taxonomy. Second, relationships should be open-set: predicates are generated as free-form text rather than chosen from a predefined label set. The motivation is that earlier 3D scene graph systems trained on labeled datasets with fixed object and relationship vocabularies are bound to those label sets, while contrastive VLMs such as CLIP are strong at object recognition but weak at compositional relational understanding (Koch et al., 2024).

Within later literature, Open3DSG is repeatedly characterized as an object-centric, spatially oriented baseline. OpenFunGraph describes it as relying on “object-level CLIP features” and “spatial-proximity edge feature distillation,” with graph nodes restricted to objects and edges representing spatial relationships (Zhang et al., 24 Mar 2025). OP3DSG similarly positions it as an object-centric open-vocabulary 3DSG approach from point clouds that focuses on spatial graphs without explicit modeling of interactive parts, functional relations, or affordances (Kim et al., 29 Jun 2026). These later characterizations do not alter the original definition, but they establish the representational boundary that subsequent systems sought to extend.

2. End-to-end pipeline

The Open3DSG pipeline begins from a 3D point cloud $P$ with class-agnostic instance masks $M=\{M_i\}$, optionally augmented by posed RGB-D frames with intrinsics and extrinsics (Koch et al., 2024). For each instance $i$, the system extracts an instance-specific point cloud $P_i$; for each pair $(i,j)$, it extracts $V=\{o_i\}$0 from the union of the bounding boxes $V=\{o_i\}$1. Shared PointNet backbones produce node features $V=\{o_i\}$2 and edge features $V=\{o_i\}$3, and triplets $V=\{o_i\}$4 are refined through a GNN with message passing.

A second stage constructs multi-view 2D distillation targets. For objects, OpenSeg pixel embeddings are averaged over projected pixels in selected views and then averaged across top-$V=\{o_i\}$5 frames to produce $V=\{o_i\}$6. For relationships, the system crops the union of per-frame boxes at multiple scales, encodes them with the InstructBLIP vision encoder, and averages these features across scales and frames to obtain $V=\{o_i\}$7 (Koch et al., 2024). Frame selection is not arbitrary: projected points are filtered by visibility and occlusion checks, and top-$V=\{o_i\}$8 views are retained when visibility or projected box area satisfies threshold criteria.

Inference combines the 2D and 3D streams. The fused object feature and relation feature are defined by average pooling,

$V=\{o_i\}$9

Object classification is then performed by comparing $E=\{(o_i,o_j)\}$0 to arbitrary text queries encoded with the CLIP text encoder. Relationship prediction is performed by feeding $E=\{(o_i,o_j)\}$1 together with predicted subject and object labels into the InstructBLIP Qformer and Vicuna 7B, prompted with the template “Describe the relationship between [object1] and [object2]?” The resulting graph is explicit: nodes carry zero-shot labels, and edges carry generated relationship text (Koch et al., 2024).

The resulting computational structure is fully connected over instances, yielding approximately $E=\{(o_i,o_j)\}$2 candidate edges. The original presentation notes that spatial heuristics such as distance thresholds can be used to reduce compute, but the defining formulation is dense pairwise relation prediction from point-cloud instances and distilled multimodal features.

3. Core technical components

The 3D backbone uses two shared PointNets that produce 1024-dimensional initial features for nodes and edges, followed by a message-passing GNN with 5 graph convolution layers and hidden dimension 2048 (Koch et al., 2024). The node projection head is a 5-layer MLP with ReLU and batch normalization that maps to a 768-dimensional CLIP/OpenSeg-aligned space. The edge projection head is a 5-layer transformer that concatenates BLIP-ViT positional encodings and maps to a 1408-dimensional BLIP-aligned space.

The central learning objective is cosine-alignment distillation:

$E=\{(o_i,o_j)\}$3

with

$E=\{(o_i,o_j)\}$4

The object-querying score is

$E=\{(o_i,o_j)\}$5

where $E=\{(o_i,o_j)\}$6 is the CLIP text embedding of an arbitrary query. Top-$E=\{(o_i,o_j)\}$7 labels are then selected per node using cosine similarity as confidence (Koch et al., 2024).

The relationship module addresses the compositional weakness of contrastive VLMs by replacing direct predicate classification with grounded language generation. Rather than asking CLIP-like models to discriminate predicates, Open3DSG feeds scene-conditioned edge features to the Qformer and LLM, with object labels supplied as contextual tokens. For closed-set evaluation, the free-form relationship text is embedded with BERT-base-uncased and mapped to target predicate labels by cosine similarity. This design allows the model to express relationships such as “tv mounted on wall” or “microwave built into kitchen cabinet,” while still reporting benchmark Recall@K on fixed taxonomies (Koch et al., 2024).

The 2D projection and view-selection machinery is also explicit. A 3D point is projected as

$E=\{(o_i,o_j)\}$8

and visibility is scored by

$E=\{(o_i,o_j)\}$9

A frame is retained when $B_i$0 or the projected area satisfies $B_i$1, with occlusion filtering based on $B_i$2 (Koch et al., 2024).

4. Training, evaluation, and empirical profile

Open3DSG is trained without labeled 3D scene graph supervision. The supervisory signals are distillation targets from OpenSeg for objects and InstructBLIP image features for relationships, aggregated over selected multi-view observations (Koch et al., 2024). The reported training setup uses 50 epochs, Adam with weight decay, an initial learning rate of $B_i$3, cosine annealing, batch size 6, mixed precision, and a single NVIDIA A100 GPU. Distillation is performed on ScanNet, while evaluation is reported on 3DSSG, the benchmark derived from 3RScan.

On 3DSSG closed-vocabulary evaluation, Open3DSG reports objects $B_i$4, $B_i$5; predicates $B_i$6, $B_i$7; and relationships $B_i$8, $B_i$9 (Koch et al., 2024). The reported zero-shot baselines are substantially weaker: naive CLIP gives object $f_i \in \mathbb{R}^d$0, predicate $f_i \in \mathbb{R}^d$1, and triplet $f_i \in \mathbb{R}^d$2, while OpenSeg + BLIP captions improves predicates but remains below Open3DSG on triplets. The paper emphasizes that Open3DSG approaches older supervised 3DSSG performance on triplets despite operating in a zero-shot regime.

Ablations separate the roles of 2D and 3D signals. The 2D-only configuration yields object $f_i \in \mathbb{R}^d$3 and predicate $f_i \in \mathbb{R}^d$4; the 3D-only configuration yields object $f_i \in \mathbb{R}^d$5 and predicate $f_i \in \mathbb{R}^d$6; and the 2D–3D ensemble yields object $f_i \in \mathbb{R}^d$7 and predicate $f_i \in \mathbb{R}^d$8 (Koch et al., 2024). Replacing OpenSeg with CLIP for node features reduces performance, and supervised relationship fine-tuning over 27 predicates raises predicate $f_i \in \mathbb{R}^d$9 to $P$0, indicating that the distillation objective serves as strong pretraining.

The paper also reports unusually strong long-tail behavior. For objects at $P$1, Open3DSG gives head $P$2, body $P$3, tail $P$4, all $P$5, compared with the cited fully supervised baseline’s head $P$6, body $P$7, tail $P$8, all $P$9. For predicates at $M=\{M_i\}$0, Open3DSG gives head $M=\{M_i\}$1, body $M=\{M_i\}$2, tail $M=\{M_i\}$3, all $M=\{M_i\}$4 (Koch et al., 2024). The same study reports semantic examples such as “kitchen cabinet” at $M=\{M_i\}$5, “dining chair” at $M=\{M_i\}$6, and predicates including “standing on” at $M=\{M_i\}$7 and “belonging to” at $M=\{M_i\}$8.

5. Subsequent reinterpretations and extensions

Later work treats Open3DSG as a reference architecture for open-vocabulary 3D scene understanding, but also as a baseline whose assumptions are too narrow for functional or part-aware reasoning. In “Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spaces,” OpenFunGraph uses two adapted Open3DSG baselines: Open3DSG* modifies the original LLM prompts to output functional relationships instead of spatial ones, and Open3DSG† uses OpenFunGraph’s fused 3D nodes rather than ground-truth segmentation masks (Zhang et al., 24 Mar 2025). On SceneFun3D, Open3DSG* reports overall node Recall@3/Recall@10 of $M=\{M_i\}$9 and overall triplet Recall@5/Recall@10 of $i$0; on FunGraph3D these fall to $i$1 for nodes and $i$2 for triplets. OpenFunGraph interprets these gaps as evidence that spatially oriented, object-only scene graphs do not transfer directly to interactive elements and functional relations.

RAG-3DSG places Open3DSG in a different extension path: open-vocabulary 3D Scene Graph generation from multi-view RGB-D, with emphasis on re-shot guided uncertainty estimation, object-level retrieval-augmented generation, and dynamic downsample-mapping (Chang et al., 15 Jan 2026). Its comparison section states that Open3DSG emphasizes embedding-first alignment for open-set relationships and queryable objects, whereas RAG-3DSG uses caption-first refinement with re-shot VLM/LLM guidance and object-level RAG. It explicitly notes that “Open3DSG” can refer either to the general task of open-vocabulary 3DSGs or to the specific system by Koch et al. (CVPR 2024).

OP3DSG extends the critique further by arguing that object-centric open-vocabulary 3DSGs encode limited relational information and miss small, thin, or interaction-critical parts such as buttons, handles, and knobs (Kim et al., 29 Jun 2026). OP3DSG therefore introduces Unified 3DSGs with object nodes, interactive part nodes, spatial relations, functional relations, and affordances. Its relation section states that Open3DSG focuses on spatial graphs without explicit modeling of interactive parts, functional relations, or affordances. This suggests that Open3DSG established the open-vocabulary scene-graph template, while later work differentiated itself by expanding node types, edge semantics, and reasoning scope.

6. Limitations, misconceptions, and naming ambiguities

The original Open3DSG paper identifies several limitations. Relationship prediction remains challenging because LLMs can hallucinate and may misinterpret spatial nuances without explicit geometric constraints; predicate diversity is lower than in supervised methods; and closed-set metrics such as Recall@K do not fully capture open-set expressivity (Koch et al., 2024). Reported failure cases include hallucinated statements such as “keyboard connected via USB to monitor” and geometric inconsistencies such as two monitors both being described as “left of” each other. The authors also note that ScanNet-to-3DSSG differences in field of view create a domain gap.

A recurrent misconception concerns the assumption set under evaluation. Later work argues that Open3DSG evaluation assumes known ground-truth nodes and focuses on spatial predicates, which motivated alternative protocols that include node detection and open-vocabulary retrieval in real-world scenes (Zhang et al., 24 Mar 2025). This does not negate the original contribution, but it marks an important distinction between zero-shot relationship prediction on structured 3D instances and end-to-end graph construction from raw RGB-D streams.

A second misconception is terminological. Open3DSG should not be conflated with Open3DGS, which refers to open-source 3D Gaussian Splatting implementations for radiance-field rendering rather than scene graphs (Lin, 2024). Nor should it be conflated with later Gaussian-splatting papers that use the term only analogically. X-GS, for example, is described as functioning as an “Open3DSG” in spirit and capability because it is an open, extensible 3DGS framework that unifies online SLAM, semantics, and multimodal models, but it is a 3D Gaussian Splatting system rather than the point-cloud scene-graph method introduced by Koch et al. (Ma et al., 10 Mar 2026). Similarly, “3D Skew Gaussian Splatting” explicitly states that it does not use the name “Open3DSG”; its official method name is 3DSGS (Zhao et al., 18 May 2026). A plausible implication is that the abbreviation has become a locus of cross-domain naming overlap between scene-graph and Gaussian-splatting literatures.

In retrospective terms, Open3DSG occupies a specific historical position: it is the first 3D point cloud method to predict explicit open-vocabulary object classes together with open-set relationships from point clouds in a zero-shot manner, using 3D-to-2D feature distillation for nodes and grounded language generation for edges (Koch et al., 2024). Later systems repeatedly use it as a baseline, modify its prompting and node assumptions, or define themselves against its object-centric spatial scope. Its lasting significance lies less in a single benchmark score than in establishing open-vocabulary 3D scene graph prediction as a distinct problem formulation with queryable objects, grounded relationship generation, and explicit separation between object classification and relational reasoning.