Canonical 3D Scene Graph Construction

• Canonical 3D scene graph construction is a method that embeds object semantics, spatial relations, and sensor attributes into a unified environmental representation.
• The pipeline combines sensor preprocessing, keyframe extraction, 2D detection, and canonical merging to ensure minimal, robust graph structures.
• Evaluation protocols focus on node accuracy, efficient graph topology, and runtime performance to validate its utility in robotics and computer vision.

Canonical 3D scene graph construction is a foundational methodology for representing physical environments in a graph-structured format, integrating object-level semantics, spatial relationships, and, in some frameworks, hierarchical or multi-modal attributes. This scene-graph formalism underpins a broad spectrum of tasks in robotics, computer vision, and spatial reasoning, facilitating compact, semantically rich, and machine-queryable environmental models.

1. Formal Definition and Representational Schemes

Let $G = (V, E)$ denote a 3D scene graph, where $V = \{v_1, ..., v_N\}$ is a set of nodes, each node representing a semantic entity such as an object, place, agent, or primitive segment. Edges $E \subseteq V \times V$ represent directed or undirected binary relations between node pairs, such as spatial (“on,” “behind”), semantic (“contains”), or action (“jumping_over”). Node and edge attributes are formally specified as $A(v)$ (such as semantic label sets, positional distributions, color histograms, deep features) for nodes and $R(e)$ (relation type, confidence, geometric parameters) for edges (Kim et al., 2019, Kassab et al., 2024).

Several representational variants are found across the literature:

• Entity graphs with merged semantic-object nodes and explicitly labeled relations (Kim et al., 2019, Kassab et al., 2024, Zhan et al., 16 Jun 2025);
• Multi-level or hierarchical graphs, where nodes encode objects, rooms, places, and top-level spatial divisions (Hughes et al., 2022, Armeni et al., 2019);
• Support-relation DAGs where directed acyclic graphs encode “who supports whom” (Ma et al., 2024).

Canonicality in this context refers to representations that are consistent, minimal (i.e., do not contain duplicate entities or edges), and suitable for downstream high-level reasoning and query tasks (Kim et al., 2019, Zhan et al., 16 Jun 2025).

2. Canonical Construction Pipelines

Pipeline architectures for canonical 3D scene graph construction integrate sensor data ingestion, semantic and geometric parsing, relational inference, and graph canonicalization. A representative modular pipeline (per (Kim et al., 2019)) includes:

• Sensor Data Preprocessing: Acquisition and resizing of RGB–D frames at ~30 Hz. Adaptive blurry-image rejection (ABIR) is applied via Laplacian variance scoring with an exponential moving average and adaptive thresholds, increasing reliability in dynamic or low-texture scenes.
• Keyframe Group Extraction: Keyframes and anchors are partitioned via frame-to-keyframe overlap metrics in projected depth/camera space. Grouping reduces redundant computations and supports efficient sequential merging.
• 2D Recognition and Relation Extraction: Object detection and classification is performed using a region-proposal network (e.g., Faster-RCNN), yielding semantic label sets and bounding boxes. Relation extraction networks (e.g., Factorizable Net) propose triplet relations with associated confidences.
• Spurious Detection Rejection (SDR): Post-detection filtering (e.g., non-maximum suppression, semantic irrelevance culling, moving-object exclusion) is applied. Spatial relations are further filtered using a learned model over 2D distances and relation dictionaries (e.g., from Visual Genome), rejecting relationships that fall below probabilistic thresholds.
• Attribute Lifting and Computation: Remaining 2D nodes/edges are “lifted” into 3D via depth back-projection and known pose, yielding attributes such as Gaussian estimates of 3D position, color histograms, or deep descriptors.
• Graph Merging / Canonicalization: To maintain a single, unified, canonical scene graph, temporally or spatially local graphs are iteratively merged. Node matching employs combined similarity metrics over labels, positions, and color features. Merged attributes (e.g., running means for pose and color histograms) contribute to temporal consistency.

This modular schema is reflected, with adaptation, in contemporary open-vocabulary pipelines (Kassab et al., 2024), indoor and outdoor SLAM-augmented frameworks (Hughes et al., 2022, Samuelson et al., 23 Sep 2025), and language-grounded approaches (Zhan et al., 16 Jun 2025).

3. Node, Edge, and Attribute Computation

Node and edge attributes encode the relevant information for downstream tasks and efficient graph manipulation:

• Node Attributes often include unique IDs, sets of semantic labels with confidences, 3D pose parameters given as Gaussian distributions (means and diagonal covariances), color/appearance histograms (in HSV or RGB space), as well as deep feature representations (e.g., CLIP embeddings, ViT feature codes) (Kim et al., 2019, Kassab et al., 2024, Zhan et al., 16 Jun 2025).
• Edge Attributes specify relation types (spatial, semantic, action, support), confidence scores from the underlying relation network, and geometric properties such as 3D distances or angles (Kim et al., 2019, Ma et al., 2024).

Modern systems may leverage free-form, natural language labels and semantic embeddings for nodes and edges, especially in LLM/LVLM-driven pipelines (Zhan et al., 16 Jun 2025), and may further align these features via associative clustering in feature and spatial space (Zhan et al., 16 Jun 2025).

Specialized frameworks compute hierarchical relations: support DAGs via combinatorial primitive classification and adjacency patterns (Ma et al., 2024), or containment/adjoinment/compositional edges via hierarchical parsing (Armeni et al., 2019, Hughes et al., 2022, Samuelson et al., 23 Sep 2025).

4. Canonical Graph Inference and Consistency

Canonical inference seeks to remove duplicates, enforce consistent node identities, and guarantee that spurious or missing entities are minimized. Graph merging employs combined similarity metrics weighted over semantic label overlap, positional proximity (normalized Mahalanobis distance), and appearance/color histogram intersection. A merging threshold $\tau_{merge}$ controls whether a candidate is assimilated or added as a new node. Attribute statistics are incrementally updated upon merge (e.g., count-weighted averaging of Gaussians for 3D position; histogram summation for color/appearance) (Kim et al., 2019).

Adaptive data association and statistical filtering are crucial for scalability and accuracy. In region-based or hierarchical graphs, spatial and semantic clustering methods (e.g., agglomerative or spectral clustering over combined geometric and appearance distances) yield multiscale region and place nodes (Samuelson et al., 23 Sep 2025). In language-driven systems, post-hoc feature alignment via spectral clustering of superpoints and CLIP embeddings ensures semantic consistency and vocabulary independence (Zhan et al., 16 Jun 2025).

Graph pruning and sublinearity in runtime complexity are enforced via thresholds on merge affinity, maximum embedding counts per point, and minimum spatial extent for regions or objects (Samuelson et al., 23 Sep 2025).

5. Evaluation Protocols and Empirical Results

Canonically constructed 3D scene graphs are evaluated on several metrics:

• Entity Accuracy: Counts of spurious (hallucinated) and missing nodes/edges, either by human annotation or via ground-truth alignment (e.g., in ScanNet or Replica datasets) (Kim et al., 2019, Kassab et al., 2024).
• Graph-based Metrics: Mean intersection-over-union (mIoU), size-weighted F-mIoU, mean accuracy (mAcc) for instance segmentation; recall@k or F1 scores for node and relation detection; adjacency/spectral section errors for graph topology (Kassab et al., 2024, Ma et al., 2024).
• Efficiency: Runtime per frame, memory usage (VRAM and growth rate with object/relation count), and scalability with respect to input scene size (Kim et al., 2019, Kassab et al., 2024).
• Downstream Task Performance: Success in VQA, task-planning, region monitoring, or navigation queries (e.g., path planning over the region or place graph, free-form question answering) (Kim et al., 2019, Zhan et al., 16 Jun 2025, Samuelson et al., 23 Sep 2025).
• Comparative Benchmarks: Canonical pipelines (inclusive of keyframe grouping, blur rejection, and spurious filtering) demonstrate marked improvements in compactness and accuracy (e.g., 3 spurious / 3 missing entities at 0.38 s/frame in (Kim et al., 2019)), and threefold computation speedups over state-of-the-art classification accuracy (Kassab et al., 2024).

These protocols validate both the semantic and operational advantages of canonical 3D scene graph representations in intelligent agent environments.

6. Extensions: Hierarchy, Scalability, and Open-Vocabulary Support

Advanced scene-graph frameworks expand canonical construction along several axes:

• Hierarchy: Multi-level graph schemas with place, region, room, object, and building nodes enable scalable, modular reasoning at various abstraction levels (Hughes et al., 2022, Armeni et al., 2019, Samuelson et al., 23 Sep 2025). Room and region detection use community detection, modularity maximization, or clustering with semantic and geometric weights.
• Semantic Versatility: FreeQ-Graph and similar methods employ LLMs and LVLMs to induce scene graphs with open-vocabulary, natural-language labels for both nodes and edge relations. Semantic alignment between visual features and language is achieved via clustering and mean-pooling fusion in feature space (Zhan et al., 16 Jun 2025).
• Outdoor and Terrain-aware Approaches: Canonical 3DSG construction methodologies are adapted for large-scale, terrain-aware mapping. Place nodes encode traversable surfaces, regions encode hierarchical semantic clusters, and connectivity encodes practical navigational and object-retrieval constraints (Samuelson et al., 23 Sep 2025).
• Real-Time and Monocular Adaptations: Canonical pipelines are realized for real-time operation on embedded hardware or with monocular/IMU inputs, leveraging efficient visual-inertial odometry, depth prediction, and segmentation nets, with adaptive allocation of computation across GPU/CPU resources (Udugama et al., 2023, Hughes et al., 2022).

These extensions maintain canonicality by preserving consistency, compactness, and semantic completeness, demonstrating scalability to varied environments and input sources.

7. Summary Table: Canonical Pipeline Components

Pipeline Stage	Function/Metric	Canonicality Role
Preprocessing	Blur rejection, keyframe grouping, resizing	Ensures data quality/efficiency
Recognition	2D object detection, relation extraction	Populates initial graph nodes/edges
Attribute Computation	3D lifting, color/feature/semantic embedding	Embeds rich, queryable metadata
Spurious Filtering	Confidence/NMS, external dictionaries, thresholds	Prunes noise, supports accuracy
Canonical Merge-Inference	Node/edge matching, attribute update	Guarantees unified, minimal graph
Hierarchical Construction	Multiscale place/region detection, community clustering	Enables scalability/multiresolution
Semantic Alignment	Feature fusion, LLM/LVLM-guided label assignment	Maintains open-vocabulary capability
Evaluation	Entity recall/precision, graph topology, runtime	Validates utility and correctness

• "3-D Scene Graph: A Sparse and Semantic Representation of Physical Environments for Intelligent Agents" (Kim et al., 2019)
• "The Bare Necessities: Designing Simple, Effective Open-Vocabulary Scene Graphs" (Kassab et al., 2024)
• "Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysis" (Zhang et al., 2021)
• "On Support Relations Inference and Scene Hierarchy Graph Construction from Point Cloud in Clustered Environments" (Ma et al., 2024)
• "FreeQ-Graph: Free-form Querying with Semantic Consistent Scene Graph for 3D Scene Understanding" (Zhan et al., 16 Jun 2025)
• "Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization" (Hughes et al., 2022)
• "Mono-hydra: Real-time 3D scene graph construction from monocular camera input with IMU" (Udugama et al., 2023)
• "Terra: Hierarchical Terrain-Aware 3D Scene Graph for Task-Agnostic Outdoor Mapping" (Samuelson et al., 23 Sep 2025)
• "3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera" (Armeni et al., 2019)