---
title: 3D Dynamic Scene Graphs
url: https://www.emergentmind.com/topics/3d-dynamic-scene-graphs-dsgs
type: topic
---

# 3D Dynamic Scene Graphs

A 3D Dynamic Scene Graph (DSG) is a layered, attributed graph-based representation that jointly encodes the geometric, semantic, topological, and temporal structure of a physical environment, including both static and dynamic entities such as movable objects, humans, and robots. DSGs extend classical scene graphs by modeling dynamic elements as first-class citizens and accommodating real-time updates reflecting environmental changes, agent activities, and semantic relations. This unifying representation supports robust SLAM, long-term spatial memory, prediction, planning, and interaction in environments characterized by spatial and temporal variability.

## 1. Structural Foundations and Formal Definitions

A 3D DSG is formally a directed, layered graph $G=(V,E)$, where nodes $V$ represent spatial entities at multiple abstraction levels (e.g., points, objects, agents, places, rooms, buildings), and edges $E$ encode spatial, semantic, topological, and spatio-temporal relations. Layering endows the DSG with hierarchical semantics; typical layers include:

- Layer 1: Metric/geometric primitives (mesh vertices, point clouds)
- Layer 2: Objects and dynamic entities (agents/humans/robots)
- Layer 3: Places and structures (navigational points, walls, floors)
- Layer 4: Rooms or spatial enclosures
- Layer 5: Buildings or global context
- Additional temporal/dynamic layers (for temporal flow, occupancy histograms, or object tracks) [2512.11903][2101.06894][2002.06289][2405.00552]

Each node $v$ is associated with attributes $\Phi_V(v)$ including position $g_v \in \mathbb{R}^3$, semantic class $s_v$, and (optionally) traversability or temporal state. Edges, annotated by $\Phi_E(e)$, represent adjacency, containment, support, motion, or hierarchical relations and may capture odometric factors, entity-keyframe links, or spatio-temporal transitions [2503.02050][2405.00552][2002.06289].

## 2. Dynamic Modeling: Nodes, Constraints, and Temporal Integration

A critical advancement in DSGs is explicit modeling of dynamic entities as graph nodes. This includes:

- **Agent and Dynamic Object Nodes:** Each detected entity (human, robot, movable furniture) is represented by a sequence of time-indexed pose nodes, supporting multi-view and temporal data association.
- **Constraint Factors:** 
  - **Entity–Keyframe Constraints** bind dynamic observations to robot keyframes, coupling robot and object/agent localization.
  - **Intra-Entity Constraints** link consecutive observations of the same entity via semantic-class motion priors, differentiating static objects (stationary unless moved) from active agents (always mobile).
  - **Entity–Floor Constraints** ensure vertical consistency relative to building structure [2503.02050].
- **Temporal/Flow Layers:** Some DSG frameworks augment the spatial graph with per-node temporal statistics, e.g., flow histograms or activity frequency models (as in Aion, employing Fourier-based "Frequency Map Enhancement" for periodicity) [2512.11903].
- **Spatio-Temporal Edges:** Direct temporal links encode dynamic trajectories, interaction sequences, or event histories (e.g., agent pose-tracks).

This explicit integration of temporal factors and motion models enables robust modeling and inference under dynamics, supporting multi-entity, multi-time reasoning [2101.06894][2503.02050][2512.11903].

## 3. Acquisition, Construction, and Update Algorithms

DSGs are constructed and maintained using multi-modal, often real-time pipelines:

**Acquisition & Construction:**
- **SLAM Backend:** Real-time visual-inertial (or LiDAR/RGB-D) SLAM reconstructs the environment; pose estimates anchor keyframes and agent/object tracks [2101.06894][2503.02050].
- **Semantic Segmentation:** Vision-language models and semantic segmentation networks provide hierarchical object, agent, and scene labels [2502.15309][2410.11989][2411.02938].
- **Hierarchical Parsing:** Geometric clustering, distance transforms, and panoptic labeling infer places, rooms, and building hierarchies [2101.06894][2002.06289][2512.15047].
- **Dynamic Object Detection:** Fiducial markers (e.g., AprilTags), motion trackers, or open-vocabulary detectors identify and disambiguate dynamic objects and agents over time [2503.02050][2502.15309][2410.11989].

**Update Mechanisms:**
- **Multi-Modal Change Detection:** Integration of vision-based perception, robot action logs, human textual input, and temporal priors yields a unified stream of “change descriptors,” each specifying node/edge additions, removals, or modifications [2411.02938].
- **Local/Partial Graph Editing:** Rather than global scene reconstruction, localized subgraph updates add/remove/adjust only affected nodes and their incident edges for computational efficiency [2410.11989]. 
- **Event-Driven Simulation:** Discrete-event simulation (as in FOGMACHINE) models object spawn/deletion, agent movement, and interaction under partial observability, supporting belief estimation and uncertainty propagation in multi-agent settings [2510.09483].
- **Joint Optimization:** Nonlinear least-squares solvers jointly optimize all state variables (robot, object, agent, and environmental structure) under the imposed constraints [2503.02050][2101.06894].

Efficient, real-time update is feasible; for instance, optimization cycles in constraint-based DSG SLAM have been demonstrated at ~81ms per cycle [2503.02050], and multimodal updates can be integrated with 10Hz planning loops [2411.02938].

## 4. Hierarchy, Semantics, and Physical Interaction

The hierarchical organization of DSGs enables representation and reasoning at multiple spatial and semantic levels:

- **Layered Containment:** Nodes are related via containment edges (e.g., object ‘in’ room, room 'in' floor, floor 'in' building) and attached hierarchically for scalable querying and navigation [2101.06894][2002.06289][2512.15047].
- **Semantic Labeling:** Nodes inherit semantic class labels from detectors and vision-language models, enabling open-vocabulary object referencing and function-aware planning [2502.15309][2410.11989].
- **Dynamic Traversability:** Traversability modeling distinguishes static obstacles from movable ones, promoting certain object nodes (“operable obstacles”) to enable interaction-aware planning. Dynamic navigational edges are created to reflect possible pathways opened by moving objects [2512.15047].
- **Functional and Affordance Reasoning:** The inclusion of physical and semantic attributes supports affordance-based planning; for example, HERO combines efficiency-driven and semantic filtering to only promote truly movable obstacles [2512.15047].
- **Topological and Geometric Abstraction:** DSGs encode adjacency, support, and spatial relations (e.g., “on-top-of,” “inside,” “adjacentTo”) for rich relational inference [2002.06289][2502.15309][2410.11989].

## 5. Downstream Applications and Quantitative Evaluation

DSGs underpin a wide range of robotics and embodied AI tasks:

**SLAM and Localization:** Joint optimization over static and dynamic nodes achieves substantial reductions in pose estimation error compared to static-world approaches (e.g., 27.57% reduction in ATE relative to S-Graphs+ [2503.02050]).

**Planning and Navigation:** DSGs enable hierarchical semantic path-planning, multi-resolution A* over building/room/place graphs, and dynamic traversability-aware planning. HERO’s dynamic navigational graph yields a 35.1% reduction in path length and a 79.4% increase in success rate versus static baselines for navigation among movable obstacles [2512.15047].

**Long-Term Prediction and Memory:** DSGs facilitate long-term agent trajectory prediction by providing rich environmental context to LLMs, supporting probabilistic rollout of human-object interactions and continuous-time Markov Chain-based filtering [2405.00552]. DSGs also support real-time update pipelines and memory management in shared environments [2411.02938].

**Open-Vocabulary and Semantic Interaction:** Scene graphs constructed with VLMs and maintained via local subgraph updates support robust language-guided manipulation and dynamic object/entity retrieval in evolving environments, with task success and scene graph accuracy notably surpassing static baselines [2410.11989][2502.15309].

**Multi-Agent Coordination and Simulation:** DSGs integrated with event-driven simulation (e.g., FOGMACHINE) support uncertainty propagation, decentralized belief updates, and communication in multi-agent scenarios [2510.09483].

Performance metrics are diverse: ATE for SLAM, SCDA for change detection, NLL/BoN ADE for prediction tasks, Recall@1 for object retrieval, and task-level metrics such as path length, navigation error, success rate, and computational latency [2503.02050][2410.11989][2502.15309][2405.00552][2512.15047].

## 6. Open Challenges, Limitations, and Research Directions

DSGs present several ongoing challenges:

- **Robust Dynamic Entity Detection:** Many approaches still rely on fiducial markers or closed-world object detectors, limiting generality. Marker-free, purely vision-based detection with robust data association remains an open research area [2503.02050][2411.02938].
- **Uncertainty Representation:** Probabilistic DSGs, modeling uncertainty in object identity, pose, existence, and movability, are necessary for robust planning under partial information and sensor noise [2510.09483][2512.15047].
- **Dynamic Physical Interaction:** Extension to richer affordance models and physical manipulation (beyond binary movability) is required for complex multi-step tasks [2512.15047].
- **Temporal Consistency and Memory:** Maintaining globally and temporally consistent DSGs under frequent environment change, especially in multi-agent settings, is nontrivial; scalable, real-time update strategies continue to evolve [2411.02938][2510.09483].
- **Integration with LLMs and Open-Vocabulary Semantics:** Ongoing work explores fusing DSGs with large language and vision-language models for natural language interaction, task grounding, and open-world understanding [2410.11989][2502.15309][2405.00552].
- **Evaluative Benchmarks:** Standardized, large-scale benchmarks and metrics for DSG maintenance, update accuracy, and task-level performance are identified as a research need [2411.02938][2405.00552].

A plausible implication is that DSG-based representations will serve as the foundational world models for future autonomous robots and embodied agents, enabling robust operation, reasoning, and collaboration in complex, dynamic, and semantically rich real-world environments.

---

**References**  
- [2503.02050] Constraint-Based Modeling of Dynamic Entities in 3D Scene Graphs for Robust SLAM  
- [2512.11903] Aion: Towards Hierarchical 4D Scene Graphs with Temporal Flow Dynamics  
- [2512.15047] HERO: Hierarchical Traversable 3D Scene Graphs for Embodied Navigation Among Movable Obstacles  
- [2502.15309] DynamicGSG: Dynamic 3D Gaussian Scene Graphs for Environment Adaptation  
- [2411.02938] Multi-Modal 3D Scene Graph Updater for Shared and Dynamic Environments  
- [2002.06289] 3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans  
- [2410.11989] Dynamic Open-Vocabulary 3D Scene Graphs for Long-term Language-Guided Mobile Manipulation  
- [2101.06894] Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphs  
- [2405.00552] Long-Term Human Trajectory Prediction using 3D Dynamic Scene Graphs  
- [2510.09483] FOGMACHINE -- Leveraging Discrete-Event Simulation and Scene Graphs for Modeling Hierarchical, Interconnected Environments under Partial Observations from Mobile Agents

Source: https://www.emergentmind.com/topics/3d-dynamic-scene-graphs-dsgs