4D Scene Graphs: Dynamic Scene Modeling

• 4D scene graphs are structured representations that encode semantic, spatial, and temporal relationships in dynamic environments, enabling robust scene understanding.
• They extend traditional 2D/3D scene graphs by incorporating temporal evolution and multi-modal data, supporting applications in robotics, autonomous driving, and surgical workflow analysis.
• Current research addresses challenges such as scalability, temporal coherence, and data scarcity while employing advanced methods like transformer-based pipelines and graph neural networks.

4D scene graphs are structured representations that model the semantic, spatial, and temporal relationships within dynamic environments. These graphs extend traditional 2D and 3D scene graphs by incorporating the evolution of entities and their interactions over time, supporting rigorous reasoning in domains such as robotics, autonomous vehicles, healthcare, and artificial intelligence. This entry surveys formal definitions, data modalities, learning paradigms, representative datasets, quantitative results, and current research challenges in 4D scene graphs, synthesizing advances across diverse application areas.

1. Formal Definitions and Representational Structure

A 4D scene graph is generally defined as a sequence or bundle of attributed relational graphs, each representing the state of a physical or conceptual scene at a discrete time index, with additional structures to encode temporal persistence and cross-time relationships.

Generic formulation:

At timestamp $t$, the scene graph is

$G_{t} = (N_{t}, E_{t}, R, X_{t})$

where

• $N_{t}$: nodes representing scene entities (e.g., humans, objects, structures).
• $R$: vocabulary of relation/predicate types (e.g., “Hold”, “NextTo”, “Cut”, “Left-of”).
• $E_{t} \subseteq N_{t} \times R \times N_{t}$: directed, labeled edges encoding relationships at time $t$.
• $X_{t}$: node attributes (positions, attributes, feature embeddings, etc).

The evolution of the system is captured as a 4D graph sequence

$\mathcal{G} = \{G_{1}, G_{2}, \ldots, G_{T}\}$

with $G_{t}$ at each of $T$ timesteps. Many frameworks explicitly link entities with unique identity edges across time, supporting temporal argumentation, object permanence, and query of historical context (Özsoy et al., 2022, Yang et al., 2024, Peddi et al., 13 Mar 2026, Catalano et al., 10 Dec 2025, Wu et al., 19 Mar 2025). Spatial attributes (e.g., 3D bounding boxes, mesh descriptors, mask-tubes) and temporal attributes (e.g., action intervals, temporal flow histograms) are canonically attached to nodes.

Specialized forms:

• Panoptic 4D Scene Graphs (PSG-4D): Integrate object tubes (segmentation masks/tracks), open-vocabulary class labels, and spatiotemporal triplets (subject, predicate, object, [start, end]) (Yang et al., 2024, Wu et al., 19 Mar 2025).
• Gaussian Scene Graphs: Nodes are sets of anisotropic Gaussians parameterized in 3D plus temporal pose trajectories; edges represent tracking and spatial/semantic interactions (Chen et al., 8 May 2026).
• World Scene Graphs: Partition entities into observed and unobserved sets, with all objects grounded in a global world coordinate system and tracked through occlusion (Peddi et al., 13 Mar 2026).
• Hierarchical Scene Graphs: Multi-resolution abstraction, with nodes spanning from geometry to navigational to semantic/global regions; temporal flows are stored as node descriptors or attached histograms (Catalano et al., 10 Dec 2025).

2. Data Modalities, Node/Edge Taxonomies, and Temporal Linking

Node taxonomy:

• Actors/Humans: Annotated with pose (3D/6D), role, or action label (Özsoy et al., 2022, Yang et al., 2024).
• Objects: Subdivided by class (equipment, furniture, tools, vehicles, background) with geometric descriptors such as OBBs, 3D bounding boxes, or panoptic mask tubes (Yang et al., 2024, Wu et al., 19 Mar 2025).
• Navigational/semantic regions: For robotics and navigation, nodes may represent waypoints, rooms, or dynamic clusters (Catalano et al., 10 Dec 2025).

Edge taxonomy:

• Semantic relations: Predicate sets tailored by domain (“Hold”, “Assist”, “Suture”, “Touch” in surgical settings; “CloseTo,” “On-top-of,” etc. in generic environments) (Özsoy et al., 2022).
• Spatiotemporal links: Temporal edges encode tracking, object identity maintenance, and motion transitions; time intervals define action span edges (Yang et al., 2024, Wu et al., 19 Mar 2025).
• Hierarchical/subgraph structure: Many frameworks decompose the full graph into clusters or abstraction layers, supporting local/global optimization (Liu et al., 2024, Catalano et al., 10 Dec 2025).

Temporal modeling:

• Explicit linking: Identity edges across $G_{t} = (N_{t}, E_{t}, R, X_{t})$0 link entities via Hungarian matching or embedding similarity (Özsoy et al., 2022, Yang et al., 2024).
• Trajectory encodings: Nodes may store explicit pose/motion vectors per frame or fit parametric trajectories for robust interpolation (Chen et al., 8 May 2026, Catalano et al., 10 Dec 2025).
• Persistence through occlusion: World scene graphs maintain latent representations for unobserved objects, allowing reasoning through temporary absence (Peddi et al., 13 Mar 2026).

3. Learning Paradigms and Computational Pipelines

Modern 4D scene graph generation employs a range of modular and end-to-end learning pipelines, built on fused multi-modal perception with explicit temporal reasoning.

Canonical pipeline components:

• Detection and Instance Segmentation: Per-frame 3D/2D human and object detections via volumetric pose estimators (e.g., VoxelPose), transformers, point-cloud encoders (e.g., GroupFree, ObjPointNet), or panoptic mask backbones (e.g., Mask2Former) (Özsoy et al., 2022, Yang et al., 2024, Wu et al., 19 Mar 2025).
• Temporal tracking: Association via keypoint matching, mask-tube linking, or learned temporal transformers (UniTrack, Kalman + RTS smoothing) (Yang et al., 2024, Peddi et al., 13 Mar 2026).
• Feature extraction: Multi-view 3D and RGB image features are fused, with per-node and per-relation encoders (e.g., RelPointNet, global EfficientNet features, ImageBind) (Özsoy et al., 2022, Wu et al., 19 Mar 2025).
• Graph construction and relation classification: GCN or Transformer-based message passing for node/edge embedding refinement and multi-label predicate classification (Özsoy et al., 2022, Yang et al., 2024, Peddi et al., 13 Mar 2026).
• Temporal reasoning: Modules include static feature buffers (PWG), bidirectional transformers (4DST), temporal edge attention, or in-context LLM prompting for scene modification (Peddi et al., 13 Mar 2026, Liu et al., 2024, Wu et al., 19 Mar 2025).
• Augmentation and loss: Weighted cross-entropy on rare relations, augmentation in pose/image/mask space (scale, translation, color), loss balancing for segmentation, association, and relation terms (Özsoy et al., 2022, Yang et al., 2024, Wu et al., 19 Mar 2025).

4. Public Datasets and Benchmarks

Several large-scale datasets and benchmarks are available for 4D scene graph research, spanning environments from synthetic worlds to real robotics and healthcare domains.

Dataset	Domain	Frames	Objects/Relations	Key Modality	Reference
4D-OR	Surgical OR	6,734	12 objects, 14R	RGB-D, 3D pose	(Özsoy et al., 2022)
PSG4D-GTA/HOI	GTA, HOI	1M+	35–46 obj, 15–43R	RGB-D, panoptic	(Yang et al., 2024)
AG4D (ActionGenome4D)	Indoor video	N/A	37 obj, 26 rel	Monocular video + 3D	(Peddi et al., 13 Mar 2026)
V2X-Seq	Automotive	26 seq	Vehicles, ped	Multi-view sync	(Chen et al., 8 May 2026)
Rich2D→4D PSG	Synthetic/Real	27k–891k	Open-vocab	RGB-D + 2D SG	(Wu et al., 19 Mar 2025)

Datasets provide rich annotations: panoptic segmentation tubes, object identity tracks, fine-grained relations with temporal intervals, clinical/navigational roles, and multi-view geometry. Annotation pipelines typically leverage a mix of manual strategies and ML-based pre-labeling (e.g., Mask pre-annotation, VLM bootstrapping, multimodal matching) (Yang et al., 2024, Peddi et al., 13 Mar 2026).

5. Quantitative Performance and Empirical Findings

Extensive benchmarking across tasks and architectures reveals several consistent patterns:

• Relation prediction: Macro-F1 scores of 0.75 for OR domain SSGs (Özsoy et al., 2022), and Recall@20 of 6.68 (PSG4DFormer) to 18.48 (4D-LLM) on PSG4D-GTA (Yang et al., 2024, Wu et al., 19 Mar 2025).
• Role/action inference: Role prediction in ORs reaches 0.85 macro-F1 via a Graphormer, outperforming heuristic baselines (Özsoy et al., 2022).
• Ablation insights: Incorporating depth and temporal cues (temporal transformers, mask-tubes, augmentations) yields up to 6–12 point gains in recall metrics (Yang et al., 2024, Wu et al., 19 Mar 2025); bidirectional transformers outperform static temporal buffers for world-scene reasoning (Peddi et al., 13 Mar 2026).
• Handling asynchrony: DUST’s dual-timeline GSGs eliminate photometric losses that scale quadratically with agent velocity and time offset (Δτ), outperforming single-timeline approaches in PSNR by >3 dB and reducing FVD by 37.7% (Chen et al., 8 May 2026).
• Scene graph transfer learning: 2D-to-4D feature transfer (transcending) boosts mean recall by 3–4 points, with LLM-driven pipelines showing strong open-vocabulary generalization (27% unseen mR) (Wu et al., 19 Mar 2025).

Table: Key experimental numbers

Task/Model	Dataset	Metric	Value	Reference
Relation Pred.	4D-OR	Macro-F1	0.75	(Özsoy et al., 2022)
Clinical Role Pred.	4D-OR	Macro-F1	0.85	(Özsoy et al., 2022)
PSG4DFormer	PSG4D-GTA	R@20	6.68	(Yang et al., 2024)
4D-LLM (ours)	PSG4D-GTA	R@20/mR@20	18.48/9.43	(Wu et al., 19 Mar 2025)
DUST (dynamic PSNR)	V2X-Seq	PSNR	+3.2 dB	(Chen et al., 8 May 2026)
Aion (Planning)	Synthetic Nav	Detours congested regions	Qual.	(Catalano et al., 10 Dec 2025)

6. Application Domains and Representative Use Cases

4D scene graphs serve as unified abstractions for:

• Surgical workflow modeling: Precise tracking and semantic relation inference in complex operating rooms; supports downstream clinical role prediction and potentially real-time intraoperative support (Özsoy et al., 2022).
• Robotics and embodied AI: Dynamic scene understanding for service robots, fusing RGB-D perception with temporal reasoning and LLM-driven policy integration (Yang et al., 2024).
• Autonomous driving and cooperative perception: Spatio-temporal alignment of multiple sensor streams, robust reconstruction under clock asynchrony, ghost-free dynamic agent rendering (Chen et al., 8 May 2026).
• Navigation and planning: Hierarchical encoding of spatial regions with node-level flow attributes to optimize pathfinding in dynamic, large-scale environments (Catalano et al., 10 Dec 2025).
• General video understanding: World-centric, temporally persistent scene graphs supporting occlusion reasoning, persistence, and long-range action aggregation (Peddi et al., 13 Mar 2026).
• 3D/4D content creation: Interactive authoring and visualization of 4D scenes in game engines, supporting projection, frustum visualization, and programmable spatial-temporal logic (Cavallo, 2021, Liu et al., 2024).

7. Challenges, Open Problems, and Future Directions

Prevailing challenges in 4D scene graph research include:

• Scalability: Maintaining efficiency, accuracy, and memory usage for complex or large-scale environments (e.g., long egocentric videos, urban scenes with many dynamic agents) (Yang et al., 2024, Chen et al., 8 May 2026).
• Temporal coherence and occlusion handling: Accurate object permanence, identity recovery through heavy occlusion, and robust reasoning when objects are unobservable (Peddi et al., 13 Mar 2026).
• Data scarcity and open vocabulary: Annotating richly labeled 4D data is resource-intensive; transfer learning from abundant 2D/3D scene graphs and LLM-facilitated reasoning show promise (Wu et al., 19 Mar 2025).
• Generalization: Out-of-vocabulary handling, zero-shot label assignment, and adaptation to new domains or unseen spatial/temporal interactions.
• Adoption in real-world systems: Integrating with robots, AR/VR, and surgical devices; dealing with sensor noise, imperfect calibration, and real-time constraints (Özsoy et al., 2022, Yang et al., 2024).
• Standardization and evaluation: Consistent metrics (recall@K, F1, PSNR, FVD), interpretability, and benchmarking in realistic scenarios (Wu et al., 19 Mar 2025, Chen et al., 8 May 2026).

Research goals include more efficient architectures, streaming/online inference, end-to-end 3D grounding, richer and more diverse public datasets, hierarchical and modular temporal models, and robust downstream integration for navigation, manipulation, and long-horizon understanding (Wu et al., 19 Mar 2025, Yang et al., 2024, Catalano et al., 10 Dec 2025, Peddi et al., 13 Mar 2026).