Spatio-temporal Scene Graphs Overview

• Spatio-temporal scene graphs are structured representations that model dynamic visual scenes by linking objects and their temporal interactions.
• They encompass varied paradigms—frame-level, world-centered, and predictive models—that are applied in video recognition, robotics, and remote sensing.
• These graphs drive improvements in action understanding and autonomous planning while addressing challenges like bias, occlusion, and complex temporal dynamics.

Spatio-temporal Scene Graphs (SSGs) are structured representations of dynamic visual scenes in which entities and their relations are organized as graphs over time. In the dominant video-understanding formulation, a scene is decomposed into objects or trajectories and their pair-wise relations, yielding either a temporally ordered sequence of scene graphs or an enriched graph with explicit temporal context; however, the term also encompasses process-centric graphs in remote sensing and world-centered 3D or 4D graphs in robotics (Ji et al., 2019, Zhu, 15 Mar 2025, 0709.3013). This suggests that SSGs are better understood as a family of dynamic relational representations than as a single canonical ontology.

1. Historical emergence and conceptual scope

A foundational formulation for mainstream video SSGs came from Action Genome, which treats actions as compositions of changing scene graphs over time and represents an action as a sequence of frame-level graphs whose nodes are a person and action-relevant objects, and whose edges are human-object relationships such as attention, spatial, and contact relations (Ji et al., 2019). In that formulation, the frame graph is written as $G=(O,R)$, where $O$ is the set of objects the person is interacting with and $R$ is the set of relationships between the person and those objects. The key conceptual shift is that an action is not a monolithic label but a trajectory through relationship space.

At the same time, earlier graph-based treatments of spatio-temporal semantics existed outside the current object-predicate-object tradition. In satellite image sequences, graph patterns $\mathcal{G}_k \subset \mathcal{G}$ were already used to represent evolving region-process structures, with vertices corresponding to multivariate Gaussian distributions for spatial classes at specific times and edges corresponding to temporal evolution between image samples (0709.3013). Those graphs were used for semantic recognition and probabilistic retrieval of phenomena such as field maturation, plowing, cloud occlusions, and annual farming practices. This precursor indicates that SSG-style reasoning did not begin as a purely object-triplet formalism.

A persistent misconception is that every SSG is simply a stack of per-frame subject-predicate-object triplets. In practice, the literature includes person-centric action graphs, trajectory-level panoptic video graphs, pseudo-3D world-space graphs, latency-aware scene graphs for teleoperation, and predictive scene graphs whose edges are probabilistic temporal filters rather than static relation labels (Malawade et al., 2021, Wang et al., 27 Sep 2025, Saavedra-Ruiz et al., 30 Apr 2026, Gao et al., 26 Feb 2026). The field is therefore unified less by a single graph schema than by a shared objective: preserving relational semantics under temporal evolution.

2. Representational forms and graph semantics

A canonical formalization treats a video as a temporally ordered sequence of per-frame scene graphs. In scene graph anticipation, each frame $I^t$ contains objects $O^t=\{o_k^t\}_{k=1}^{N(t)}$, each relationship instance is $r_{ijk}^t=(o_i^t,p_{ijk}^t,o_j^t)$, and the scene graph at frame $t$ is $\mathcal{G}^t=\{r_{ijk}^t\}_{ijk}$; the video induces the sequence $\{\mathcal{G}^t\}_{t=1}^T$ (Peddi et al., 2024). This is a temporally indexed graph sequence rather than a single persistent graph.

A second line of work makes temporal relations explicit graph elements. In dynamic scene graph generation, one proposal enriches the scene graph definition from $O$0 to $O$1, where $O$2 is a temporal context set and each $O$3 represents the temporal dependency between object $O$4 in frame $O$5 and object $O$6 in frame $O$7 (Zhu, 15 Mar 2025). In this view, temporal relations are first-class graph components rather than latent attention weights or diffuse temporal mixing.

A third family is world-centered. The $O$8D formulation lifts per-frame detections into pseudo-3D, registers them into a shared spatio-temporal space, and separates static and dynamic subgraphs, while a 4D scene graph formulation defines $O$9, where each object node is represented by a STEP-token sequence encoding semantic attributes, geometric extent, and temporal evolution (Cherian et al., 2022, Sohn et al., 18 Dec 2025). In robotics, PredictiveGraphs defines a metric-semantic graph $R$0 whose nodes are semi-static objects, receptacles, and background elements, while edges function as Perpetua$R$1 filters encoding spatio-semantic relationships over time (Saavedra-Ruiz et al., 30 Apr 2026). More recently, World Scene Graph Generation has been formalized as constructing a world scene graph at each timestamp that encompasses all interacting objects in the scene, both observed and unobserved, explicitly targeting objects that are temporarily unobserved due to occlusion or camera motion (Peddi et al., 13 Mar 2026).

Interval-based SSGs also appear in reasoning systems. In video reward reasoning, the graph is $R$2, where entities have categories and attributes, and relations are tuples $R$3 with a temporal interval $R$4 in seconds (Kim et al., 10 Jun 2026). This formulation is persistent over the clip rather than frame-local. The common thread across these variants is that time can enter an SSG as frame order, explicit inter-frame edges, interval annotations, world-state persistence, or predictive edge beliefs.

3. Construction and learning paradigms

One established paradigm constructs a graph per frame and then learns temporal structure over graph embeddings. Scene Graph Feature Banks convert each frame graph into an $R$5 confidence matrix, flatten it into a feature vector, and aggregate the sequence over time for action recognition (Ji et al., 2019). roadscene2vec follows the same high-level logic in road scenes: it extracts a graph sequence $R$6, embeds each graph with a multi-relational GNN, and then uses an LSTM and optional attention to produce spatio-temporal embeddings for risk assessment and collision prediction (Malawade et al., 2021). This frame-graph-plus-temporal-encoder design remains one of the simplest operationalizations of SSGs.

A second paradigm learns explicit temporal structure rather than relying only on temporal aggregation. The Salient Temporal Relation Encoder selectively builds temporal connections only between temporal-relevant object pairs and represents the temporal relations as explicit edges in the scene graph; the resulting sparse and explicit temporal representation improves upon strong scene graph generation baselines by up to $R$7 in Scene Graph Detection and yields a $R$8 gain in mAP for action recognition (Zhu, 15 Mar 2025). The underlying claim is that dense all-to-all temporal connections encode many non-meaningful interactions, whereas selected inter-frame edges provide a cleaner temporal graph.

A third paradigm treats future graph evolution itself as the task. Scene Graph Anticipation introduces forecasting of future scene graphs $R$9 conditioned on observed video $\mathcal{G}_k \subset \mathcal{G}$0, and SceneSayer models future relation embeddings in continuous time using NeuralODEs and NeuralSDEs (Peddi et al., 2024). Here the temporal structure is latent rather than explicit in the graph topology, but the target remains future pair-wise relationships between objects.

A fourth paradigm uses large language or vision-LLMs as structured scene analyzers. SceneLLM maps video into an implicit language signal through VQ-VAE tokenization, Spatial Information Aggregation, and Optimal Transport, then uses a LoRA-adapted LLaMA-13B and a transformer-based SGG predictor to decode dynamic scene graphs (Zhang et al., 2024). VOST-SGG instead keeps a one-stage DETR-style pipeline but adds dual-source query initialization, which disentangles what to attend to from where to attend, and a multi-modal feature bank fusing visual, textual, and spatial cues derived from a VLM (Sugandhika et al., 5 Dec 2025). TraSeR, trained on SVG2, uses trajectory-aligned token arrangement plus an object-trajectory resampler and a temporal-window resampler to generate compact spatio-temporal scene graphs in a single forward pass (Gao et al., 26 Feb 2026). These methods indicate a broader movement from graph-only inductive bias toward graph generation mediated by pretrained multimodal priors.

A fifth paradigm centers on temporal robustness rather than architecture alone. Meta Video Scene Graph Generation treats VidSGG as a setting with spatio-temporal conditional bias and uses a meta-learning support/query construction over 15 conditional bias types to improve generalization under spatial and temporal distribution shift (Xu et al., 2022). This suggests that in SSGs, temporal context is both indispensable and a source of shortcut learning.

4. Benchmarks, datasets, and evaluation protocols

Benchmark design has decisively shaped what counts as an SSG. Action Genome remains the standard person-centric benchmark for frame-level video scene graph prediction, while VidVRD and VidOR emphasize video-level or trajectory-level relations, PVSG introduces panoptic video scene graphs, and SVG2 scales the problem to open-vocabulary panoptic supervision (Ji et al., 2019, Xu et al., 2022, Gao et al., 26 Feb 2026).

Dataset	Brief characteristics	Role in SSG research
Action Genome	9,848 / 10K videos; 234,253 annotated frames; 476,229 bounding boxes; 1,715,568 relationship instances; 35 object classes; 25 relationship classes	Canonical action-centric video SSG benchmark (Ji et al., 2019)
VidVRD	1000 videos; 35 object categories; 132 predicate categories	Video-level VidSGG benchmark (Xu et al., 2022)
VidOR	7.0K videos; 34.6K objects; 0.3M relations	Open-world video relation benchmark (Gao et al., 26 Feb 2026)
PVSG	338 videos; 6.3K objects; 3.6K relations	Panoptic video scene graph benchmark (Gao et al., 26 Feb 2026)
SVG2	636K videos; 6.6M objects; 52.0M attributes; 6.7M relations	Large-scale panoptic open-vocabulary SSG dataset (Gao et al., 26 Feb 2026)
ActionGenome4D	Action Genome upgraded into 4D scenes via feed-forward 3D reconstruction, world-frame oriented bounding boxes, and dense relationship annotations including objects temporarily unobserved due to occlusion or camera motion	World-scene-graph benchmark direction (Peddi et al., 13 Mar 2026)

Evaluation protocols differ substantially across subfields. Action Genome commonly uses Predicate Classification, Scene Graph Classification, and Scene Graph Detection, evaluated with Recall@K and increasingly mean Recall@K because of long-tailed predicate distributions (Zhu, 15 Mar 2025, Sugandhika et al., 5 Dec 2025). VidVRD additionally uses relation detection and relation tagging, with metrics such as mAP, R@50, R@100, and P@1/P@5/P@10 (Xu et al., 2022). In open-vocabulary SVG2 evaluation, object, relation, and triplet correctness are judged semantically with an LLM-based evaluator, and relation correctness also requires temporal IoU $\mathcal{G}_k \subset \mathcal{G}$1 with ground-truth intervals (Gao et al., 26 Feb 2026).

Dataset scale has become a substantive research issue rather than a bookkeeping detail. SVG2$\mathcal{G}_k \subset \mathcal{G}$2, though only 100 videos, still contains 3.2K objects, 9.7K attributes, 3.3K relations, 749 object categories, and 249 relation categories, and human verification on sampled SVG2 videos reports object label accuracy $\mathcal{G}_k \subset \mathcal{G}$3, relation accuracy $\mathcal{G}_k \subset \mathcal{G}$4, and attribute accuracy $\mathcal{G}_k \subset \mathcal{G}$5 (Gao et al., 26 Feb 2026). This scale and vocabulary breadth depart sharply from earlier fixed-taxonomy SSG datasets and push the field toward open-vocabulary generation and semantic evaluation.

5. Downstream uses and empirical impact

SSGs were initially motivated by action understanding, and that application remains central. Scene Graph Feature Banks improve Charades action recognition from $\mathcal{G}_k \subset \mathcal{G}$6 mAP for LFB to $\mathcal{G}_k \subset \mathcal{G}$7 mAP for SGFB, while the few-shot setting reaches $\mathcal{G}_k \subset \mathcal{G}$8 mAP using as few as 10 examples (Ji et al., 2019). A later spatial-temporal scene graph feature bank that includes explicit temporal context reaches $\mathcal{G}_k \subset \mathcal{G}$9 mAP on Charades, improving over SGFB and the authors’ spatial-only bank (Zhu, 15 Mar 2025). These results support the claim that decomposing actions into evolving relations can improve both standard and low-shot recognition.

Video question answering has also been an important testbed. The $I^t$0D scene graph representation attains $I^t$1 accuracy on NExT-QA and $I^t$2 mean rank on AVSD-QA while also reducing graph size through static-node merging (Cherian et al., 2022). More generally, explicit SSGs can serve as an intermediate representation for downstream reasoning: when TraSeR-generated scene graphs are sent to a VLM for video question answering, they provide a $I^t$3 to $I^t$4 absolute accuracy gain over using video only or video augmented with Qwen2.5-VL-generated scene graphs (Gao et al., 26 Feb 2026).

Autonomous driving and robotics have broadened the scope of SSGs beyond human-action video. roadscene2vec operationalizes road SSGs as graph sequences over road entities such as car, motorcycle, bicycle, pedestrian, lane, light, and sign, and reports strong transfer learning: training on 271-syn yields ResNet-50 $I^t$5 from source to 571-honda, while MRGCN yields $I^t$6 (Malawade et al., 2021). For teleoperation, ST-OVSG reports $I^t$7 percent node accuracy on the Replica benchmark and a planning success rate of $I^t$8 percent in the latency-robustness experiment, arguing that spatio-temporal open-vocabulary scene graphs help resolve local-remote state mismatches caused by transmission delay (Wang et al., 27 Sep 2025). PredictiveGraphs extends this further by forecasting future object-receptacle relations in semi-static scenes and outperforms baselines in predictive navigation, while SNOW uses a 4D Scene Graph as a persistent prior for embodied reasoning and reports $I^t$9 overall on NuScenes-QA and $O^t=\{o_k^t\}_{k=1}^{N(t)}$0 average on RoboSpatial-Home (Saavedra-Ruiz et al., 30 Apr 2026, Sohn et al., 18 Dec 2025).

SSGs are also being used as explicit grounding substrates for higher-level reasoning. SG-PVR treats the scene graph as a persistent structured visual reference throughout reasoning for text-to-video reward modeling, decomposing prompts into atomic claims and verifying them against both video and a temporally grounded scene graph (Kim et al., 10 Jun 2026). This suggests that SSGs are increasingly valued ոչ only as predictors of relations but as externalized world models for verification, planning, and compositional reasoning.

6. Limitations, ambiguities, and active research directions

A recurrent criticism of existing video SSG methods is frame-centrism. Recent work explicitly argues that existing methods remain fundamentally frame-centric because they reason only about currently visible objects, discard entities upon occlusion, and operate in 2D; the proposed response is World Scene Graph Generation, which constructs a world scene graph at each timestamp including both observed and unobserved interacting objects (Peddi et al., 13 Mar 2026). Even methods that use temporal context often keep the output frame-local, as in scene graph anticipation, where temporal linkage is handled in latent representation rather than through explicit inter-frame graph edges (Peddi et al., 2024). This suggests an unresolved divide between frame-conditioned graph prediction and persistent world-state graph modeling.

Bias, noise, and incompleteness remain major obstacles. Meta Video Scene Graph Generation identifies spatio-temporal conditional bias as a central issue and organizes it into 15 conditional bias types, showing that temporal context can become a shortcut rather than a robust signal (Xu et al., 2022). In SSG-grounded reward reasoning, the extracted graph has high precision—entity precision $O^t=\{o_k^t\}_{k=1}^{N(t)}$1, attribute precision $O^t=\{o_k^t\}_{k=1}^{N(t)}$2, relation precision $O^t=\{o_k^t\}_{k=1}^{N(t)}$3—but low overall requirement coverage $O^t=\{o_k^t\}_{k=1}^{N(t)}$4, with entity coverage $O^t=\{o_k^t\}_{k=1}^{N(t)}$5, attribute coverage $O^t=\{o_k^t\}_{k=1}^{N(t)}$6, and action coverage $O^t=\{o_k^t\}_{k=1}^{N(t)}$7 (Kim et al., 10 Jun 2026). Large-scale automatic datasets partly alleviate annotation scarcity but introduce their own quality ceilings, as SVG2’s human verification numbers make clear (Gao et al., 26 Feb 2026).

There is also a terminological ambiguity in the acronym itself. In one paper, “SSG” denotes “Situational Scene Graph,” a frame-level human-centric graph enriched with semantic role-value frames for person, object, and verb predicate, rather than “spatio-temporal scene graph” in the standard sense (Sugandhika et al., 2024). That representation is valuable for semantic augmentation, but it is not a temporally explicit graph process model. This suggests that the literature now contains at least two partially overlapping uses of “SSG”: one temporal, one situational.

Current directions are correspondingly diverse. Explicit sparse temporal edges are being extended beyond adjacent-frame modeling (Zhu, 15 Mar 2025). Open-ended ST-SGG methods are incorporating VLM priors through semantically grounded queries and multimodal banks, while also exposing the need for pose and motion cues and for open-ended scene graph generation beyond fixed vocabularies (Sugandhika et al., 5 Dec 2025). World-centered models are moving toward 4D scene graphs, latency-aware planning graphs, predictive graphs over semi-static environments, and world scene graphs that persist through occlusion (Sohn et al., 18 Dec 2025, Wang et al., 27 Sep 2025, Saavedra-Ruiz et al., 30 Apr 2026, Peddi et al., 13 Mar 2026). Taken together, these trajectories indicate that the field is shifting from per-frame relation recognition toward persistent, queryable, and eventually predictive world models with explicit relational structure.