---
title: 'ST-OVSG: Open-Vocabulary Scene Graph'
url: https://www.emergentmind.com/topics/spatio-temporal-open-vocabulary-scene-graph-st-ovsg
type: topic
---

# ST-OVSG: Open-Vocabulary Scene Graph

Spatio-Temporal Open-Vocabulary Scene Graph (ST-OVSG) denotes a graph-structured scene representation that jointly models open-vocabulary entities, their relations, and their evolution over time. In the formulation introduced for robot teleoperation, ST-OVSG enriches open-vocabulary perception with temporal dynamics and lightweight latency annotations, uses LVLMs to construct open-vocabulary 3D object representations, extends them into the temporal domain via Hungarian assignment with a temporal matching cost, and provides task-oriented subgraph filtering so that an LVLM planner can retrospectively access the scene state actually seen by the operator under transmission delay [2509.23107]. Closely related work under adjacent names—including 4D Scene Graphs, dynamic open-vocabulary 3D scene graphs, PredictiveGraphs, and World Scene Graph Generation—extends the same general problem space toward SLAM-grounded 4D memory, local graph updates, future-state prediction, and reasoning over unobserved objects [2512.16461][2410.11989][2605.00121][2603.13185].

## 1. Formal definition and representational scope

In the teleoperation setting, ST-OVSG is a dynamic scene representation for open-world, changing environments in which each object node can carry a bounding box, mask, visual features, text features, a 3D centroid, a 3D size, a semantic point cloud, and an observation timestamp, while spatial relations are predicted within each frame and temporal edges link object instances across frames [2509.23107]. The observation time is explicitly delay-aware: each frame-level graph stores a capture timestamp $\tau_n$, an estimated latency $\Delta T_n$, and the effective observation time seen by the user is $\tau_n + \Delta T_n$. This makes the graph not merely temporal but temporally aligned to remote human perception.

A central consequence of this formulation is that the graph is simultaneously semantic, geometric, and operational. Spatial edges encode relations such as open-vocabulary predicates predicted by LVLMs; temporal edges encode identity persistence and history; latency tags encode the discrepancy between robot-time and operator-time. The task-oriented subgraph filtering mechanism then scores nodes against the instruction embedding,
$$
s_{i,t} = \cos(\mathbf{g}_u, \mathbf{f}^{\text{txt}}_i) + \beta \cos(\mathbf{g}_u, \mathbf{f}^{\text{img}}_i),
$$
and retains the most relevant objects together with their spatial neighbors and temporal trajectories for planner consumption [2509.23107].

Related representations broaden this notion beyond teleoperation. SNOW defines a 4D Scene Graph whose nodes are STEP-tokenized object instances and whose state is
$$
\mathcal{M}^{t} = (\mathcal{G}^{t-T:t}, \{S_k^{t-T:t}\}),
$$
with STEP tokens integrating patch-level appearance, centroid, shape, and temporal interval [2512.16461]. World Scene Graph Generation formalizes a graph at each timestamp that includes both observed and unobserved objects in a global 3D frame, explicitly targeting object permanence under occlusion and camera motion [2603.13185]. This suggests that ST-OVSG is best understood as a family of temporally persistent, open-vocabulary scene graph formalisms rather than a single architectural template.

## 2. Intellectual lineage from open-vocabulary scene understanding and video scene graphs

The ST-OVSG problem emerges from the intersection of two previously separate trajectories: open-vocabulary scene parsing and temporally aware scene graph generation. On the open-vocabulary side, SGDN was introduced as the first framework to explicitly exploit scene graph cues for open-vocabulary object detection, using a scene-graph-based decoder, scene-graph-based prediction, scene-graph-based offset regression, and cross-modal learning; it achieved 37.5% AP50 on novel classes on COCO, 23.6% mAP on rare classes on LVIS, and 31.1% mAP overall, while also showing the ability to perform open-vocabulary scene graph detection [2307.03339]. PGSG then recast scene graph generation as image-to-sequence generation with a generative VLM, yielding the first end-to-end framework for scene graph generation with open-vocabulary predicates and establishing explicit relation modeling as a transfer mechanism to downstream vision-language tasks [2404.00906]. OvSGTR further systematized the field by distinguishing Closed-set SGG, OvD-SGG, OvR-SGG, and OvD+R-SGG, and by introducing visual-concept alignment, relation-aware pretraining, and knowledge distillation to retain open-vocabulary relation knowledge during fine-tuning [2311.10988].

On the temporal side, STKET treated Video Scene Graph Generation as a problem of embedding statistical spatial co-occurrence and temporal transition priors into transformer cross-attention, improving mR@50 by 8.1%, 4.7%, and 2.1% across the standard tasks and increasing R@50 for the 10 least frequent relationships from 9.98% to 23.98% [2309.13237]. This is significant because long-tail relation bias remains one of the main bottlenecks when a graph must generalize across time as well as vocabulary.

Taken together, these lines of work indicate that ST-OVSG inherits two nontrivial requirements. First, open-vocabulary performance depends on visual-language alignment, generative decoding, or similarity-based classification rather than fixed categorical heads. Second, temporal consistency requires more than frame-wise graph prediction: it requires explicit cross-frame association, memory, or temporal priors. ST-OVSG arises when both requirements are imposed simultaneously.

## 3. Graph construction, association, and multimodal inference

The canonical ST-OVSG pipeline for teleoperation starts from RGB-D frames processed by open-vocabulary LVLM components for object detection and segmentation. Each object is lifted into 3D and represented by semantic, geometric, and temporal attributes, after which temporal association is performed by solving a Hungarian assignment problem over a cost matrix that combines 3D position, image-feature similarity, and class agreement:
$$
C_{\iota, i} = w_{\text{pos}} \min \left( \frac{||\hat{\mathbf{c}}^-_{\iota} - \mathbf{c}_{i,n}||_2}{d_{\max}}, 1 \right)
+ w_{\text{vis}} \left(1 - \cos(\hat{\mathbf{f}}^{\text{img}}_\iota, \mathbf{f}^{\text{img}}_{i,n})\right)
+ \delta_{\text{cls}} \mathbb{1}[y_\iota \neq y_{i,n}].
$$
Matches below threshold update tracks, unmatched detections initialize new tracks, and unmatched tracks are marked as disappeared; temporal edges then connect matched instances over time [2509.23107]. Spatial relations are predicted in open vocabulary, and ambiguous pairs are disambiguated with a geometric cost based on overlap, area ratio, and center distance.

A complementary line is represented by VOST-SGG, which injects VLM priors directly into one-stage spatio-temporal scene graph generation. Its dual-source query initialization separates semantic content from positional anchoring, so that queries encode both “what to attend to” and “where to attend,” while a multi-modal feature bank fuses visual, textual, and spatial cues for predicate classification [2512.05524]. The reported gains are especially notable in mean recall and rare predicates, and the formulation explicitly targets open-vocabulary object and predicate recognition. Relative to the teleoperation ST-OVSG formulation, this is a tighter coupling of VLM semantics with the decoder rather than with downstream planning.

Open-vocabulary spatio-temporal grounding is also relevant to ST-OVSG construction even when the end task is not graph prediction. Video-GroundingDINO formulates Open-Vocabulary Spatio-Temporal Video Grounding as text-conditioned localization of tubes and temporal segments, using frozen spatial grounding backbones and video-specific temporal modules; it surpasses prior methods in both closed-set and open-vocabulary evaluations and is explicitly described as an enabling technology for spatio-temporal scene graph construction [2401.00901]. This suggests that ST-OVSG can be assembled either by direct graph generation or by composing open-vocabulary grounding, temporal association, and relation inference modules.

## 4. World grounding, persistence, and temporal memory

A recurring limitation of frame-centric graph models is that they reason only over currently visible objects. Several systems address this by pushing ST-OVSG toward persistent world models. SNOW is a training-free, backbone-agnostic framework that clusters point clouds with HDBSCAN, uses SAM2-based segmentation guided by object proposals, encodes each object as a set of STEP tokens, and anchors all tokens in a global coordinate frame with a SLAM backend, yielding a queryable 4D Scene Graph [2512.16461]. In this formulation, open-vocabulary semantics comes from VLM-derived tokens, while geometry and temporal continuity come from point-cloud structure and SLAM alignment.

DovSG addresses the same persistence problem in long-term mobile manipulation through dynamic open-vocabulary 3D scene graphs that can be updated locally rather than reconstructed globally. It represents the scene as $G_t=\langle O_t,E_t\rangle$, uses vision-language models for open-vocabulary object detection, extracts low-level spatial relations from fused 3D geometry, and updates only the affected subgraph after environmental change [2410.11989]. The local update mechanism is central in dynamic settings because it keeps the graph operational during long-horizon task execution.

PredictiveGraphs extends the temporal dimension from persistence to prediction. Its edges are Perpetua$^*$ filters that encode temporal dynamics of object–receptacle relations, enabling Bayesian prediction of likely future states and future object locations in semi-static environments [2605.00121]. World Scene Graph Generation pushes further by formalizing world-centric scene graphs that include both observed and unobserved objects, and by proposing PWG, MWAE, and 4DST as three inductive biases for object permanence: last-known-state buffering, masked completion with associative retrieval, and differentiable temporal attention with 3D motion and camera-pose features [2603.13185]. A common misconception is that spatio-temporal scene graphs are simply 2D frame graphs with links across adjacent frames; these world-centric systems show that the stronger formulation requires persistent identity, global coordinates, occlusion reasoning, and, in some cases, prediction beyond the last observation.

## 5. Representative systems, datasets, and reported performance

Evaluation protocols for ST-OVSG-related systems are heterogeneous. Open-vocabulary detection papers report AP or mAP; image and video SGG papers report Recall@K and mean Recall@K; robot-oriented 3D graph systems report node and edge accuracy or task success rates; world-centric systems increasingly evaluate downstream reasoning. As a result, numerical comparisons are meaningful primarily within each task family.

| System | Core mechanism | Reported result |
|---|---|---|
| SGDN [2307.03339] | Scene-graph-based open-vocabulary object detection | 37.5% AP50 on COCO novel classes; 23.6% LVIS rare mAP; 31.1% overall mAP |
| STKET [2309.13237] | Spatial and temporal knowledge embedded into transformer cross-attention | mR@50 improves by 8.1%, 4.7%, and 2.1%; rare-relation R@50 rises from 9.98% to 23.98% |
| PGSG [2404.00906] | Image-to-graph sequence generation with VLMs | VG SGDet reaches 8.3 mR@100; PSG reaches 16.4 mR@100 and 11.3 mR@100 for novel predicates |
| VOST-SGG [2512.05524] | VLM-aided one-stage ST-SGG with dual-source queries and multimodal feature bank | predCLS R@10 76.1%, mR@10 46.5%; SGDET R@10 34.2%, mR@10 23.0% |
| ST-OVSG [2509.23107] | Latency-aware open-vocabulary 3D scene graph for teleoperation | 74 percent node accuracy on Replica; planning success rate 70.5 percent |
| TRaSER on SVG2 [2602.23543] | Trajectory-aligned token arrangement with dual resamplers | relation detection +15 to 20%, object prediction +30 to 40%, attribute prediction +15% over strongest open-source baselines |

The dataset question is equally important. Existing video graph benchmarks have historically been limited in vocabulary breadth and temporal scope, which partially explains the reliance on weak supervision, synthetic generation, or VLM prompting. SVG2 attempts to change the scale of supervision by introducing over 636K videos with 6.6M objects, 52.0M attributes, and 6.7M relations, together with 54,200 object classes and 35,300 unique relation classes, and TRaSER uses this resource to improve downstream video question answering by +1.5 to 4.6% absolute accuracy when its generated scene graphs are given to a VLM as an intermediate representation [2602.23543]. This suggests that the future empirical trajectory of ST-OVSG may depend as much on data construction pipelines as on model design.

## 6. Applications, misconceptions, and open problems

The most immediate application of ST-OVSG is latency-robust teleoperation. By attaching latency tags to graph states and retrieving the subgraph whose observation time best matches the operator’s command time, the planner can resolve local–remote state mismatches caused by transmission delay [2509.23107]. In robotics more broadly, structured 4D priors support embodied reasoning: SNOW presents the resulting 4DSG as a queryable world model through which VLMs can interpret spatial scene structure and temporal dynamics, while DovSG uses dynamic open-vocabulary 3D scene graphs to support long-term language-guided mobile manipulation in changing environments [2512.16461][2410.11989].

A second application is as an intermediate representation for downstream vision-language reasoning. PGSG reports gains when scene-graph-trained weights are transferred to visual question answering, visual grounding, and captioning, and TRaSER shows that explicit spatio-temporal scene graphs improve video question answering relative to video-only prompting or video plus VLM-generated graphs [2404.00906][2602.23543]. The common pattern is that explicit graph structure can externalize compositional relations that are only implicitly present in latent video or image embeddings.

Several misconceptions recur in this area. One is that open-vocabulary capability automatically implies robust relation generalization. In fact, relation-involved open-vocabulary SGG was introduced precisely because prior methods largely handled object openness while leaving novel relations near zero, motivating visual-concept alignment, relation-aware pretraining, and knowledge distillation [2311.10988]. Another is that better node semantics necessarily imply better graph quality. The teleoperation ST-OVSG reports 74 percent node accuracy on Replica, but its static edge accuracy is 0.67 compared with 0.88 for ConceptGraph, and its dynamic benchmark totals are 0.66 for nodes, 0.64 for spatial edges, and 0.61 for temporal edges, indicating that relation extraction and temporal consistency remain harder than open-vocabulary node recognition [2509.23107].

Open problems follow directly from these observations. Data scarcity remains a bottleneck, especially for temporally grounded open-vocabulary relations and for world-centric annotations under occlusion. Temporal persistence and future prediction are still not standard components of most graph pipelines, despite the advances of PredictiveGraphs and WSGG [2605.00121][2603.13185]. Finally, system design is split between direct graph generators, grounding-centric pipelines, training-free world models, and planner-facing graph memories. This suggests that the mature ST-OVSG stack may eventually combine all four: open-vocabulary grounding, persistent world alignment, explicit temporal memory, and task-adaptive graph serialization.

Source: https://www.emergentmind.com/topics/spatio-temporal-open-vocabulary-scene-graph-st-ovsg