THYME: Temporal Hierarchical Cyclic Scene Graph

• THYME is a scene graph formulation for video understanding that hierarchically organizes object trajectories to capture dynamic interactivity over time.
• It integrates multi-scale intra-frame aggregation with cyclic temporal attention to maintain fine-grained spatial detail and robust temporal consistency.
• Evaluated on both ground-view and aerial datasets, THYME outperforms competitors in relation recall and handles occlusion and clutter more effectively.

Temporal Hierarchical Cyclic Scene Graph (THYME) denotes a scene-graph formulation for video understanding in which object-centric representations are organized hierarchically within frames and refined cyclically across time to improve temporal coherence. In the work that formally introduces THYME, the method is presented as Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graph Generation (VidSGG) in both ground-view and aerial footage. Given a video $\{I_t\}_{t=1}^{T}$, THYME produces a video scene graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ whose nodes are object trajectories with unified features and whose edges carry predicates spanning five interactivity types: Appearance, Situation, Position, Interaction, and Relation (Nguyen et al., 12 Jul 2025).

1. Formalization and problem setting

THYME is situated in Video Scene Graph Generation, where the objective is not merely to infer frame-local object relations but to maintain a temporally coherent relational description over an entire clip. In its formulation, each node $S_i \in \mathcal{V}$ is an object trajectory rather than an isolated frame detection, and each edge $(S_i,S_j)\in \mathcal{E}$ is labeled with a predicate drawn from a predicate set for one of the five interactivity types. A video scene graph is therefore treated as an implicit sequence of per-frame graphs $G_t=(V_t,E_t)$ whose node and edge identities must remain consistent over time (Nguyen et al., 12 Jul 2025).

The method is motivated by three difficulties that are especially acute in aerial footage: fine-grained spatial detail, long-range temporal modeling, and temporal coherence. Frame-level scene graph generation methods are described as strong in per-frame spatial reasoning but weak in temporal consistency, which leads to predicate flickering and fragmented relational narratives. Video-level methods improve long-term consistency, but often blur fine spatial structure and micro-interactions, particularly when objects are small, viewpoints shift sharply, or clutter is severe. THYME is designed to address these issues jointly by combining multi-scale spatial hierarchy with cyclic temporal refinement (Nguyen et al., 12 Jul 2025).

A central aspect of the formulation is its decomposition of scene semantics into five interactivity types. Appearance and Situation are single-actor descriptions; Position, Interaction, and Relation are double-actor predicates. This decomposition gives THYME a broader representational scope than formulations limited to pairwise spatial or action predicates alone. A plausible implication is that the model is intended not only to localize objects and relations but also to capture environmental context and higher-level semantic ties within the same graph formalism.

2. Network architecture and scene-graph construction

THYME’s pipeline has four stages: per-frame feature extraction, hierarchical feature aggregation, temporal refinement via cyclic attention, and scene graph construction (Nguyen et al., 12 Jul 2025).

At the first stage, each frame $I_t$ is processed with DETR, which produces object query embeddings

$\{q_i^{(t)} \in \mathbb{R}^{d_0}\}_{i=1}^{N_t}.$

These are treated as level-0 object features,

$\mathcal{F}_t^{(0)}(S_i^{(t)}) = q_i^{(t)}.$

The nodes in frame $t$ are therefore DETR queries interpreted as scene-graph object instances.

Within each frame, THYME performs attention-based hierarchical aggregation. For hierarchy level $l$, the neighborhood of each object is the full object set of the frame, $\mathcal{G}=(\mathcal{V},\mathcal{E})$0. Attention weights are computed as

$\mathcal{G}=(\mathcal{V},\mathcal{E})$1

and the updated representation is

$\mathcal{G}=(\mathcal{V},\mathcal{E})$2

with learnable $\mathcal{G}=(\mathcal{V},\mathcal{E})$3, $\mathcal{G}=(\mathcal{V},\mathcal{E})$4, and ReLU nonlinearity. The resulting hierarchy moves from raw object-query features toward increasingly context-enriched intra-frame representations (Nguyen et al., 12 Jul 2025).

For relation prediction, THYME reuses self-attention outputs from the DETR decoder as relation candidates. Multi-layer relation representations are built from projected query and key matrices, then fused with a gating mechanism derived from EGTR. The final relation score tensor is

$\mathcal{G}=(\mathcal{V},\mathcal{E})$5

where $\mathcal{G}=(\mathcal{V},\mathcal{E})$6 and $\mathcal{G}=(\mathcal{V},\mathcal{E})$7 encode subject-object relation features from intermediate and final decoder layers, and $\mathcal{G}=(\mathcal{V},\mathcal{E})$8, $\mathcal{G}=(\mathcal{V},\mathcal{E})$9 are gate activations. Nodes are thus unified object trajectories, while edges are ordered object-pair predicates scored from gated DETR relational features.

3. Hierarchical aggregation and cyclic temporal refinement

THYME’s defining contribution is the coupling of hierarchical intra-frame modeling with cyclic temporal attention (Nguyen et al., 12 Jul 2025).

After hierarchical aggregation, each trajectory $S_i \in \mathcal{V}$0 is represented as a temporal sequence

$S_i \in \mathcal{V}$1

These features are projected into queries, keys, and values, and temporal refinement is performed by

$S_i \in \mathcal{V}$2

where the attention weights also use modulo $S_i \in \mathcal{V}$3 indexing. This modulo operation is the mechanism that makes the temporal structure cyclic: the last time step can attend to the first, and every step attends over a temporal ring rather than a line. The paper characterizes this as a directed cyclic structure that preserves sequence directionality, mitigates boundary effects, and supports long-range periodic or recurrent patterns (Nguyen et al., 12 Jul 2025).

The hierarchical component addresses the spatial side of the problem. At $S_i \in \mathcal{V}$4, features are raw DETR queries carrying local appearance and coarse spatial cues. As levels deepen, each object representation aggregates evidence from all objects in the frame, allowing it to encode broader context such as latent scene structure, relative positions, and group-level organization. This is especially important in aerial scenes, where objects are small but also distributed over large spatial extents.

The model does not introduce a separate temporal smoothness loss in the form of an explicit $S_i \in \mathcal{V}$5. Instead, temporal consistency is induced by the cyclic transformer encoder and by the requirement that refined trajectory features support correct predicate classification over the full clip. A common misconception is to equate “cyclic” with explicit graph cycles over time-indexed scene copies. In THYME, cyclicity is realized in the attention operator through modulo indexing, not through a separate time-layered graph topology. This suggests a design in which cyclic temporal refinement is a property of feature propagation rather than of discrete graph duplication.

4. Interactivity taxonomy and benchmarking corpora

THYME adopts a five-part interactivity schema and evaluates it on two datasets: ASPIRe for ground-view video and AeroEye-v1.0 for aerial video (Nguyen et al., 12 Jul 2025).

The five interactivity types are as follows. Appearance denotes single-actor visual attributes such as vehicle type, color, or shape. Situation denotes single-actor contextual attributes such as scene type, weather, time, or events. Position denotes double-actor spatial relations such as “in front of” or “behind.” Interaction denotes dynamic, action-like double-actor relations such as “approaching,” “overtaking,” or “towing.” Relation denotes higher-level functional or semantic ties such as “assisting,” “escorting,” or “guiding.” In the graph formalism, Position, Interaction, and Relation are edge labels over ordered object pairs, while Appearance and Situation are treated as per-node labels or unary attributes.

AeroEye-v1.0 is introduced as a major contribution of the THYME work. It contains 2,260 drone videos, 261,503 frames, 56 object categories, and over 2 million bounding boxes with tracking. It is annotated with 157 appearance predicates, 128 situation predicates, 135 Position predicates with approximately 752K annotations, 142 Interaction predicates with approximately 318K annotations, and 125 Relation predicates with approximately 178K annotations. The dataset includes aerial, oblique, and ground perspectives, and is described as the only aerial VidSGG dataset with all five interactivity types fully annotated per frame (Nguyen et al., 12 Jul 2025).

ASPIRe provides a complementary ground-view benchmark with approximately 1.5K videos, 1.6M frames, 833 object classes, and 4.5K relation classes, again annotated for the same five interactivity types. Evaluation follows established VidSGG protocol with Recall and mean Recall at $S_i \in \mathcal{V}$6, reported separately for each interactivity type. The separation by type is important because it exposes whether gains arise from unary appearance-context recognition, binary spatial reasoning, or more difficult long-tail interaction semantics.

5. Experimental results and ablations

On ASPIRe, THYME reports the following R/mR@20 values: Appearance 18.23 / 1.07, Situation 6.57 / 0.26, Position 18.52 / 1.22, Interaction 19.52 / 0.32, and Relation 21.02 / 1.12. On AeroEye-v1.0, the corresponding values are Appearance 16.52 / 0.68, Situation 5.53 / 0.61, Position 15.52 / 1.05, Interaction 13.07 / 0.16, and Relation 16.03 / 0.95. In the reported comparisons, these scores exceed those of STTran, TEMPURA, HIG, and CYCLO, with especially notable improvements over CYCLO on aerial Position, Interaction, and Relation recall (Nguyen et al., 12 Jul 2025).

The ablation studies isolate the contributions of hierarchy and cyclic temporal modeling. When hierarchical depth is varied on AeroEye-v1.0, Appearance R@20 rises from 14.12 at 1/4 hierarchy to 16.52 at Full, Position R@20 rises from 12.32 to 15.52, and Relation R@20 rises from 13.03 to 16.03. This monotonic pattern indicates that deeper hierarchical aggregation materially improves double-actor reasoning. A parallel ablation replacing cyclic attention with standard attention shows Appearance R@20 15.12 $S_i \in \mathcal{V}$7 16.52, Position 13.42 $S_i \in \mathcal{V}$8 15.52, and Relation 14.03 $S_i \in \mathcal{V}$9 16.03, confirming that cyclic attention is not merely architectural ornamentation but a substantive contributor to performance (Nguyen et al., 12 Jul 2025).

Temporal window size also matters. The reported experiments compare 1/2, 3/4, and full temporal windows, with the full window yielding the best or near-best results. This is consistent with the model’s emphasis on long-range context. Qualitative analyses reinforce the quantitative picture. In ground-view basketball sequences, THYME maintains a more robust trajectory representation through occlusion than HIG or CYCLO. In aerial chase scenarios, it better captures relation changes such as “chasing” $(S_i,S_j)\in \mathcal{E}$0 “approaching” $(S_i,S_j)\in \mathcal{E}$1 “crashing,” whereas HIG and CYCLO are reported to miss or mis-update those transitions (Nguyen et al., 12 Jul 2025).

6. Position within temporal scene-graph research

THYME belongs to a broader family of temporal scene-graph methods, but its particular combination of hierarchical intra-frame aggregation and cyclic temporal attention distinguishes it from adjacent lines of work.

In robotics-oriented mapping, Aion embeds temporal flow dynamics into a hierarchical 3D scene graph by attaching FreMEn-based motion descriptors to navigational nodes, thereby producing a hierarchical 4D scene graph with implicit cyclic temporal modeling. Its cycles are periodic patterns encoded in node-level temporal models rather than explicit temporal graph cycles (Catalano et al., 10 Dec 2025). Hi-Dyna Graph separates a persistent global static graph from dynamic local subgraphs built from video streams, with temporally segmented relations and a sliding-window update regime, but it does not explicitly model temporal cycles or long-term memory beyond that window (Hou et al., 30 May 2025).

In predictive video scene-graph research, SceneSayer formulates Scene Graph Anticipation as future graph prediction and uses NeuralODE and NeuralSDE to model the latent evolution of object-pair relations in continuous time. This gives a strong continuous-time treatment of temporal dynamics, but the hierarchy described there is latent and prospective rather than the explicit hierarchical intra-frame aggregation used by THYME (Peddi et al., 2024). At the same time, the older (2.5+1)D Spatio-Temporal Scene Graphs framework organizes static and dynamic subgraphs in a shared pseudo-3D space and reasons in a spatio-temporal hierarchical latent space, yet it does not discuss cyclic reasoning explicitly (Cherian et al., 2022).

A different, robotics-facing trajectory appears in Predictive Spatio-Temporal Scene Graphs for Semi-Static Scenes, where Perpetua$(S_i,S_j)\in \mathcal{E}$2 filters are attached to object-receptacle edges in a 3D scene graph to learn periodic behaviors such as daily object relocation routines. There, cyclicity is explicit in the temporal prior over edge activity, often using FreMEn or LLM-derived schedules, but the focus is semi-static object-state prediction rather than VidSGG over dense frame sequences (Saavedra-Ruiz et al., 30 Apr 2026). Taken together, these works suggest that “temporal hierarchical cyclic scene graph” now names a broader design space spanning vision-centric VidSGG, dynamic robotic world models, and predictive spatio-semantic mapping. THYME occupies the video-centric end of that space.

7. Limitations and future directions

The THYME paper identifies several open problems. Occlusion and missing detections remain failure modes even with cyclic attention and hierarchy. Dense crowds and clutter in aerial views continue to challenge both detection and relation recognition. Very long sequences remain computationally expensive, since temporal modeling still grows with sequence length even when features are pooled or sub-clipped (Nguyen et al., 12 Jul 2025).

The reported future directions include multimodal cues such as audio and text, bias reduction and stronger handling of long-tail predicates, and domain adaptation beyond the current ground-view and aerial settings. A plausible implication is that future THYME-like systems may combine cyclic temporal refinement with richer multimodal grounding, or integrate explicit event and memory structures when minute- or hour-scale temporal reasoning becomes necessary.

Within the broader literature, a second frontier concerns the relationship between implicit cyclic refinement and explicit temporal graph structure. Robotics systems such as Aion, Hi-Dyna Graph, and PredictiveGraphs already attach temporal processes to semantic graph elements, but often at the level of navigational nodes, relation intervals, or object-receptacle edges rather than full video-trajectory graphs (Catalano et al., 10 Dec 2025, Hou et al., 30 May 2025, Saavedra-Ruiz et al., 30 Apr 2026). This suggests that future work may converge on representations that combine THYME’s cyclic trajectory refinement with longer-horizon temporal memory, explicit event abstractions, and graph-theoretic planning or anticipation.