Hierarchical Temporal Graphs

• Hierarchical temporal graphs are mathematical structures that capture evolving, multi-level relationships in dynamic systems through explicit temporal and spatial hierarchies.
• They employ methodologies such as self-supervised pooling, explicit scene decomposition, and latent structure inference to construct robust multi-scale representations.
• These graphs improve practical applications like activity recognition, temporal forecasting, and link prediction by leveraging techniques such as hyperbolic embeddings and hierarchical aggregation.

A hierarchical temporal graph is a mathematical structure designed to represent and reason about dynamic systems with interacting entities, where temporal evolution is accompanied by explicit or implicit multi-level organization—such as partonomies in actions, event hierarchies, abstraction in scene structure, or tree-like information propagation. These graphs have gained importance in computer vision, time series forecasting, temporal link prediction, spatial-temporal reasoning, and dynamic knowledge graph modeling.

1. Formal Structures and Definitions

Hierarchical temporal graphs (HTGs) generalize temporal graphs by endowing them with a hierarchy, usually realized in one or more of the following ways:

• Temporal hierarchy: Multi-scale aggregation of events or interactions across distinct time-scales. For example, in "TimeGraphs," input frame-level graphs are recursively pooled to produce higher-level event nodes, reflecting the uneven distribution of information over time (Maheshwari et al., 2024).
• Structural hierarchy: Nested or multiresolution spatial/relational groupings. A prominent instantiation is multi-level scene graphs (objects → rooms → buildings) in 3D/4D scene representations (Catalano et al., 10 Dec 2025), or variable aggregation levels in hierarchical forecasting graphs (Sriramulu et al., 2024).
• Latent or statistical hierarchy: In models like THERGM, evolving community/cluster structure is modeled at multiple levels with probabilistic generative processes (Cao, 2017).
• Hypergraph hierarchy: HTGs may use hypergraphs at each layer, where nodes (fine-grained) are grouped into hyperedges and successively pooled, e.g., in hierarchical hypergraph transformer models for time series (Wang et al., 4 Aug 2025).

Time is modeled either as discrete snapshots (e.g., $\mathcal{G}_t$) or continuous intervals, with temporal edges, node trajectories, or explicit evolutionary linkage between layers and timesteps. Hierarchical edges connect entities across levels (e.g., parent-child, supernode-constituent).

The following table summarizes representative formalizations:

Framework	Node Types	Hierarchy	Temporal Mechanism
Action Genome (Ji et al., 2019)	Actor, object per frame	Event partonomy (action sub-steps)	Sequence of spatio-temporal graphs
TimeGraphs (Maheshwari et al., 2024)	Entities, super-events	Event hierarchy (multi-level)	Streaming, frame-to-frame, multi-scale pooling
THERGM (Cao, 2017)	Nodes, clusters (latent)	Clustering	Markovian transitions on labels, TERGM within clusters
DeepHGNN (Sriramulu et al., 2024)	Series at each hierarchy level	Cross-sectional aggregation	Graph GNNs propagate in time and aggregation layers
HGTS-Former (Wang et al., 4 Aug 2025)	Time series patches, hyperedges	Multi-level hypergraph	Self-attention, hierarchical hypergraphs per patch/channel

2. Methodologies for Constructing and Learning HTGs

Construction of hierarchical temporal graphs proceeds via one or more of the following algorithmic approaches:

• Self-supervised hierarchical pooling: "TimeGraphs" constructs event hierarchies by maximizing mutual information between pooled super-nodes and their local neighborhoods. The VIPool procedure greedily selects node sets optimizing an MI criterion and recursively pools to form higher-level event nodes. Hierarchy edges are introduced to link super-nodes with their constituents (Maheshwari et al., 2024).
• Explicit scene/decomposition schemes: "Action Genome" samples and annotates per-frame scene graphs, constructing a temporal sequence. Hierarchical structure arises by relating trajectories of actor-object relationships over sampled frames; the evolution of these relationships encodes sub-events (Ji et al., 2019).
• Latent structure inference: THERGM posits a Markov chain on cluster labels and uses a dynamic latent space model for clustering, followed by cluster-specific temporal ERGMs (Cao, 2017).
• Hypergraph-based aggregation: HGTS-Former uses two-level hierarchical hypergraphs: intra-channel hyperedges (patch-level grouping within a variable), then inter-channel hyperedges (across variables), both constructed by attention and incidence masking (Wang et al., 4 Aug 2025).
• Propagative GNNs with hierarchical edges: DeepHGNN builds a block-diagonal graph with both temporal (sequential) and hierarchical (cross-sectional, parent-child) edges, enabling message passing and end-to-end reconciliation across all levels (Sriramulu et al., 2024).

Temporal dependencies are modeled via:

• Temporal message passing (e.g., RNNs, GRUs across time in THERGM and TNA (Bonner et al., 2019)),
• Temporal edges or recurrent cell construction,
• Temporal convolution (e.g., dilated convolutions in HGWaveNet (Bai et al., 2023)),
• Explicit evolutionary mechanisms (e.g., Markov or autoregressive models in HyperVC (Sohn et al., 2022)).

3. Mathematical Properties and Geometric Embeddings

HTGs capture complex topological and hierarchical properties, often leveraging hyperbolic geometry due to its exponential volume growth and suitability for tree-like, scale-free networks:

• Hyperbolic dynamic embeddings: Both HGWaveNet and HTGN embed evolving graph snapshots in hyperbolic space (Poincaré ball $\mathbb{B}_c^n$) using Möbius addition, exponential/logarithmic maps, and distance to capture global and local hierarchies (Yang et al., 2021, Bai et al., 2023).
• Variable curvature and TKG chronologies: HyperVC models entire snapshot chronologies and their internal hierarchies by assigning variable hyperbolic curvatures to each snapshot graph and embedding all graphs in a shared or time-dependent space; this approach captures both global event branching and local hierarchical degree (Sohn et al., 2022).
• Clustering and aggregation operators: DeepHGNN constructs a summing matrix $S^{(\ell)}$ to pool lower-level series into higher-level aggregates, ensuring exact hierarchical aggregation in forecasting applications (Sriramulu et al., 2024). In THERGM, cluster transitions obey Markovian structure with preferential attachment for new cluster entries (Cao, 2017).
• Streaming/incremental graph construction: TimeGraphs and Aion construct or update HTGs in a streaming and online manner, introducing partial hierarchies as new frame-level observations arrive and updating higher-level event nodes incrementally (Maheshwari et al., 2024, Catalano et al., 10 Dec 2025).

4. Representative Model Architectures and Algorithms

Several model classes have operationalized HTGs for learning and inference, including:

• Scene Graph Feature Bank (SGFB): Integrates CNN-extracted spatio-temporal features with per-frame scene-graph confidence matrices using a Feature Bank Operator, producing hierarchical spatio-temporal summaries for improved activity recognition (Ji et al., 2019).
• Temporal Neighbourhood Aggregation (TNA): Stacks hierarchical GRU blocks for each graph neighborhood depth, maintaining time-recursive hidden states, and optimizing embeddings via a variational ELBO for future link prediction (Bonner et al., 2019).
• HGWaveNet and HTGN: Employ hyperbolic graph convolutions (HGCN), diffusion or attention-based temporal modules (HDGC, HDCC, HTA), and hyperbolic RNNs (HGRU) to capture both temporal evolution and implicit or explicit hierarchy (Yang et al., 2021, Bai et al., 2023).
• HGTS-Former: Constructs two-level hierarchical hypergraphs for time series patches, applies masked cross-attention and multi-head self-attention, with edge-to-node modules for feature projection and iterative block stacking for deep temporal reasoning (Wang et al., 4 Aug 2025).
• Hierarchical Exponential Random Graph models (THERGM): Two-stage estimation process: dynamic latent space modeling for cluster assignment, then cluster-specific TERGM fitting, capturing evolving community structure and local network features (Cao, 2017).

5. Applications and Empirical Impact

Hierarchical temporal graphs are foundational in a range of tasks where temporal and hierarchical dependencies must be captured:

• Activity and event recognition: Action Genome demonstrates that decomposition into spatio-temporal scene-graph sequences improves standard and few-shot activity classification; symbolic hierarchical priors yield notable mAP gains over long-term feature banks, especially with few examples (e.g., 42.7% mAP with k=10 for few-shot) (Ji et al., 2019).
• Temporal reasoning and dynamic event segmentation: TimeGraphs achieves up to 12.2% improvement in event prediction, robust zero-shot generalization, and streaming adaptability, with graceful degradation under data sparsity (Maheshwari et al., 2024).
• Hierarchical multivariate forecasting: DeepHGNN leverages cross-level aggregation and end-to-end reconciliation to enforce forecast coherence and accuracy, outperforming dominant frequency- and graph-based hierarchical methods in WAPE and MASE metrics across multiple benchmark datasets (Sriramulu et al., 2024).
• Link prediction and graph evolution: Hyperbolically embedded models (HGWaveNet, HTGN, HyperVC) provide substantial AUC/ap improvements in link prediction on datasets with high latent hierarchy, confirming the crucial role of geometry in modeling evolving graph structure (Bai et al., 2023, Yang et al., 2021, Sohn et al., 2022).
• Spatiotemporal planning and navigation: Aion’s 4D hierarchical scene graphs with flow descriptors enable interpretable motion prediction and entropy-aware path planning, drastically reducing computational and memory costs relative to uniform grid baselines (Catalano et al., 10 Dec 2025).
• Clustering and evolving communities: THERGM reflects the need for explicit cluster evolution models; the two-stage latent space + ERGM approach excels in recovery and link-prediction accuracy even with rapidly evolving or ambiguous community structure (Cao, 2017).

6. Current Challenges and Interpretative Insights

• Geometry selection and distortion: Multiple works demonstrate the superiority of hyperbolic over Euclidean geometry for graphs with tree-like, power-law structure. When the Krackhardt hierarchical score (Khs) or other measures indicate strong latent hierarchy, embedding in hyperbolic space with fixed or variable curvature is essential for reducing distortion and supporting downstream tasks (e.g., with improvements of 8 points in Hits@1 on WIKI in HyperVC) (Sohn et al., 2022).
• Balance of efficiency and accuracy: Streaming construction and hierarchical pooling—via self-supervised methods or sparse, online representation (TimeGraphs, Aion)—address the challenge of computation over long or densely sampled temporal graphs without sacrificing multiscale information.
• End-to-end reconciliation: Ensuring forecast and inference coherence across all hierarchical levels is addressed by mechanisms such as differentiable pooling/unpooling (DeepHGNN), hierarchical cross-network regularization (TimeGraphs), or explicit loss terms.

A plausible implication is that the optimal structure—temporal, structural, or combined—of the hierarchy may itself be time-varying, data-dependent, or task-adaptive, motivating research directions in dynamic or neural architecture-driven hierarchy selection.

7. Summary Table: Exemplars of Hierarchical Temporal Graph Modeling

Paper/Model	Hierarchy Type	Temporal Structure	Core Task(s)	Key Quantitative Result
Action Genome (Ji et al., 2019)	Action partonomy	Spatio-temporal scene-graph sequence	Recognition, few-shot activity	+17.8% mAP with oracle graphs (Charades)
TimeGraphs (Maheshwari et al., 2024)	Event multiscale	Self-supervised frame→event hierarchy	Temporal reasoning, streaming	+12.2% event EM, robust under sparsity
DeepHGNN (Sriramulu et al., 2024)	Aggregation/forecasting	Inter-level & temporal GNN	Hierarchical time series forecasting	~7% WAPE reduction over DPMN
HGWaveNet/HTGN (Yang et al., 2021, Bai et al., 2023)	Implicit, geometry-induced	Hyperbolic GNNs, temporal convolution	Dynamic link prediction	+6.67–11.4% AUC (vs. Euclidean)
THERGM (Cao, 2017)	Community clustering	Markov chain on clusters, per-cluster TERGM	Dynamic clustering, link prediction	AUC > 0.9 when misclustering <10%
Aion (Catalano et al., 10 Dec 2025)	4D spatial semantics	Temporal flows via sparse MoD	Navigation, path planning	O(1) update, grid-level accuracy, low memory

These models collectively demonstrate the power and flexibility of hierarchical temporal graph frameworks for capturing the richness of spatio-temporal, relational, and multi-scale temporal data across domains of activity understanding, reasoning, representation learning, and statistical forecasting.