---
title: Spatiotemporal Multigraph Representation
url: https://www.emergentmind.com/topics/spatiotemporal-multigraph-representation
type: topic
---

# Spatiotemporal Multigraph Representation

Spatiotemporal multigraph representation denotes a family of graph formalisms in which spatial structure, temporal evolution, and their coupling are encoded in a single learnable object rather than separated into an ordinary graph plus an external sequence model. In the literature represented here, that object appears in several mathematically distinct forms: as \(M\) modality-specific graphs \(G^{(m)}=(V,E^{(m)})\) over a common node set; as a third-order adjacency tensor \(\mathcal A\in\mathbb R^{N\times N\times T}\); as a product graph on \(V_t\times V_s\); as a block adjacency matrix augmented with learned temporal links; and as a continuous-time event sequence \(\mathcal E=\{(u_i,v_i,t_i,e_i)\}_{i=1}^M\) in which repeated timestamped interactions define a multigraph. The shared purpose is to support representation learning, forecasting, link prediction, detection, or graph inference while preserving spatial dependencies, temporal dependencies, and interdependence between them [1905.11395, 2408.02704, 2206.15174, 2310.02606, 2407.16959].

## 1. Formal scope and mathematical definitions

A canonical discrete multigraph formulation uses a fixed vertex set \(V\) with \(N\) regions and \(M\) relation modalities. Each modality induces a graph
\[
G^{(m)}=(V,E^{(m)}), \qquad A^{(m)}\in\mathbb R^{N\times N}, \quad m=1,\dots,M,
\]
with degree matrix \(D^{(m)}=\mathrm{diag}(A^{(m)}\mathbf 1)\) and symmetric normalized Laplacian
\[
L^{(m)}=I-(D^{(m)})^{-1/2}A^{(m)}(D^{(m)})^{-1/2}.
\]
A time-indexed signal may be a scalar observation \(x_t\in\mathbb R^N\), node features \(X_t\in\mathbb R^{N\times F}\), or a lag stack
\[
X_{t-K+1:t}=[X_{t-K+1},\dots,X_t]\in\mathbb R^{K\times N\times F}.
\]
This formulation is explicit in urban forecasting, where neighborhood, POI-similarity, and road-connectivity are modeled as separate spatial modalities [1905.11395].

A second formulation treats the entire evolving graph as a tensor. In STGCNDT, a dynamic graph is represented by a third-order adjacency tensor
\[
\mathcal A\in\mathbb R^{N\times N\times T}, \qquad \mathcal A_{:,:,t}=A^{(t)},
\]
and a node-feature tensor
\[
\mathcal X\in\mathbb R^{N\times F\times T}, \qquad \mathcal X_{i,:,t}=x_i^{(t)}.
\]
The first two modes are spatial and the third mode is temporal. In GVNN, a related tensorial object is the graph-variate tensor
\[
\Omega_{:,:,t}=\Omega(t)=S\circ J(t),
\]
where \(S\in\mathbb R^{N\times N}\) is a stable support and \(J(t)\) is an instantaneous connectivity matrix produced by a node-pair function \(F_V(x_i(t),x_j(t))\) [2408.02704, 2509.20311].

A third formulation expands the node set itself. In graph-time models, one defines a spatial graph \(G_s=(V_s,E_s)\) with \(|V_s|=N\) and a temporal graph \(G_t=(V_t,E_t)\) with \(|V_t|=T\), then forms a product graph \(G_p\) with node set \(V_t\times V_s\), hence \(|V_p|=NT\). In GL-LRSS, the same idea appears as a space-time multigraph with nodes \((i,t)\), spatial edges \((i,t)\)–\((j,t)\), and temporal edges \((i,t)\)–\((i,t-1)\), represented by
\[
A_{\rm ST}=I_M\otimes W + W_T\otimes I_N,
\qquad
L_{\rm ST}=I_M\otimes L + L_T\otimes I_N.
\]
This suggests that “multigraph” in this literature may refer either to multiple edge modalities on a common node set or to a lifted graph whose vertices are space-time pairs [2206.15174, 1911.08018].

A fourth formulation is explicitly event-based. In continuous-time dynamic graphs, the interaction sequence
\[
\mathcal E=\{(u_i,v_i,t_i,e_i)\}_{i=1}^M
\]
is interpreted as a spatiotemporal multigraph because a node pair may interact multiple times at different timestamps. In event-based object detection, a multigraph is written directly as
\[
G=(V,E_s,R_s,E_t,R_t),
\]
with decoupled spatial edges \(E_s\) and temporal edges \(E_t\) coexisting over the same event-node set \(V\) [2407.16959, 2507.15150].

## 2. Principal construction paradigms

The literature uses several recurring constructions to make spatiotemporal dependencies learnable.

| Construction | Joint object | Characteristic use |
|---|---|---|
| Multi-modal graph family | \(\{G^{(m)}\}_{m=1}^M\) | multiple spatial relations |
| Dynamic graph tensor | \(\mathcal A\in\mathbb R^{N\times N\times T}\) | time-indexed adjacency slices |
| Product graph | \(V_t\times V_s\) | explicit space-time coupling |
| Mended block adjacency | \(A_B^M\) | learned temporal off-diagonal links |
| Decoupled event multigraph | \(G=(V,E_s,R_s,E_t,R_t)\) | separate spatial and temporal edge sets |

In multi-modal urban forecasting, the defining construction is a family of graphs sharing one node set but differing by modality. Lower layers permit inter-modality convolutions across ordered pairs \((i\to j)\), whereas higher layers retain only intra-modality outputs while imposing a tensor-normal prior across modalities. Final prediction is obtained by averaging the modality-wise scalar outputs,
\[
\hat x_{t+1}=\frac1M\sum_{m=1}^M H^{(L)}_m.
\]
The same work states that the input uses the past \(K=5\) slices \(\{x_{t-4},\dots,x_t\}\) to capture closeness, period, and trend [1905.11395].

In graph-time formulations, the core construction is the product graph. Kronecker, Cartesian, and strong products yield
\[
A_K=A_t\otimes A_s,\qquad
A_C=A_t\otimes I_N + I_T\otimes A_s,\qquad
A_S=A_C+A_K.
\]
A parametric product graph introduces four learnable scalars,
\[
S_\theta=\theta_{00}(I_T\otimes I_N)+\theta_{01}(I_T\otimes A_s)+\theta_{10}(A_t\otimes I_N)+\theta_{11}(A_t\otimes A_s),
\]
or equivalently \(A_\theta\), enabling the spatiotemporal coupling itself to be learned from data [2206.15174, 2103.01730].

A contrasting strategy starts from a disconnected union of time-indexed graphs. The block adjacency matrix
\[
A_B=
\begin{pmatrix}
A^{(1)} & 0 & \cdots & 0\\
0 & A^{(2)} & \cdots & 0\\
\vdots & \vdots & \ddots & \vdots\\
0 & 0 & \cdots & A^{(T)}
\end{pmatrix}
\]
contains only intra-time spatial edges. “Mending” augments it by a feature-aware projection and Transformer encoder, then symmetrizes and rectifies:
\[
A_B^M=\mathrm{ReLU}\!\Bigl(A_B+\tfrac12(A_{B,\mathrm{enc}}+A_{B,\mathrm{enc}}^\top)\Bigr).
\]
The resulting graph has learned temporal edges in the off-diagonal blocks, and the paper reports that the original block adjacency has \(T\) zero Laplacian eigenvalues, whereas the modified graph has exactly one zero eigenvalue and a strictly positive Fiedler value [2310.02606].

Event-based asynchronous vision uses an explicitly decoupled multigraph. Spatial neighbors satisfy
\[
\frac{\|v_i^{xy}-v_j^{xy}\|}{R_{XY}+|t_i-t_j|/R_t}<1,\qquad t_i<t_j,
\]
with \(|N_s(i)|\le 16\), \(R_{XY}=0.04\cdot(\text{image size})\), and \(R_t=5\text{ ms}\). Temporal neighbors instead use an inverted ellipsoid with semi-major axis along time, \(R_t=40\text{ ms}\), semi-minor axis \(R_{XY}=0.01\cdot\text{size}\), and \(|N_t(i)|\le 12\). This separation is then justified computationally by replacing a 3D spline kernel with a 2D anisotropic spline plus temporal attention [2507.15150].

Further variants specialize the multigraph to the data domain. DSTGNN constructs a spatial tensor graph \(\mathcal A\in\mathbb R^{N\times N\times T}\) and a temporal tensor graph \(\mathcal B\in\mathbb R^{T\times T\times N}\), then entangles them through PEPS. ST-GraphRL uses a weighted directed graph \(G=(V,E,W)\) whose edge tensor stores
\[
w_{ij}=[f_{ij},d_{ij},\tau_{ij}]\in\mathbb R^3,
\]
that is, movement frequency, spatial distance, and average duration. The paper states that this directed graph with edge attributes captures both spatial and temporal characteristics of human movements as a multigraph [2003.08729, 2312.04055].

## 3. Convolution, message passing, and fusion operators

Spatiotemporal multigraph representation is tightly linked to the operator used to propagate information on the joint structure. In the multi-modal Chebyshev setting, grouped GCN defines a weight tensor for every ordered modality pair,
\[
W^l_{i,j}\in\mathbb R^{K\times f_l\times f_{l+1}},
\]
and updates modality \(j\) by
\[
H^{(l+1)}_j=
\sigma\Biggl(
\sum_{i=1}^M\sum_{\alpha=0}^K
\bigl(L^{(i)}\bigr)^\alpha H_i^{(l)}W^l_{i,j}[\alpha]+b_j^l
\Biggr).
\]
To avoid overfitting over the \(O(M^2)\) parameter blocks, the model uses a grouped \(\ell_2\) penalty
\[
J_1^l=\alpha\sum_{i=1}^M\|W_{i,i}^l\|_2+\sum_{i\ne j}\|W_{i,j}^l\|_2,
\]
with tunable \(\alpha<1\). Higher layers switch to a multi-linear-relationship GCN whose weights are collected in a four-way tensor \(W^l\in\mathbb R^{f_l\times f_{l+1}\times K\times M}\) with tensor-normal prior \(\Sigma_I\otimes\Sigma_O\otimes\Sigma_C\otimes\Sigma_M\) [1905.11395].

Tensorized dynamic-graph models replace graph polynomials by tensor products. STGCNDT defines an M-product
\[
\mathcal X\ast_M\mathcal Y
=
\bigl((\mathcal X\times_3 M)\odot(\mathcal Y\times_3 M)\bigr)\times_3 M^{-1},
\]
and a GTCN layer
\[
\mathcal H=\sigma\bigl(\mathcal A\ast_M\mathcal X\ast_M\mathcal W\bigr).
\]
The transform \(M\) is the temporal filter. The paper studies DFT, DCT, and HWT, and ensembles their outputs by
\[
\mathcal H=\alpha\,\mathcal H^{\rm DFT}+\beta\,\mathcal H^{\rm DCT}+\gamma\,\mathcal H^{\rm HWT},
\qquad
\alpha+\beta+\gamma=1,
\]
with \(\alpha=\beta=\gamma=1/3\) in the experiments [2408.02704].

Product-graph approaches instead retain the ordinary shift-and-sum principle on the enlarged graph. With a product-graph shift \(S_p\), a \(K\)-th-order filter is
\[
h(S_p)=\sum_{k=0}^K\alpha_k S_p^k,
\]
and for multi-feature input the output feature \(y^{(g)}\) is
\[
y^{(g)}=\sum_{f=1}^{F_{\rm in}}\sum_{k=0}^K \alpha_k^{(g,f)}S_p^k x^{(f)}.
\]
GTCNN layers then combine graph-time convolution, zero-pad pooling, and nonlinearity; the parametric coefficients \(s_{ij}\) in the product graph are learned jointly with the filter taps, with an \(\ell_1\)-penalty on \(s_{ij,\ell}\) to induce sparsity [2206.15174, 2103.01730].

Other models keep spatial and temporal aggregation separate but fuse them explicitly. In eGSMV, spatial messages use a 2D tensor-product B-spline kernel
\[
W_s(\xi,\eta)=\sum_{k=0}^{M}\sum_{\ell=0}^{M}C_{k\ell}B_{k,m}(\xi)B_{\ell,m}(\eta),
\]
whereas temporal messages use motion-vector attention with queries \(q_i^h\), keys \(k_j^h\), headwise coefficients \(\alpha_{ij}^h\), and output
\[
m_i^t=\mathrm{concat}_h(m_i^{t,h})W_o.
\]
Fusion is then
\[
m_i=\mathrm{MLP}_{\mathrm{fuse}}([m_i^s;m_i^t]),\qquad
v_i^{(n)}=v_i^{(n-1)}+\mathrm{MLP}_{\mathrm{update}}(m_i).
\]
DSTGNN alternates temporal graph convolution over \(\mathcal B[:,:,n]\) and spatial graph convolution over \(\mathcal A[:,:,t]\). ST-GraphRL similarly decouples spatial and temporal encoders before a joint space-time GNN re-entangles node features, edge features, and normalized weights \(\hat w_{ij}\) [2507.15150, 2003.08729, 2312.04055].

## 4. Learning objectives, priors, and theoretical properties

A notable property of spatiotemporal multigraph methods is that graph construction and representation learning are frequently optimized together. In the multi-modal forecasting model, the total loss combines an RMSE term with lower-layer grouped penalties and higher-layer tensor-normal penalties:
\[
\mathcal L=
\sum_{(X,Y)\in S}\|\hat Y-Y\|_2^2
+\alpha_{\rm low}\sum_{l=1}^{L_{\rm low}}J_1^l
+\alpha_{\rm high}\sum_{l=L_{\rm low}+1}^{L}J_2^l.
\]
The paper states that temporal shifting is handled both by including period and trend slices in the input and by the MR-GCN prior, which freezes the feature-mode covariances \(\Sigma_I=I_{f_l}\) and \(\Sigma_O=I_{f_{l+1}}\) while updating \(\Sigma_C\) and \(\Sigma_M\) by flip-flop equations [1905.11395].

Unsupervised objectives can use the spatiotemporal graph only as an encoder while placing the learning signal in time. STDGI encodes each snapshot with a two-layer GCN and maximizes mutual information between node embeddings at time \(t\) and raw node features at future times \(t+k\), with \(k\in\{1,3,6\}\). The discriminator loss is
\[
\mathcal L_k=
-\,\mathbb E_{(h,x^+)\sim p_{\rm joint}}[\log \sigma(\mathcal D_k(h,x^+))]
-\,\mathbb E_{(h,x^-)\sim p_{\rm marginals}}[\log(1-\sigma(\mathcal D_k(h,x^-)))],
\]
and the total objective is \(\mathcal L_{\rm STDGI}=\sum_{k\in\{1,3,6\}}\mathcal L_k\). The paper also states explicitly that STDGI, as presented, is a single-relation model; multi-relation or multigraph extensions would require replacing each GCN layer by a relational GCN or using separate adjacency tensors \(W^r\) [1904.06316].

In graph learning, the multigraph itself may be the optimization target. GL-LRSS jointly estimates a latent low-rank signal \(\mathbf X\) and graph Laplacian \(L\) through
\[
\min_{L\in\mathcal L,\ \mathrm{tr}(L)=N,\ \mathbf X}
\|\mathcal D(\mathbf X)-\mathcal D(\mathbf Y)\|_F^2
+\alpha\,\mathrm{tr}\!\bigl[\mathcal D(\mathbf X)^TL\mathcal D(\mathbf X)\bigr]
+\beta\|L\|_F^2
+\gamma\|\mathbf X\|_*.
\]
Here \(\mathcal D(\mathbf X)=\mathbf X-R\mathbf X B\), the nuclear norm enforces low rank, and the trace term is the spatiotemporal smoothness prior [1911.08018].

Connectivity constraints can also be regularized spectrally or sparsely. The STBAM framework trains a classifier with cross-entropy plus a sparsity-promoting norm on the learned adjacency,
\[
\mathcal L=
-\sum_i y_i\log(\hat y_i)
+\lambda\Bigl\|\tfrac12(A_{B,\mathrm{enc}}+A_{B,\mathrm{enc}}^\top)\Bigr\|_p,
\]
and reports that \(p=1\) gave the best trade-off between connectivity and parsimony. In graph-time convolutional theory, stability under support perturbation \(S\to \hat S=S+E\), \(\|E\|\le \epsilon\), is quantified by
\[
\|\Phi(X;S)-\Phi(X;\hat S)\|_2
\le
C\cdot L\cdot F_{\rm out}\cdot(1+8TN)\cdot \epsilon\cdot \|X\|_2,
\]
which the paper interprets as an expressivity–robustness trade-off: larger \(K\), deeper \(L\), or more outputs \(F_{\rm out}\) improve discriminability but degrade robustness [2310.02606, 2206.15174].

## 5. Continuous-time, instantaneous, and sample-wise multigraphs

Continuous-time dynamic graphs replace discrete slices by timestamped events and therefore require positional or memory mechanisms that operate at event time. CorDGT assumes a fixed node set \(V\), raw features \(\mathcal X\in\mathbb R^{N\times d_n}\), and event stream
\[
\mathcal E=\{(u_i,v_i,t_i,e_i)\}_{i=1}^M.
\]
Under a homogeneous Poisson point-process assumption for prior interaction times \(T(w,w_0,t_{\rm pred})=\{t_1,\dots,t_n\}\), the interaction intensity is estimated parameter-free by
\[
\hat\lambda_{\rm MLE}=\frac{n}{t_n}.
\]
This enters the Temporal Distance
\[
\mathrm{TD}(w,w_0,t_{\rm pred})
=
\alpha\,\frac{t_n}{t_{\rm pred}n}
+
\beta\,\frac{t_{\rm pred}-t_n}{t_{\rm pred}},
\qquad \alpha,\beta>0,
\]
which is combined with hop-based Spatial Distance in sinusoidal encodings, unitary encoding \(\mathrm{STPE\_u}\), and correlated encoding
\[
\mathrm{STPE\_c}(w;(u,v),t)=\mathrm{STPE\_u}(w;u,t)+\mathrm{STPE\_u}(w;v,t).
\]
This makes self-attention sensitive to whether a context node is simultaneously close in space and time to both target endpoints [2407.16959].

CTDG-SSM addresses long-range temporal propagation by reformulating continuous-time graph memory as a state-space model. A topology-aware polynomial filter
\[
p(L(t))=\sum_{k=0}^{K-1}\alpha_k (L(t))^k
\]
is used to project HiPPO memory onto graph topology, yielding the continuous-time ODE
\[
\frac{d}{dt}m(t)=A(L(t))\,m(t)+B(L(t))\,h(t),
\]
and readout
\[
y(t)=p(L(t))\,m(t).
\]
Under zero-order hold between irregular event times \(t_k\), this becomes
\[
m[k+1]=A_d(L[k])\,m[k]+B_d(L[k])\,h[k].
\]
The paper states that the \(K\)-th-order filter captures up to \(K\)-hop neighbors, while the underlying HiPPO projection compresses the entire past into a fixed-size state in online \(O(d^2)\) time [2606.04672].

GVNN occupies an intermediate position between discrete and continuous viewpoints. At each time \(t\), it builds an instantaneous graph
\[
J_{ij}(t)=F_V(x_i(t),x_j(t))
\]
and fuses it with a stable support \(S\) by Hadamard product,
\[
\Omega(t)=S\circ J(t).
\]
The layer update then mixes the raw signal and the graph-variate convolution through
\[
Z=X\,\mathrm{diag}(a_{1:T})+(\Omega*X)\,\mathrm{diag}(b_{1:T}),
\qquad
X^{(\ell+1)}=\sigma(Z\Theta).
\]
The paper emphasizes three properties: sample-by-sample interactions via \(J(t)\), regularization by the stable support \(S\), and linear complexity in sequence length,
\[
O(N^2T),
\]
instead of the \(O(N^2T^2)\) cost of a naive Kronecker spatiotemporal convolution [2509.20311].

## 6. Empirical behavior, interpretability, and conceptual boundaries

Across domains, spatiotemporal multigraph representations are evaluated not only by downstream accuracy but also by robustness, convergence, interpretability, and computational profile. In ride-hailing demand forecasting, the combined GGCN+MRGCN model reduces RMSE by \(10\%+\) over the single-graph baselines MGCN and STMGCN, cuts convergence time roughly in half when the MR-prior is included, and remains stable as test-set temporal divergence from training grows. The learned modality covariance \(\Sigma_M\), visualized as a Hinton diagram, shows strong positive coupling between POI-similarity and road-connectivity and between POI-similarity and neighborhood, but weak or no coupling between neighborhood and road, matching the structural overlap of those graphs [1905.11395].

Dynamic graph tensor methods report similar advantages from unified spatial-temporal processing. STGCNDT states that its fused tensor convolution significantly outperforms state-of-the-art models on four communication-network dynamic graphs in link-weight estimation because diversified transformations capture periodic, smooth, and bursty behaviors more effectively than a single temporal basis [2408.02704].

In event-based detection, the efficiency claim is unusually explicit. eGSMV reports over a \(6\%\) improvement in detection accuracy compared to previous graph-based works, with a \(5\times\) speedup, reduced parameter count, and no increase in computational cost. The anisotropic decomposition is quantified by
\[
P_3=C_{\rm out}C_{\rm in}K^3,\qquad
P_2=C_{\rm out}C_{\rm in}K^2,
\]
so that for \(K=8\),
\[
P_2/P_3=1/8,
\]
that is, \(87.5\%\) fewer parameters and FLOPs per message. The reported Gen1 and eTraM scores are \(0.371\) mAP and \(0.431\) mAP@50 with \(5.6\)M parameters, compared with \(0.163\) mAP and \(20.1\)M parameters for AEGNN on Gen1 [2507.15150].

Learned temporal connectivity in block-adjacency models is likewise tied to both accuracy and graph connectivity. STBAM-64 achieves \(80.67\%\) accuracy on C2D2, compared with \(57.72\%\) for 3D-ResNet-34 and \(77.83\%\) for STAG-NN-BA-GSP, using \(3.392\)M parameters. On SurgVisDom it reports weighted \(F1=0.53\), global \(F1=0.40\), and balanced accuracy \(0.447\) [2310.02606].

Trajectory modeling gives a different interpretability criterion: whether representation distance correlates with distributional similarity. ST-GraphRL reports that the correlation \(r_{st}\) between representation-space distance and true joint spatiotemporal distribution distance rises from approximately \(0.26\) for DGI or \(0.03\) for Summary-Trajectory to \(0.77\) on Chengdu, and that it outperforms sequence-only, graph-only, temporal-only, and ablated baselines in Accuracy, Precision, and \(F_1\) by margins up to \(+0.20\) [2312.04055].

A common misconception is that any spatiotemporal graph model is automatically a multigraph. The literature here does not support that equivalence. STDGI is explicitly described as a single-relation spatiotemporal graph model with fixed adjacency \(W\), even though it is compatible with multi-relation extensions. Conversely, product-graph, graph-tensor, and event-stream models may all qualify as spatiotemporal multigraph representations despite using very different mathematical objects. A plausible implication is that the term is best understood as a representational family unified by joint space-time graph structure, rather than as one canonical adjacency format [1904.06316, 2206.15174, 2408.02704, 2407.16959].

Source: https://www.emergentmind.com/topics/spatiotemporal-multigraph-representation