Pose-TGCN: Spatio-Temporal Graph Networks

• The paper introduces Pose-TGCN, a unified spatio-temporal graph convolutional model that improves long-term pose forecasting by over 32% with just 1.7% of competitor parameters.
• It factors the global adjacency into independently learnable spatial and temporal matrices, enabling efficient and compact modeling of joint dynamics.
• Empirical evaluations on datasets like Human3.6M, AMASS, and 3DPW confirm its superior predictive accuracy and enhanced interpretability compared to traditional methods.

Pose-Temporal Graph Convolutional Networks (Pose-TGCN), also referred to as Space-Time-Separable Graph Convolutional Networks (STS-GCN), constitute a class of neural architectures for human pose forecasting that model spatio-temporal joint dynamics using a fully graph-based approach. Unlike traditional methods that treat spatial and temporal dependencies separately (e.g., via kinematic trees and sequence models), Pose-TGCN encodes the evolution of all body joints across time within a single spatio-temporal graph, leveraging learnable factorizations to capture structural dynamics with significant parameter efficiency and predictive performance (Sofianos et al., 2021).

1. Spatio-Temporal Graph Formulation

Pose-TGCN takes as input an observed sequence of $T$ 3D human poses, each consisting of $V$ joints. This input is represented as a tensor $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$, where each slice encodes the coordinates of all joints at a particular time. The spatio-temporal graph constructed for Pose-TGCN comprises $VT$ nodes, each node corresponding to a specific joint at a specific frame.

Rather than forming a full $(VT)\times(VT)$ adjacency matrix, which would be computationally prohibitive and prone to overfitting, the approach introduces a parameter bottleneck via space-time separability. The global adjacency is factored into two independently learnable matrices: a spatial affinity $A^s\in\mathbb R^{V\times V}$ encoding joint-joint relationships, and a temporal affinity $A^t\in\mathbb R^{T\times T}$ encoding frame-frame relationships per joint. This factorization recovers a full $(VT)\times(VT)$ structural representation through the product $A^sA^t$, with only $V^2 + T^2$ trainable parameters per layer, as opposed to $V$0 in non-separable designs.

2. Network Architecture

2.1. Pose-TGCN Encoding

The Pose-TGCN core consists of four stacked GCN layers, each propagating information jointly in space and time. For a given layer $V$1 with input feature tensor $V$2 and learnable weight matrix $V$3, the transformation is:

$V$4

where $V$5 denotes PReLU activation. Each $V$6 and $V$7 are fully learned, signed, directed, and unconstrained, enabling the model to discover data-driven joint coordination and temporal dependencies.

2.2. Temporal Decoding

The latent features from Pose-TGCN are mapped to $V$8 future frames using four 1D temporal convolutional layers. This decoding step processes the spatio-temporal representations into future joint coordinates, yielding an output tensor of predicted poses for all forecasted frames.

3. Mathematical Formalism

A conventional (non-separable) GCN layer updates features as:

$V$9

where $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$0 is the full adjacency.

Space–time separability in Pose-TGCN imposes $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$1 as the only structural prior, with $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$2 and $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$3 both unconstrained and trained end-to-end. No assumptions are made about symmetry, normalization, or connectivity beyond learnability. This enables the space-time graph to capture highly nontrivial spatial and temporal correlations that may substantially deviate from anatomical (kinematic tree) or sequential (linear-time) structures.

4. Training Methodology and Loss

Supervision is performed over all $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$4 frames. During training, teacher forcing is applied to the first $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$5 frames, allowing the network to predict all future $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$6 frames given ground-truth past sequences. Loss minimization is conducted end-to-end, directly optimizing the full set of learnable affinities, GCN, and temporal decoding weights.

5. Empirical Evaluation and Benchmarking

Pose-TGCN was benchmarked on three recent large-scale datasets: Human3.6M [Ionescu et al. TPAMI'14], AMASS [Mahmood et al. ICCV'19], and 3DPW [Von Marcard et al. ECCV'18]. In direct comparison, Pose-TGCN outperformed previous state-of-the-art methods, including [Mao et al. ECCV'20], with over 32% improvement at the long-term prediction horizon, while utilizing only 1.7% of the competitor's model parameters. This indicates both strong generalization capacity and parameter efficiency in modeling complex human pose dynamics within a fully graph-based, learnably separable framework (Sofianos et al., 2021).

6. Interpretability and Structural Insights

Analysis of the learned affinity matrices reveals that spatial and temporal connections identified by the model diverge from standard kinematic and time-series assumptions, highlighting the utility of unconstrained joint-joint and time-time correlation learning. Factored adjacency enables illustration and investigation of both spatial and temporal dependencies, facilitating interpretability which may be obscured in monolithic, unconstrained $$\mathcal X_{\rm in}\in\mathbb R^{3\times V\times T}$$7 graphs. This suggests that Pose-TGCN captures novel structural patterns in human motion not accessible to approaches that pre-impose anatomical or sequential priors.

7. Significance and Context

Pose-TGCN represents a departure from hybrid approaches that treat space and time with separate model components. By unifying spatio-temporal reasoning in a compact, fully graph-based and end-to-end learnable architecture, Pose-TGCN sets a new standard for pose forecasting tasks. The minimal structural prior—space-time separability—coupled with full learnability distinguishes it from methods constrained to human-interpretable or pre-engineered graphs, with implications for future work in structured sequence modeling beyond human pose (Sofianos et al., 2021).