SGNN: Skeleton Graph Neural Network

• SGNN is a graph-based learning system that models skeletal motion by representing joints as nodes and bones as edges, enabling nuanced action recognition.
• SGNN architectures integrate spatial operators, recurrent modules, and hypergraph attention to capture dynamic intra-frame anatomy and inter-frame motion.
• Recent studies focus on efficiency and adaptive topology learning, extending SGNN applications to sign language, gait recognition, and multi-agent behavior analysis.

Searching arXiv for recent and foundational papers on skeleton graph neural networks and skeleton-based action recognition. Search query: "skeleton graph neural network action recognition graph convolution" Search query: "skeleton graph neural network SGNN arXiv action recognition hypergraph spatio-temporal" Skeleton Graph Neural Network (SGNN) denotes a class of graph-based learning systems that represent a skeleton as a graph and perform message passing over its spatial or spatio-temporal structure. In the dominant usage, SGNNs model human or animal pose sequences with joints as nodes and bones as edges, so that intra-frame anatomy and inter-frame motion can be processed jointly for tasks such as action recognition, gait recognition, sign language production, social-behavior analysis, and motion prediction (Chen et al., 2021). Across this literature, the central design question is not whether skeletons should be encoded as graphs, but how graph topology, temporal dynamics, feature order, and computational budget should be coupled in a task-specific architecture (Dong et al., 19 Jan 2025).

1. Formal definition and graph representations

In skeleton-based action recognition, a skeleton graph is commonly defined as $G=(V,E)$ or $G=\{V,E\}$, where $V$ denotes joints and $E$ denotes bones, with connectivity represented by an adjacency matrix $A\in\mathbb{R}^{N\times N}$ or $A\in\{0,1\}^{N\times N}$ (Chen et al., 2021). A sequence is then organized as a tensor such as $X\in\mathbb{R}^{C\times T\times N}$ or $X\in\mathbb{R}^{B\times C\times T\times V}$, depending on whether batching and modality streams are made explicit (Dong et al., 19 Jan 2025). In 3D settings, a joint is typically represented by coordinates $(x,y,z)$; in 2D settings, confidence scores may be appended as an additional channel (Li et al., 2020).

A standard SGNN operator uses normalized adjacency. Representative formulations include

$$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$

followed by

$G=\{V,E\}$0

which appears in HFGCN and is also consistent with ST-GCN-like practice (Dong et al., 19 Jan 2025). Partitioned spatial operators are likewise common. PR-GCN adopts a three-way partition into self, centripetal, and centrifugal subsets, with

$G=\{V,E\}$1

where $G=\{V,E\}$2 and $G=\{V,E\}$3 is a learnable attention map (Li et al., 2020). AGC-LSTM uses an analogous partitioned GCN inside recurrent gates, with

$G=\{V,E\}$4

at each time step (Si et al., 2019).

Although most SGNNs are node-centric, edge-centric alternatives also exist. Graph Edge Convolutional Neural Networks treat bones as graph edges and define neighborhoods by shortest-path distance between edges, using a normalized edge convolution

$G=\{V,E\}$5

thereby shifting the primitive from joint relations to bone relations (Zhang et al., 2018). More recently, Graph-in-Graph Neural Network generalizes the representational unit further by allowing each vertex of a global graph to contain an inner graph; in skeleton video analysis, an inner graph can represent the joints of a single frame, while proxy vertices mediate inter-frame communication (Wang et al., 2024).

2. Spatial operators, temporal dynamics, and recurrent formulations

The canonical SGNN decomposes learning into spatial aggregation over skeleton topology and temporal modeling over pose sequences. DualHead-Net makes this separation explicit through a backbone that combines a factorized pathway, MS-GCN plus MS-TCN, with a full MS-G3D pathway, while maintaining a dual-head design for coarse- and fine-grained motion patterns (Chen et al., 2021). Its spatial update is written as

$G=\{V,E\}$6

and temporal modeling is implemented by multi-scale temporal convolutions with different dilation rates (Chen et al., 2021).

Temporal modeling is not restricted to convolutions. AGC-LSTM replaces the affine transforms of a conventional LSTM by graph convolutions on both the current input and the previous hidden state:

$G=\{V,E\}$7

with analogous equations for $G=\{V,E\}$8, $G=\{V,E\}$9, and the candidate cell state (Si et al., 2019). This design lets spatial neighborhood aggregation participate directly in temporal recurrence, rather than treating spatial and temporal processing as separate stages.

Several architectures pursue explicit multi-granularity. DualHead-Net uses a coarse head with lower temporal resolution and a fine head at full temporal resolution, with cross-head communication through residual re-weighting:

$V$0

so that fine motion context informs coarse reasoning, while coarse context highlights pivotal joints for fine reasoning (Chen et al., 2021). FGCN addresses temporal progression differently: it splits a sequence into multiple temporal stages, samples one clip from each stage, and feeds stage-wise features through a Feedback Graph Convolutional Block,

$V$1

thereby re-injecting higher-level semantics into later-stage processing and enabling early predictions (Yang et al., 2020).

A recurrent misconception is that SGNN temporal modeling is synonymous with temporal convolution. The literature instead contains convolutional, recurrent, hierarchical, and feedback-based temporal schemes, each imposing different assumptions about receptive field growth, long-range dependency capture, and computational cost (Si et al., 2019).

3. Topology learning, higher-order relations, and feature order

A major axis of SGNN research concerns the inadequacy of fixed pairwise bone topology for modeling coordinated body motion. HFGCN addresses this by introducing kinematics-informed hypergraphs. It fuses three skeleton classifications: a functional topology $V$2, a biomechanics-informed body-part grouping $V$3, and a distance-from-core grouping $V$4, then represents these categories with incidence matrices $V$5 (Dong et al., 19 Jan 2025). Hypergraph propagation is defined as

$V$6

with per-strategy convolution

$V$7

This extends ordinary GCN message passing from pairwise edges to higher-order joint-group relations (Dong et al., 19 Jan 2025).

HFGCN further introduces a Hypergraph Attention Module that produces dynamic per-frame topologies

$V$8

with $V$9, and fuses three branches in the Hypergraph Convolution Module:

$E$0

The model reports that, on NTU RGB+D 120 X-Sub with four-modality fusion, $E$1 alone gives $E$2, $E$3 gives $E$4, and $E$5 gives $E$6, indicating gains from combining kinematics-informed topologies (Dong et al., 19 Jan 2025).

Another route to higher-order structure is feature order rather than topology order. Qin et al. fuse first-order joint features, second-order bone features, and higher-order angular encodings, using

$E$7

with angular velocity

$E$8

to encode third- and fourth-order geometry (Qin et al., 2021). Their results show that angular encoding improves both static and velocity streams on NTU60 and NTU120, and that center-oriented angles are the most effective single category (Qin et al., 2021).

Topology can also be learned by search. The NAS-designed GCN of Zhao et al. includes dynamic spatial, temporal, and spatio-temporal graph modules, alongside higher-order Chebyshev components $E$9, $A\in\mathbb{R}^{N\times N}$0, $A\in\mathbb{R}^{N\times N}$1, and $A\in\mathbb{R}^{N\times N}$2 (Peng et al., 2019). The searched architecture retained $A\in\mathbb{R}^{N\times N}$3 across almost all layers and selected $A\in\mathbb{R}^{N\times N}$4 consistently, while the pure physical $A\in\mathbb{R}^{N\times N}$5 operator was not selected at any layer (Peng et al., 2019). This suggests that layer-specific dynamic adjacency and higher-order propagation are not incidental refinements, but stable solutions under a large search space.

4. Architectural diversification beyond conventional action recognition

Although skeleton-based action recognition remains the dominant application, SGNNs have diversified into several neighboring domains. GaitGraph uses a ResGCN-style skeleton model on 17 COCO keypoints for gait recognition, with input $A\in\mathbb{R}^{N\times N}$6 and $A\in\mathbb{R}^{N\times N}$7 corresponding to $A\in\mathbb{R}^{N\times N}$8 (Teepe et al., 2021). On CASIA-B, it reports rank-1 means of $A\in\mathbb{R}^{N\times N}$9 for NM, $A\in\{0,1\}^{N\times N}$0 for BG, and $A\in\{0,1\}^{N\times N}$1 for CL, surpassing the model-based baseline PoseGait on all three conditions (Teepe et al., 2021).

In sign language production, SGSA preserves skeletal structure throughout an encoder-decoder Transformer by applying a degree-normalized spatio-temporal adjacency directly to the value pathway:

$A\in\{0,1\}^{N\times N}$2

with

$A\in\{0,1\}^{N\times N}$3

This design retains global temporal attention while enforcing anatomical locality in message passing (Saunders et al., 2021). On PHOENIX14T, SGSA with $A\in\{0,1\}^{N\times N}$4 local temporal adjacency achieves BLEU-4 of $A\in\{0,1\}^{N\times N}$5 on dev and $A\in\{0,1\}^{N\times N}$6 on test, described as an $A\in\{0,1\}^{N\times N}$7 and $A\in\{0,1\}^{N\times N}$8 improvement over competing methods for the dev and test sets (Saunders et al., 2021).

For mouse social behaviour, CS-IGANet extends SGNN reasoning to multi-agent interaction graphs. Its Cross-Skeleton Node-level Interaction module combines intra-skeleton, inter-skeleton, and cross-skeleton interactions, while dense geometric distances and velocities drive adaptive inter-skeleton adjacency estimation (Zhou et al., 2022). This line of work suggests that SGNNs need not be confined to a single-body kinematic tree; they can instead mediate structured interaction between multiple deformable skeletons.

Motion prediction is a further extension, though the supplied DD-GCN details are explicitly marked as not extracted from the original paper content and therefore should not be treated as paper-grounded empirical evidence (Wang et al., 2023). By contrast, the generic Graph-in-Graph framework is paper-grounded: it treats each frame-level skeleton graph as an inner graph and uses GVU and GGU modules to propagate information within and across inner graphs (Wang et al., 2024). This representation generalizes SGNN beyond flat spatio-temporal graphs toward nested graph structures.

5. Efficiency, compactness, and spiking SGNNs

Efficiency has become a first-class objective in SGNN design, especially for robotics and edge deployment. PR-GCN exemplifies a compact architecture that combines pose refinement, parallel position and motion flows, gradual fusion, and early temporal downsampling (Li et al., 2020). Relative to ST-GCN on Kinetics, PR-GCN with PRM+TAM uses $A\in\{0,1\}^{N\times N}$9M parameters and $X\in\mathbb{R}^{C\times T\times N}$0 GFLOPs versus $X\in\mathbb{R}^{C\times T\times N}$1M and $X\in\mathbb{R}^{C\times T\times N}$2 GFLOPs, corresponding to approximately $X\in\mathbb{R}^{C\times T\times N}$3 fewer parameters and $X\in\mathbb{R}^{C\times T\times N}$4 fewer FLOPs, while reaching $X\in\mathbb{R}^{C\times T\times N}$5 top-1/top-5 on Kinetics and $X\in\mathbb{R}^{C\times T\times N}$6 on NTU RGB+D X-Sub/X-View (Li et al., 2020). The paper further reports CPU throughput up to $X\in\mathbb{R}^{C\times T\times N}$7 FPS with PRM+TAM, making it suitable for robot platforms without GPUs (Li et al., 2020).

STIGCN pursues a different efficiency regime through an inception-style split-transform-merge block. Spatial branches use Chebyshev polynomials up to order $X\in\mathbb{R}^{C\times T\times N}$8,

$X\in\mathbb{R}^{C\times T\times N}$9

while temporal branches process both positions and motion signals (Huang et al., 2020). The model reports $X\in\mathbb{R}^{B\times C\times T\times V}$0M parameters and $X\in\mathbb{R}^{B\times C\times T\times V}$1 GFLOPs, compared with $X\in\mathbb{R}^{B\times C\times T\times V}$2M and $X\in\mathbb{R}^{B\times C\times T\times V}$3 GFLOPs for 2S-AGCN and $X\in\mathbb{R}^{B\times C\times T\times V}$4M and $X\in\mathbb{R}^{B\times C\times T\times V}$5 GFLOPs for NAS-GCN, while achieving $X\in\mathbb{R}^{B\times C\times T\times V}$6 on NTU RGB+D joint-only X-Sub/X-View and $X\in\mathbb{R}^{B\times C\times T\times V}$7 top-1/top-5 on Kinetics-Skeleton (Huang et al., 2020).

Spiking SGNNs push efficiency further by replacing dense floating-point operations with spike-driven computation. MK-SGN proposes spiking multimodal fusion, self-attention spiking graph convolution, spiking temporal convolution, and integrated knowledge distillation, and states that it achieves a remarkable reduction in energy consumption, exceeding $X\in\mathbb{R}^{B\times C\times T\times V}$8 compared to conventional GCN-based approaches (Zheng et al., 2024). Signal-SGN and Signal-SGN++ make the temporal dimension of the skeleton sequence itself serve as the spiking timestep, then add frequency-domain modeling and wavelet-based multi-scale fusion (Zheng et al., 2024). Signal-SGN reports, for the four-stream ensemble, $X\in\mathbb{R}^{B\times C\times T\times V}$9 on NTU60 Xs/Xv, $(x,y,z)$0 on NTU120 Xs/Xv, and $(x,y,z)$1 on NW-UCLA, with theoretical energy of $(x,y,z)$2 mJ (Zheng et al., 2024). Signal-SGN++ extends this with Topology-Shift Self-Attention and topology-aware time-frequency fusion, reaching $(x,y,z)$3 on NTU60 Xs/Xv, $(x,y,z)$4 on NTU120 Xs/Xt, and $(x,y,z)$5 on NW-UCLA for the four-stream ensemble, at $(x,y,z)$6 mJ (Zheng et al., 22 Dec 2025).

These results support a practical distinction within SGNN research. One branch prioritizes maximal accuracy through richer topology and multi-stream fusion; another prioritizes parameter, FLOP, latency, or energy budgets. The trade-off is architectural rather than purely implementation-level.

6. Benchmarks, misconceptions, and open issues

Across the supplied literature, NTU RGB+D 60, NTU RGB+D 120, Kinetics-Skeleton, and NW-UCLA function as the dominant evaluation beds for human action recognition (Chen et al., 2021). Representative results span very different design points. DualHead-Net reports $(x,y,z)$7 on NTU RGB+D 60 with four streams, $(x,y,z)$8 on NTU RGB+D 120, and $(x,y,z)$9 top-1/top-5 on Kinetics-Skeleton (Chen et al., 2021). HFGCN reports four-modality fusion accuracies of $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$0 on NTU60, $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$1 on NTU120, and $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$2 on NW-UCLA, while also exceeding InfoGCN on the four NTU60/120 benchmarks in the multi-modal setting by $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$3, $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$4, $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$5, and $$\tilde{A}=A+I,\qquad D=\operatorname{diag}\!\left(\sum_j \tilde{A}_{ij}\right),\qquad \hat{A}=D^{-1/2}\tilde{A}D^{-1/2},$$6 respectively (Dong et al., 19 Jan 2025).

Several misconceptions recur. The first is that SGNN denotes a single canonical architecture. The literature instead includes feedforward spatial-temporal GCNs, graph-convolutional recurrent networks, feedback networks, hypergraph models, inception-style models, edge-centric models, graph-in-graph models, and spiking variants (Yang et al., 2020). The second is that skeleton topology is fixed and obvious. In practice, the field remains divided between physical adjacency, partitioned adjacency, learnable masks, dynamic attention, higher-order Chebyshev propagation, and hypergraph incidence structures (Peng et al., 2019). The third is that better topology alone is sufficient. Multiple papers argue that feature order, temporal scale, modality composition, and robustness to pose noise are equally consequential (Qin et al., 2021).

Open issues are also explicit in the sources. HFGCN notes that its hypergraph categories are manually designed based on biomechanics and suggests that future work can learn categories end-to-end or adapt them per dataset or task automatically (Dong et al., 19 Jan 2025). PR-GCN notes that static adjacency limits expressivity and that adaptive adjacency or lightweight attention could improve performance (Li et al., 2020). DualHead-Net observes sensitivity to aggressive temporal subsampling and to pose noise in fine-grained recognition (Chen et al., 2021). Signal-SGN++ identifies wavelet filter design and spiking-hyperparameter sensitivity as continuing limitations (Zheng et al., 22 Dec 2025).

A final terminological ambiguity should also be noted. In one paper on generic weighted graphs, SGNN denotes a “Skeleton Graph Neural Network” that operates on a multi-resolution skeleton graph for shortest-path search rather than on human or animal pose sequences (Liu et al., 4 Aug 2025). This suggests that the acronym SGNN is no longer semantically unique. In contemporary computer vision and action-recognition usage, however, it most often denotes graph neural architectures for skeletal motion data.