TSkel-Mamba: Hybrid Skeleton Action Recognition

• TSkel-Mamba is a skeleton-based action recognition framework that combines Spatial Transformers with a Mamba-based temporal module to model both joint dependencies and long-range motion patterns.
• It employs Multi-scale Temporal Interaction and bidirectional selective state-space scanning to enable efficient, linear-time temporal modeling with explicit cross-channel interactions.
• Empirical evaluations on NTU, NW-UCLA, and UAV-Human datasets demonstrate notable accuracy gains and computational efficiency compared to state-of-the-art graph-based and temporal methods.

Searching arXiv for TSkel-Mamba and closely related skeleton/Mamba papers. TSkel-Mamba is a skeleton-based action recognition framework that combines a Spatial Transformer for per-frame joint-relation modeling with a Mamba-based temporal module for long-range sequence dynamics. In the form introduced in “TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition” (Liu et al., 12 Dec 2025), it is a hybrid Transformer–Mamba architecture designed for sequences of human skeletons represented as 3D joint coordinates, with the central claim that temporal selective state-space modeling should be augmented with explicit cross-channel temporal interaction. In a broader usage, “TSkel-Mamba” can also denote the more general design pattern of applying Mamba or selective state-space models to temporal skeleton modeling while retaining graph- or topology-aware spatial priors, a pattern also visible in related systems such as Simba (Chaudhuri et al., 2024) and STG-Mamba (Tang et al., 9 Jul 2025).

1. Definition and problem setting

TSkel-Mamba addresses human skeleton-based action recognition, where the input is a sequence of skeletons and the output is an action label (Liu et al., 12 Dec 2025). A skeleton at frame $t$ with $N$ joints is represented as $X_t \in \mathbb{R}^{3 \times N}$, and a sequence is $\{X_t\}_{t=1..T}$ (Liu et al., 12 Dec 2025). The underlying problem requires simultaneous modeling of spatial dependencies among joints within a frame and temporal dependencies across frames, including long-term and multi-scale motion patterns (Liu et al., 12 Dec 2025).

The method is motivated by a perceived division of labor between architectural families. Transformers are effective at long-range dependency modeling but incur quadratic complexity in sequence length; CNN and TCN temporal modules are efficient but predominantly local; Mamba provides linear-time temporal modeling through selective state-space scanning, but in its standard form processes channels independently and therefore under-exploits cross-channel temporal interactions (Liu et al., 12 Dec 2025). TSkel-Mamba is formulated as a direct response to this limitation by pairing Spatial Transformer blocks with a Temporal Dynamic Modeling (TDM) block that uses Mamba while explicitly injecting multi-scale temporal-channel interactions (Liu et al., 12 Dec 2025).

The task formulation in (Liu et al., 12 Dec 2025) uses multiple standard skeleton modalities: raw joint coordinates, bone vectors, joint motion, and bone motion, fused by summing softmax scores in two-stream or four-stream configurations. This positions TSkel-Mamba within the mainstream multi-stream action-recognition literature rather than as a single-modality departure (Liu et al., 12 Dec 2025).

2. Architectural composition

TSkel-Mamba is organized as a stack of hybrid layers in which Spatial Transformer (ST) modules perform spatial feature learning and TDM modules perform temporal modeling (Liu et al., 12 Dec 2025). The high-level processing sequence is: input skeleton sequence, tokenization per frame, Spatial Transformer with relative positional encoding, features $H \in \mathbb{R}^{B \times C \times T \times N}$, TDM, repeated over $L$ layers, and finally a recognition head with pooling and fully connected classification; optionally, covariance pooling with knowledge distillation is used during training (Liu et al., 12 Dec 2025).

The Spatial Transformer treats joints as tokens and uses multi-head self-attention with relative positional encoding constructed from shortest-path distances on the human-body topology (Liu et al., 12 Dec 2025). This preserves a topological prior while allowing long-range spatial interactions beyond direct anatomical adjacency. The backbone is described as a 7-layer hybrid backbone with 216 channels and 9 heads (Liu et al., 12 Dec 2025).

The TDM block is the defining temporal component. Its internal sequence is: normalization and $1 \times 1$ channel projection to $C/2$, Multi-scale Temporal Interaction (MTI), pure temporal scanning per joint in forward and reversed order, bidirectional selective SSM with token-dependent gating, concatenation of the two streams, layer normalization, temporal pooling, and residual fusion (Liu et al., 12 Dec 2025). The paper explicitly characterizes TDM as a plug-and-play module and reports that replacing early temporal layers in existing backbones yields gains in CTR-GCN, Block-GCN, and Hyperformer (Liu et al., 12 Dec 2025).

A useful comparative context is provided by related Mamba-based skeleton systems. Simba places a Mamba block at the bottleneck of a U-shaped ShiftGCN encoder–decoder, flattening all joints in a frame to a single per-frame embedding before temporal scanning (Chaudhuri et al., 2024). STG-Mamba, although aimed at music-guided dance video synthesis rather than action recognition, also implements Mamba-like temporal skeleton modeling using graph-structured spatial operations plus forward and backward temporal SSM scans (Tang et al., 9 Jul 2025). This suggests that TSkel-Mamba belongs to a broader family of “temporal skeleton Mamba” designs, but it is distinguished by its explicit Spatial Transformer front-end and its MTI-enhanced temporal block (Liu et al., 12 Dec 2025).

3. State-space modeling and temporal dynamics

The temporal core of TSkel-Mamba is grounded in continuous-time and discrete-time state-space modeling. The paper gives the continuous-time equations

$$\dot{x}(t) = A(t)x(t) + B(t)u(t), \qquad y(t) = C(t)x(t) + D(t)u(t),$$

and the discrete-time form

$$x_t = \bar{A}_t x_{t-1} + \bar{B}_t u_t, \qquad y_t = \bar{C}_t x_t + \bar{D}_t u_t,$$

with time-varying parameters induced by discretization (Liu et al., 12 Dec 2025). The Mamba interpretation is that token-dependent parameterization and gating turn this into a selective, context-adaptive linear-time scan over sequence length (Liu et al., 12 Dec 2025).

The paper presents a schematic selective update with gate

$N$0

thereby emphasizing that relevant information is selected dynamically during temporal propagation (Liu et al., 12 Dec 2025). The important design claim is not merely that linear-time recurrence is used, but that temporal modeling remains streaming-friendly and scales as $N$1 per sequence (Liu et al., 12 Dec 2025).

The key criticism of standard Mamba within this domain is that it commonly assigns separate SSM dynamics to individual channels, which limits explicit inter-channel temporal coupling (Liu et al., 12 Dec 2025). In skeleton understanding, cross-channel coordination is treated as indispensable because subtle actions often rely on coherent evolution across coordinate axes, joints, or learned feature subspaces (Liu et al., 12 Dec 2025). TSkel-Mamba therefore inserts a dedicated interaction mechanism before the SSM scan rather than relying on the SSM alone to discover such dependencies.

This design differs from Simba’s intermediate Mamba block, which also uses selective SSM scanning over temporal embeddings but is embedded between spatial encoder and decoder components of a U-ShiftGCN (Chaudhuri et al., 2024). It also differs from STG-Mamba, whose SG-SSM, TGF-SSM, and TGB-SSM divide the problem into per-frame spatial modeling and bidirectional temporal scans over skeleton trajectories for a generative task (Tang et al., 9 Jul 2025). TSkel-Mamba’s particular contribution is the assertion that temporal scanning should remain per joint and be explicitly preceded by temporal-channel interaction (Liu et al., 12 Dec 2025).

4. Temporal Dynamic Modeling and Multi-scale Temporal Interaction

The Temporal Dynamic Modeling (TDM) block is introduced to strengthen Mamba’s temporal capability for skeleton data while compensating for its per-channel limitation through the Multi-scale Temporal Interaction (MTI) module (Liu et al., 12 Dec 2025). After pre-normalization and $N$2 projection to $N$3, MTI applies Cycle operators that inject cross-channel temporal interactions by assigning channel-index-dependent temporal offsets (Liu et al., 12 Dec 2025).

For a projected feature tensor $N$4, the Cycle operator with kernel size $N$5 uses

$N$6

for example producing offsets $N$7 when $N$8 (Liu et al., 12 Dec 2025). The operator is defined as

$N$9

and the multi-scale aggregation is

$X_t \in \mathbb{R}^{3 \times N}$0

with scales such as $X_t \in \mathbb{R}^{3 \times N}$1 (Liu et al., 12 Dec 2025).

The stated purpose of MTI is to create multi-scale temporal-channel mixing before selective scanning, thereby enriching motion representation with a small computational overhead (Liu et al., 12 Dec 2025). After MTI, TDM performs pure temporal scanning per joint, constructing forward and backward sequences, applying bidirectional selective SSM with gating, flipping the backward stream back to chronological order, concatenating the two streams, and then applying normalization, temporal pooling, and residual fusion (Liu et al., 12 Dec 2025).

Ablation studies in (Liu et al., 12 Dec 2025) isolate the effect of these components on NTU120 X-Sub with joint input. Starting from a baseline of 84.2%, T-Scan Mamba reaches 85.8, adding MTI single-scale ($X_t \in \mathbb{R}^{3 \times N}$2) reaches 86.4, MTI multi-scale with $X_t \in \mathbb{R}^{3 \times N}$3 reaches 86.7, and the full TDM with bidirectionality and pooling reaches 87.4 (Liu et al., 12 Dec 2025). Additional ablations report that T-Scan outperforms TS-Scan, ST-Scan, and S-Scan, and that multi-scale $X_t \in \mathbb{R}^{3 \times N}$4 performs better than $X_t \in \mathbb{R}^{3 \times N}$5, $X_t \in \mathbb{R}^{3 \times N}$6, and $X_t \in \mathbb{R}^{3 \times N}$7 (Liu et al., 12 Dec 2025). These results are used in the paper to argue that temporal-only per-joint scanning is more suitable than flattened spatiotemporal scanning and that overly coarse temporal receptive fields may be detrimental (Liu et al., 12 Dec 2025).

A plausible implication is that TSkel-Mamba’s temporal modeling strategy is less an unmodified adoption of Mamba than a domain-specific reformulation in which selective SSM is embedded within a structured preconditioning step tailored to skeleton sequences.

5. Spatial modeling, topology, and relation to graph-based methods

The spatial side of TSkel-Mamba is handled by a Spatial Transformer rather than a graph convolutional operator (Liu et al., 12 Dec 2025). Its relative positional encoding is derived from shortest-path distances on the physical skeleton graph, so the attention mechanism remains topology-aware despite its global receptive field (Liu et al., 12 Dec 2025). In single-head form, the paper writes

$X_t \in \mathbb{R}^{3 \times N}$8

where $X_t \in \mathbb{R}^{3 \times N}$9 is a learnable tensor parameterized by graph distances (Liu et al., 12 Dec 2025).

This choice places TSkel-Mamba in a distinct position relative to earlier skeleton Mamba systems. Simba argues that plain Transformers underperform graph-convolutional methods for skeleton action recognition because of the lack of strong structural priors, and therefore keeps Shift S-GCN as the dominant spatial mechanism while inserting Mamba only at the temporal bottleneck (Chaudhuri et al., 2024). STG-Mamba likewise uses GraphConv1d to facilitate spatial message passing among joints before applying SG-SSM and temporal SSMs (Tang et al., 9 Jul 2025).

TSkel-Mamba instead uses Transformer attention for spatial modeling but constrains it through topology-aware relative positional encoding (Liu et al., 12 Dec 2025). This suggests an intermediate position between pure graph-prior architectures and unconstrained attention. The paper’s reported improvements over strong spatial-dominant baselines such as Block-GCN and Hyperformer are presented as evidence that this combination of topology-aware attention and TDM is effective (Liu et al., 12 Dec 2025).

The treatment of spatial and temporal modeling is therefore deliberately decoupled. Spatial Transformer handles long-range joint relations within each frame; TDM handles temporally extended motion patterns with near-linear temporal cost (Liu et al., 12 Dec 2025). This decomposition is central to the method’s identity and to the explanation given for its empirical performance.

6. Datasets, training procedure, and empirical results

TSkel-Mamba is evaluated on NTU-RGB+D 60, NTU-RGB+D 120, NW-UCLA, and UAV-Human (Liu et al., 12 Dec 2025). The paper states that NTU-RGB+D 60 contains 56,880 samples with 25 joints and uses Cross-Subject (X-Sub) and Cross-View (X-View) splits; NTU-RGB+D 120 contains 114,480 samples with 25 joints and uses Cross-Subject (X-Sub) and Cross-Setup (X-Set); NW-UCLA uses 20 joints and a cross-view split; UAV-Human contains 22,476 clips and 155 classes with the CSv1 split (Liu et al., 12 Dec 2025).

Input sequences are resized to 64 frames using random cropping and bilinear interpolation following prior practice, with batch sizes 64 for NTU60/120 and 16 for NW-UCLA (Liu et al., 12 Dec 2025). Training uses Python/PyTorch on a single RTX 4090 GPU, SGD, weight decay $\{X_t\}_{t=1..T}$0, initial learning rate 0.025, decay by $\{X_t\}_{t=1..T}$1 during epochs 110–120, and a total of 120 epochs (Liu et al., 12 Dec 2025). The loss is cross-entropy for classification, with an optional knowledge-distillation term when covariance pooling is used (Liu et al., 12 Dec 2025).

The principal reported top-1 accuracies are as follows.

Dataset / split	Configuration	Accuracy
NTU60 X-Sub	joint	91.4
NTU60 X-Sub	two-stream	92.9
NTU60 X-Sub	four-stream	93.1
NTU60 X-View	joint	95.8
NTU60 X-View	two-stream	96.8
NTU60 X-View	four-stream	97.2
NTU120 X-Set	joint	88.7
NTU120 X-Set	two-stream	90.9
NTU120 X-Set	four-stream	91.6
NTU120 X-Sub	joint	87.4
NTU120 X-Sub	two-stream	89.7
NTU120 X-Sub	four-stream	90.4
NW-UCLA	TSkel-Mamba	97.0
UAV-Human CSv1	TSkel-Mamba	47.2

With the optional CPKD training strategy, the paper reports improved numbers such as 91.6/93.0/93.2 on NTU60 X-Sub, 96.2/97.1/97.4 on NTU60 X-View, 88.9/91.0/91.7 on NTU120 X-Set, 87.9/90.0/90.6 on NTU120 X-Sub, and 97.2 on NW-UCLA (Liu et al., 12 Dec 2025).

The comparative claims are concrete. Against Hyperformer, TSkel-Mamba is reported as +0.9% on NTU120 X-Set and +1.3% on NTU120 X-Sub with fewer FLOPs; against Koopman it reports +1.4% and +1.0% on NTU60 X-Sub/X-View joint and +2.2% and +1.4% on NTU120 X-Sub/X-Set joint with approximately half the parameters; on UAV-Human it reports +3.0% over Koopman in CSv1 (Liu et al., 12 Dec 2025). The paper also states that TDM improves the ten hardest NTU120 X-Sub classes, with a maximum gain of +16.18% and average gain +9.38%, while gains on 105 classes average +3.76% and negatives on 10 classes average −0.94% (Liu et al., 12 Dec 2025).

For broader context, Simba reports 96.34 top-1 on NW-UCLA in four-stream form and strong results on NTU datasets using a U-ShiftGCN plus Mamba bottleneck (Chaudhuri et al., 2024). This indicates that Mamba-based temporal modeling had already become competitive in skeleton action recognition before TSkel-Mamba, though under a different architectural philosophy.

7. Efficiency, optional CPKD, limitations, and broader significance

Efficiency is a central part of the TSkel-Mamba argument. The paper reports 2.4M parameters, 8.2G FLOPs, and 87.9% on NTU120 X-Sub for the joint configuration (Liu et al., 12 Dec 2025). In the same comparison, ST-TR is reported at 12.1M, 259.4G, 82.7%; DSTA-Net at 3.4M, 16.2G, 84.0%; Hyperformer at 2.7M, 9.6G, 86.6%; and Koopman pooling at 5.3M, 8.8G, 85.7% (Liu et al., 12 Dec 2025). The paper further states that linear-time temporal scan enables high throughput and that, under the same hardware, the method achieves the best accuracy with competitive inference time (Liu et al., 12 Dec 2025).

An additional training mechanism is Covariance Pooling with Knowledge Distillation (CPKD) (Liu et al., 12 Dec 2025). Covariance pooling replaces global average pooling with a second-order representation built from covariance $\{X_t\}_{t=1..T}$2, applies a matrix square root through Newton–Schulz iteration, vectorizes the upper triangular part, and feeds it to a classifier (Liu et al., 12 Dec 2025). To avoid inference overhead, the covariance-pooling teacher is distilled into a GAP student using

$\{X_t\}_{t=1..T}$3

with $\{X_t\}_{t=1..T}$4 and $\{X_t\}_{t=1..T}$5 (Liu et al., 12 Dec 2025). The paper states that this improves accuracy without changing student parameters or FLOPs and notes that compact students such as 0.9M params, 3.5G FLOPs can reach 87.7% (Liu et al., 12 Dec 2025).

The limitations stated in (Liu et al., 12 Dec 2025) are restrained rather than expansive. TDM is described as somewhat heavier than pure TCNs; gains are less pronounced on very short sequences; future work includes adaptive scale selection, learnable data-dependent Cycle permutations, jointly learned channel grouping and offsets, multimodal fusion with RGB or IMU and language supervision, online forward-only variants, and tighter integration of second-order temporal statistics (Liu et al., 12 Dec 2025). These are best read as targeted development directions rather than as evidence of instability or failure.

Within the broader Mamba-for-skeleton literature, TSkel-Mamba can be understood as a specific maturation of the “temporal skeleton Mamba” concept. Simba demonstrates that inserting Mamba into a graph-convolutional U-shaped backbone can materially improve skeleton action recognition (Chaudhuri et al., 2024). STG-Mamba shows that graph-aware, bidirectional Mamba-style scanning over skeleton sequences is useful even in a different task domain, namely music-guided dance video synthesis (Tang et al., 9 Jul 2025). TSkel-Mamba extends this trajectory by arguing that temporal selective state-space modeling for skeletons should be per-joint, bidirectional, topology-aware on the spatial side, and explicitly cross-channel on the temporal side (Liu et al., 12 Dec 2025). That combination defines its main place in the literature.