---
title: 'TSkel-Mamba: Hybrid Skeleton Action Recognition'
url: https://www.emergentmind.com/topics/tskel-mamba
type: topic
---

# TSkel-Mamba: Hybrid Skeleton Action Recognition

Searching arXiv for TSkel-Mamba and closely related skeleton/Mamba papers.
TSkel-Mamba is a skeleton-based action recognition framework that combines a Spatial Transformer for per-frame joint-relation modeling with a Mamba-based temporal module for long-range sequence dynamics. In the form introduced in “TSkel-Mamba: Temporal Dynamic Modeling via State Space Model for Human Skeleton-based Action Recognition” [2512.11503], it is a hybrid Transformer–Mamba architecture designed for sequences of human skeletons represented as 3D joint coordinates, with the central claim that temporal selective state-space modeling should be augmented with explicit cross-channel temporal interaction. In a broader usage, “TSkel-Mamba” can also denote the more general design pattern of applying Mamba or selective state-space models to temporal skeleton modeling while retaining graph- or topology-aware spatial priors, a pattern also visible in related systems such as Simba [2404.07645] and STG-Mamba [2507.06689].

## 1. Definition and problem setting

TSkel-Mamba addresses **human skeleton-based action recognition**, where the input is a sequence of skeletons and the output is an action label [2512.11503]. A skeleton at frame $t$ with $N$ joints is represented as $X_t \in \mathbb{R}^{3 \times N}$, and a sequence is $\{X_t\}_{t=1..T}$ [2512.11503]. The underlying problem requires simultaneous modeling of spatial dependencies among joints within a frame and temporal dependencies across frames, including long-term and multi-scale motion patterns [2512.11503].

The method is motivated by a perceived division of labor between architectural families. Transformers are effective at long-range dependency modeling but incur quadratic complexity in sequence length; CNN and TCN temporal modules are efficient but predominantly local; Mamba provides linear-time temporal modeling through selective state-space scanning, but in its standard form processes channels independently and therefore under-exploits cross-channel temporal interactions [2512.11503]. TSkel-Mamba is formulated as a direct response to this limitation by pairing **Spatial Transformer** blocks with a **Temporal Dynamic Modeling (TDM)** block that uses Mamba while explicitly injecting multi-scale temporal-channel interactions [2512.11503].

The task formulation in [2512.11503] uses multiple standard skeleton modalities: raw joint coordinates, bone vectors, joint motion, and bone motion, fused by summing softmax scores in two-stream or four-stream configurations. This positions TSkel-Mamba within the mainstream multi-stream action-recognition literature rather than as a single-modality departure [2512.11503].

## 2. Architectural composition

TSkel-Mamba is organized as a stack of hybrid layers in which **Spatial Transformer (ST)** modules perform spatial feature learning and **TDM** modules perform temporal modeling [2512.11503]. The high-level processing sequence is: input skeleton sequence, tokenization per frame, Spatial Transformer with relative positional encoding, features $H \in \mathbb{R}^{B \times C \times T \times N}$, TDM, repeated over $L$ layers, and finally a recognition head with pooling and fully connected classification; optionally, covariance pooling with knowledge distillation is used during training [2512.11503].

The Spatial Transformer treats joints as tokens and uses multi-head self-attention with relative positional encoding constructed from shortest-path distances on the human-body topology [2512.11503]. This preserves a topological prior while allowing long-range spatial interactions beyond direct anatomical adjacency. The backbone is described as a **7-layer hybrid backbone** with **216 channels** and **9 heads** [2512.11503].

The TDM block is the defining temporal component. Its internal sequence is: normalization and $1 \times 1$ channel projection to $C/2$, Multi-scale Temporal Interaction (MTI), pure temporal scanning per joint in forward and reversed order, bidirectional selective SSM with token-dependent gating, concatenation of the two streams, layer normalization, temporal pooling, and residual fusion [2512.11503]. The paper explicitly characterizes TDM as a **plug-and-play** module and reports that replacing early temporal layers in existing backbones yields gains in CTR-GCN, Block-GCN, and Hyperformer [2512.11503].

A useful comparative context is provided by related Mamba-based skeleton systems. Simba places a Mamba block at the bottleneck of a U-shaped ShiftGCN encoder–decoder, flattening all joints in a frame to a single per-frame embedding before temporal scanning [2404.07645]. STG-Mamba, although aimed at music-guided dance video synthesis rather than action recognition, also implements Mamba-like temporal skeleton modeling using graph-structured spatial operations plus forward and backward temporal SSM scans [2507.06689]. This suggests that TSkel-Mamba belongs to a broader family of “temporal skeleton Mamba” designs, but it is distinguished by its explicit Spatial Transformer front-end and its MTI-enhanced temporal block [2512.11503].

## 3. State-space modeling and temporal dynamics

The temporal core of TSkel-Mamba is grounded in continuous-time and discrete-time state-space modeling. The paper gives the continuous-time equations
$$
\dot{x}(t) = A(t)x(t) + B(t)u(t), \qquad y(t) = C(t)x(t) + D(t)u(t),
$$
and the discrete-time form
$$
x_t = \bar{A}_t x_{t-1} + \bar{B}_t u_t, \qquad y_t = \bar{C}_t x_t + \bar{D}_t u_t,
$$
with time-varying parameters induced by discretization [2512.11503]. The Mamba interpretation is that token-dependent parameterization and gating turn this into a selective, context-adaptive linear-time scan over sequence length [2512.11503].

The paper presents a schematic selective update with gate
$$
g_t = \sigma(W_g u_t + b_g), \qquad z_t = g_t \odot f(x_t),
$$
thereby emphasizing that relevant information is selected dynamically during temporal propagation [2512.11503]. The important design claim is not merely that linear-time recurrence is used, but that temporal modeling remains **streaming-friendly** and scales as $O(T \cdot C)$ per sequence [2512.11503].

The key criticism of standard Mamba within this domain is that it commonly assigns separate SSM dynamics to individual channels, which limits explicit inter-channel temporal coupling [2512.11503]. In skeleton understanding, cross-channel coordination is treated as indispensable because subtle actions often rely on coherent evolution across coordinate axes, joints, or learned feature subspaces [2512.11503]. TSkel-Mamba therefore inserts a dedicated interaction mechanism before the SSM scan rather than relying on the SSM alone to discover such dependencies.

This design differs from Simba’s intermediate Mamba block, which also uses selective SSM scanning over temporal embeddings but is embedded between spatial encoder and decoder components of a U-ShiftGCN [2404.07645]. It also differs from STG-Mamba, whose SG-SSM, TGF-SSM, and TGB-SSM divide the problem into per-frame spatial modeling and bidirectional temporal scans over skeleton trajectories for a generative task [2507.06689]. TSkel-Mamba’s particular contribution is the assertion that temporal scanning should remain **per joint** and be explicitly preceded by temporal-channel interaction [2512.11503].

## 4. Temporal Dynamic Modeling and Multi-scale Temporal Interaction

The **Temporal Dynamic Modeling (TDM)** block is introduced to strengthen Mamba’s temporal capability for skeleton data while compensating for its per-channel limitation through the **Multi-scale Temporal Interaction (MTI)** module [2512.11503]. After pre-normalization and $1 \times 1$ projection to $C/2$, MTI applies **Cycle operators** that inject cross-channel temporal interactions by assigning channel-index-dependent temporal offsets [2512.11503].

For a projected feature tensor $\tilde{H} \in \mathbb{R}^{B \times C/2 \times T \times N}$, the Cycle operator with kernel size $K$ uses
$$
\delta_t(c) = (c \bmod K) - \lfloor K/2 \rfloor,
$$
for example producing offsets $\{-1,0,+1\}$ when $K=3$ [2512.11503]. The operator is defined as
$$
f_{\mathrm{Cycle}}^K(\tilde{H})(:, t, n) = \sum_{c=0}^{C/2-1} \tilde{H}(c, t+\delta_t(c), n)W_c + b,
$$
and the multi-scale aggregation is
$$
f_{\mathrm{MTI}}(\tilde{H}) = \tilde{H} + \sum_{K \in S_K} f_{\mathrm{Cycle}}^K(\tilde{H}),
$$
with scales such as $\{1,3,5\}$ [2512.11503].

The stated purpose of MTI is to create **multi-scale temporal-channel mixing** before selective scanning, thereby enriching motion representation with a small computational overhead [2512.11503]. After MTI, TDM performs **pure temporal scanning per joint**, constructing forward and backward sequences, applying bidirectional selective SSM with gating, flipping the backward stream back to chronological order, concatenating the two streams, and then applying normalization, temporal pooling, and residual fusion [2512.11503].

Ablation studies in [2512.11503] isolate the effect of these components on NTU120 X-Sub with joint input. Starting from a baseline of **84.2%**, T-Scan Mamba reaches **85.8**, adding MTI single-scale ($K=3$) reaches **86.4**, MTI multi-scale with $K \in \{1,3,5\}$ reaches **86.7**, and the full TDM with bidirectionality and pooling reaches **87.4** [2512.11503]. Additional ablations report that **T-Scan** outperforms TS-Scan, ST-Scan, and S-Scan, and that multi-scale $\{1,3,5\}$ performs better than $\{3\}$, $\{1,3\}$, and $\{1,3,5,7\}$ [2512.11503]. These results are used in the paper to argue that temporal-only per-joint scanning is more suitable than flattened spatiotemporal scanning and that overly coarse temporal receptive fields may be detrimental [2512.11503].

A plausible implication is that TSkel-Mamba’s temporal modeling strategy is less an unmodified adoption of Mamba than a domain-specific reformulation in which selective SSM is embedded within a structured preconditioning step tailored to skeleton sequences.

## 5. Spatial modeling, topology, and relation to graph-based methods

The spatial side of TSkel-Mamba is handled by a **Spatial Transformer** rather than a graph convolutional operator [2512.11503]. Its relative positional encoding is derived from shortest-path distances on the physical skeleton graph, so the attention mechanism remains topology-aware despite its global receptive field [2512.11503]. In single-head form, the paper writes
$$
H_{SA} = \mathrm{softmax}(QK^\top + QR^\top)V,
$$
where $R$ is a learnable tensor parameterized by graph distances [2512.11503].

This choice places TSkel-Mamba in a distinct position relative to earlier skeleton Mamba systems. Simba argues that plain Transformers underperform graph-convolutional methods for skeleton action recognition because of the lack of strong structural priors, and therefore keeps Shift S-GCN as the dominant spatial mechanism while inserting Mamba only at the temporal bottleneck [2404.07645]. STG-Mamba likewise uses GraphConv1d to facilitate spatial message passing among joints before applying SG-SSM and temporal SSMs [2507.06689].

TSkel-Mamba instead uses Transformer attention for spatial modeling but constrains it through topology-aware relative positional encoding [2512.11503]. This suggests an intermediate position between pure graph-prior architectures and unconstrained attention. The paper’s reported improvements over strong spatial-dominant baselines such as Block-GCN and Hyperformer are presented as evidence that this combination of topology-aware attention and TDM is effective [2512.11503].

The treatment of spatial and temporal modeling is therefore deliberately decoupled. Spatial Transformer handles long-range joint relations within each frame; TDM handles temporally extended motion patterns with near-linear temporal cost [2512.11503]. This decomposition is central to the method’s identity and to the explanation given for its empirical performance.

## 6. Datasets, training procedure, and empirical results

TSkel-Mamba is evaluated on **NTU-RGB+D 60**, **NTU-RGB+D 120**, **NW-UCLA**, and **UAV-Human** [2512.11503]. The paper states that NTU-RGB+D 60 contains **56,880 samples** with **25 joints** and uses **Cross-Subject (X-Sub)** and **Cross-View (X-View)** splits; NTU-RGB+D 120 contains **114,480 samples** with **25 joints** and uses **Cross-Subject (X-Sub)** and **Cross-Setup (X-Set)**; NW-UCLA uses **20 joints** and a cross-view split; UAV-Human contains **22,476 clips** and **155 classes** with the **CSv1** split [2512.11503].

Input sequences are resized to **64 frames** using random cropping and bilinear interpolation following prior practice, with batch sizes **64** for NTU60/120 and **16** for NW-UCLA [2512.11503]. Training uses **Python/PyTorch** on a **single RTX 4090 GPU**, **SGD**, weight decay **$4 \times 10^{-4}$**, initial learning rate **0.025**, decay by **$\times 0.1$** during epochs **110–120**, and a total of **120 epochs** [2512.11503]. The loss is cross-entropy for classification, with an optional knowledge-distillation term when covariance pooling is used [2512.11503].

The principal reported top-1 accuracies are as follows.

| Dataset / split | Configuration | Accuracy |
|---|---:|---:|
| NTU60 X-Sub | joint | 91.4 |
| NTU60 X-Sub | two-stream | 92.9 |
| NTU60 X-Sub | four-stream | 93.1 |
| NTU60 X-View | joint | 95.8 |
| NTU60 X-View | two-stream | 96.8 |
| NTU60 X-View | four-stream | 97.2 |
| NTU120 X-Set | joint | 88.7 |
| NTU120 X-Set | two-stream | 90.9 |
| NTU120 X-Set | four-stream | 91.6 |
| NTU120 X-Sub | joint | 87.4 |
| NTU120 X-Sub | two-stream | 89.7 |
| NTU120 X-Sub | four-stream | 90.4 |
| NW-UCLA | TSkel-Mamba | 97.0 |
| UAV-Human CSv1 | TSkel-Mamba | 47.2 |

With the optional CPKD training strategy, the paper reports improved numbers such as **91.6/93.0/93.2** on NTU60 X-Sub, **96.2/97.1/97.4** on NTU60 X-View, **88.9/91.0/91.7** on NTU120 X-Set, **87.9/90.0/90.6** on NTU120 X-Sub, and **97.2** on NW-UCLA [2512.11503].

The comparative claims are concrete. Against Hyperformer, TSkel-Mamba is reported as **+0.9%** on NTU120 X-Set and **+1.3%** on NTU120 X-Sub with fewer FLOPs; against Koopman it reports **+1.4%** and **+1.0%** on NTU60 X-Sub/X-View joint and **+2.2%** and **+1.4%** on NTU120 X-Sub/X-Set joint with approximately half the parameters; on UAV-Human it reports **+3.0%** over Koopman in CSv1 [2512.11503]. The paper also states that TDM improves the ten hardest NTU120 X-Sub classes, with a maximum gain of **+16.18%** and average gain **+9.38%**, while gains on **105 classes** average **+3.76%** and negatives on **10 classes** average **−0.94%** [2512.11503].

For broader context, Simba reports **96.34** top-1 on NW-UCLA in four-stream form and strong results on NTU datasets using a U-ShiftGCN plus Mamba bottleneck [2404.07645]. This indicates that Mamba-based temporal modeling had already become competitive in skeleton action recognition before TSkel-Mamba, though under a different architectural philosophy.

## 7. Efficiency, optional CPKD, limitations, and broader significance

Efficiency is a central part of the TSkel-Mamba argument. The paper reports **2.4M parameters**, **8.2G FLOPs**, and **87.9%** on NTU120 X-Sub for the joint configuration [2512.11503]. In the same comparison, **ST-TR** is reported at **12.1M**, **259.4G**, **82.7%**; **DSTA-Net** at **3.4M**, **16.2G**, **84.0%**; **Hyperformer** at **2.7M**, **9.6G**, **86.6%**; and **Koopman pooling** at **5.3M**, **8.8G**, **85.7%** [2512.11503]. The paper further states that linear-time temporal scan enables high throughput and that, under the same hardware, the method achieves the best accuracy with competitive inference time [2512.11503].

An additional training mechanism is **Covariance Pooling with Knowledge Distillation (CPKD)** [2512.11503]. Covariance pooling replaces global average pooling with a second-order representation built from covariance $\Sigma = O \hat{I} O^\top$, applies a matrix square root through Newton–Schulz iteration, vectorizes the upper triangular part, and feeds it to a classifier [2512.11503]. To avoid inference overhead, the covariance-pooling teacher is distilled into a GAP student using
$$
L_{KD}(P^S, P^T) = \alpha \, KL(P_b^S \| P_b^T) + \beta \, KL(P_m^S \| P_m^T),
$$
with $\alpha = 1$ and $\beta = 8$ [2512.11503]. The paper states that this improves accuracy without changing student parameters or FLOPs and notes that compact students such as **0.9M params, 3.5G FLOPs** can reach **87.7%** [2512.11503].

The limitations stated in [2512.11503] are restrained rather than expansive. TDM is described as somewhat heavier than pure TCNs; gains are less pronounced on very short sequences; future work includes adaptive scale selection, learnable data-dependent Cycle permutations, jointly learned channel grouping and offsets, multimodal fusion with RGB or IMU and language supervision, online forward-only variants, and tighter integration of second-order temporal statistics [2512.11503]. These are best read as targeted development directions rather than as evidence of instability or failure.

Within the broader Mamba-for-skeleton literature, TSkel-Mamba can be understood as a specific maturation of the “temporal skeleton Mamba” concept. Simba demonstrates that inserting Mamba into a graph-convolutional U-shaped backbone can materially improve skeleton action recognition [2404.07645]. STG-Mamba shows that graph-aware, bidirectional Mamba-style scanning over skeleton sequences is useful even in a different task domain, namely music-guided dance video synthesis [2507.06689]. TSkel-Mamba extends this trajectory by arguing that temporal selective state-space modeling for skeletons should be **per-joint, bidirectional, topology-aware on the spatial side, and explicitly cross-channel on the temporal side** [2512.11503]. That combination defines its main place in the literature.

Source: https://www.emergentmind.com/topics/tskel-mamba