Unified Dense Skeleton Representation Learning

• The paper demonstrates that USDRL achieves dense, reusable features that retain key temporal and spatial structures for improved action recognition and prediction.
• It employs a dual-stream Dense Spatio-Temporal Encoder combining Dense Shift Attention and Convolutional Attention to preserve local and global features.
• USDRL outperforms prior methods across various benchmarks by avoiding momentum encoders and enabling effective transfer, retrieval, and dense prediction tasks.

Unified Skeleton-based Dense Representation Learning (USDRL) denotes a line of self-supervised skeleton representation learning that seeks a single reusable representation space for skeleton sequences, with dense spatio-temporal features at the frame, joint, or token level and a global embedding for downstream action understanding. In current usage, the term refers both to a specific feature-decorrelation framework centered on a Dense Spatio-Temporal Encoder (DSTE) and to a broader research objective spanning unified multimodal, heterogeneous, and vision-pretrained skeleton encoders for recognition, retrieval, dense prediction, and transfer (Weng et al., 2024, Wang et al., 18 Aug 2025).

1. Scope and problem formulation

USDRL is formulated on skeleton sequences represented as

$$\mathbf{X} \in \mathbb{R}^{C_{in} \times T \times V \times M},$$

where $T$ is the number of frames, $V$ the number of joints, $M$ the number of persons, and $C_{in}$ the input channels. The central claim of the USDRL formulation is that skeleton representation learning should not collapse the sequence immediately into a single instance vector, because dense prediction tasks require temporally and spatially localized features. The encoder is therefore designed to produce temporal dense representations $\mathbf{y}_t \in \mathbb{R}^{T \times C_r}$ and spatial dense representations $\mathbf{y}_s \in \mathbb{R}^{V \times C_r}$, while still supporting pooled sequence-level descriptors for coarse prediction (Wang et al., 18 Aug 2025).

This formulation is motivated by limitations in prior skeleton self-supervision. Negative-based contrastive methods require momentum encoders and memory banks, and predominantly target global instance representations for recognition and retrieval. Reconstruction-based methods can learn excessive low-level detail. USDRL positions dense skeleton features as a unifying substrate for action recognition, action retrieval, action detection, action segmentation, action prediction, and transferred prediction, rather than treating sequence classification as the sole downstream objective (Weng et al., 2024).

2. Core encoder design

The canonical USDRL architecture is the Transformer-based Dense Spatio-Temporal Encoder. It reshapes the input into a temporal view

$$\mathbf{X}_{t} \in \mathbb{R}^{T \times (M \times V \times C_{in})}$$

and a spatial view

$$\mathbf{X}_{s} \in \mathbb{R}^{(M \times V) \times (T \times C_{in})},$$

then linearly embeds them into $\mathbf{F}_{t} \in \mathbb{R}^{T \times C_e}$ and $T$0. The two streams are processed in parallel so that temporal dynamics and spatial structure remain explicit throughout the encoder (Weng et al., 2024).

Each DSTE layer combines Dense Shift Attention (DSA) and Convolutional Attention (CA). In DSA, an MLP over the sequence dimension produces

$T$1

followed by a DenseShift operation

$T$2

and self-attention plus feed-forward processing on both $T$3 and $T$4. CA applies 1D convolution over time or joints, adds a residual, and then applies self-attention and an FFN. The layer output is

$T$5

This combination preserves per-frame and per-joint alignment while injecting both local and global structure into the dense features (Weng et al., 2024).

In the later foundation-model formulation, DSTE is described as two parallel streams learning temporal dynamic and spatial structure features, with CA modeling local interactions and DSA modeling dense global dependencies. The same encoder family is then reused across coarse prediction, dense prediction, and transfer settings, which is the architectural basis for calling USDRL a skeleton foundation model (Wang et al., 18 Aug 2025).

3. Training objectives and representation regularization

The original USDRL training objective replaces negative-based contrastive learning with Multi-Grained Feature Decorrelation (MG-FD). Projected features are constructed in three domains: instance, temporal, and spatial. The total loss is

$T$6

with

$T$7

Here, $T$8 enforces intra-sample consistency across augmentations through similarity and invariance terms, while $T$9 enforces inter-sample separability through a variance term $V$0, an auto-covariance term $V$1, and a cross-correlation term $V$2. The stated purpose is to reduce redundancy among dimensions of the representations and maximize information extraction from features (Weng et al., 2024).

The 2025 extension retains MG-FD and adds Multi-Perspective Consistency Training (MPCT). MPCT comprises multi-view and multi-modal self-supervised consistency training. The multi-view component uses different camera viewpoints of the same action, with separate data augmentations, to enhance high-level semantics and mitigate low-level discrepancies. The multi-modal component uses joint, bone, and motion inputs with early fusion and a shared DSTE backbone, rather than late fusion with separate backbones. In this formulation, USDRL becomes not only negative-free but also explicitly viewpoint-aware and modality-aware (Wang et al., 18 Aug 2025).

A notable consequence of these objectives is that USDRL avoids momentum encoders, memory banks, and decoder-heavy reconstruction pipelines. This suggests a design preference for embedding-space regularization over explicit negative sampling or full signal recovery. The literature does not treat this preference as exclusive, but it is central to the specific USDRL framework defined in the 2024 and 2025 papers.

4. Downstream tasks and empirical profile

On the benchmarks reported in the original USDRL paper, joint-only USDRL with DSTE achieves 85.2% and 91.7% for unsupervised action recognition on NTU-60 X-sub and X-view, 76.6% and 78.1% on NTU-120 X-sub and X-set, and 54.4% on PKU-MMD II X-sub. The three-stream configuration reaches 87.1%, 93.2%, 79.3%, 80.6%, and 59.7% on the same settings. For semi-supervised NTU-60 recognition, DSTE reports 57.3% and 80.2% on X-sub with 1% and 10% labels, and 60.7% and 84.0% on X-view. For retrieval, joint-only DSTE reports 75.0% and 89.3% on NTU-60 and 63.3% and 66.7% on NTU-120. For dense prediction on PKU-MMD I, DSTE reaches 75.7% mAP$V$3 and 74.9% mAP$V$4 at IoU 0.5, substantially above prior self-supervised baselines reported in the same study (Weng et al., 2024).

The foundation-model version broadens the evaluation to 25 benchmarks across 9 skeleton-based action understanding tasks, explicitly covering coarse prediction, dense prediction, and transferred prediction. Reported dense-task results include temporal action detection, action segmentation, and action prediction. On PKU-MMD II action segmentation, the reported STTR-based USDRL result is 68.7% Acc, 67.5% Edit, and 56.2% F1@50. On NTU-60 action prediction, DSTE surpasses P-TSL at observation ratios of $V$5 and above, including 73.6% versus 67.4% at $V$6 observation. The same work also reports transfer learning from NTU-60 or NTU-120 to PKU-MMD II and transfer retrieval benchmarks, reinforcing the claim that dense skeleton features can serve as a common backbone across task families (Wang et al., 18 Aug 2025).

These results anchor the term “dense” in measurable downstream behavior. In USDRL, dense representation learning is not merely token retention inside the encoder; it is validated by frame-level localization, sequence retrieval, early action prediction, and cross-dataset reuse.

5. Related formulations and adjacent research directions

The broader literature contains several formulations that pursue the same unification objective through different input normalizations, pretext tasks, or architectural biases.

Framework	Unifying mechanism	Reported emphasis
ReL-SAR (Naimi et al., 2024)	Lightweight convolutional transformer, Selection-Permutation for 15 essential joints, BYOL without negatives	Robust general skeleton representations on limited-size datasets
UmURL (Sun et al., 2023)	Single-stream early fusion of joint, bone, and motion with intra- and inter-modal consistency	Unified multi-modal unsupervised representation with low complexity
MacDiff (Wu et al., 2024)	Semantic encoder plus masked conditional diffusion with random masking	Unified representation learning and generation
STJD-CL / STJD-MP (Gunasekara et al., 29 May 2025)	Learnable Spatio-Temporal Joint Density, prime-joint selection, contrastive and reconstruction objectives	Dense representations focused on moving and static discriminative joints
S2I (Yang et al., 6 Mar 2026)	Skeleton-to-image encoding into $V$7 with body-part canonicalization	Vision-pretrained skeleton learning and cross-format transfer
Heterogeneous Skeleton-Based Action Representation Learning (Wang et al., 4 Jun 2025)	2D-to-3D auxiliary network, prompted unified skeleton, semantic motion encoding, shared backbone	Unified action representation across heterogeneous skeleton formats

Taken together, these papers show that “unified” has been instantiated along several axes: modality unification through early fusion, topology unification through prompted or canonicalized skeleton layouts, objective unification through diffusion or decorrelation, and representation-format unification through image-like encoding. A plausible implication is that USDRL has evolved from a single negative-free framework into a broader systems question: how to define one skeleton representation space that remains reusable when the modality, topology, or pretraining regime changes.

6. Limitations, misconceptions, and open problems

A common misconception is that USDRL is identical to one specific negative-free decorrelation recipe. The literature suggests a broader usage. ReL-SAR treats the goal as a self-supervised skeleton encoder based on a lightweight convolutional transformer, Selection-Permutation, and BYOL; MacDiff treats it as masked conditional diffusion with a semantic encoder; UmURL treats it as early-fused multimodal learning with VICReg-style regularization; S2I treats it as a dense image-like encoding compatible with vision-pretrained models (Naimi et al., 2024, Wu et al., 2024, Sun et al., 2023, Yang et al., 6 Mar 2026).

Several limitations recur across these works. ReL-SAR is evaluated on relatively small datasets with 2D pose estimation, uses only 2D skeletons, and relies on a fixed 15-joint selection and body-part permutation (Naimi et al., 2024). STJD-based methods report slight performance drops on subtle hand-motion actions such as “Playing with a mobile phone” and “Type on the keyboard,” and note that posture variation can produce misdetected prime joints, motivating an extension from prime joints to prime parts (Gunasekara et al., 29 May 2025). S2I notes that interpolation to a fixed $V$8 grid can smooth fine-grained temporal changes, and that performance depends strongly on image pretraining, mask ratio, and masking type (Yang et al., 6 Mar 2026). The heterogeneous framework is limited to up to two people in its current implementation and centers on 25-joint and 17-joint heterogeneity rather than arbitrary morphologies (Wang et al., 4 Jun 2025). The foundation-model formulation still relies on fine-tuning for new tasks, does not yet provide strong zero-shot open-set recognition, and identifies stronger multimodal integration and text or semantic priors as future directions (Wang et al., 18 Aug 2025).

Open directions therefore follow directly from the present literature: larger and more diverse skeleton corpora, stronger zero-shot transfer, denser multimodal alignment, adaptive rather than fixed joint grouping, broader heterogeneous-topology support, and integration with larger vision or multimodal backbones. What remains stable across these proposals is the central USDRL premise: skeleton representation learning is most useful when the learned features are dense, reusable, and unified across tasks rather than optimized only for clip-level classification.