Multimodal Skeleton Action Representation

• Multimodal skeleton-based action representation is a framework that integrates skeletal data with complementary modalities to capture dynamic action cues.
• It leverages early fusion, cross-modal alignment, and deep sequence modeling to enhance recognition, segmentation, and transfer performance.
• Recent advances incorporate language prompts and energy-efficient spiking networks to achieve robust action understanding with reduced computational cost.

Multimodal skeleton-based action representation encompasses a range of methodologies designed to encode, fuse, and leverage skeletal data in conjunction with complementary modalities—most commonly appearance-based (RGB, optical flow), language, and dynamic (event-based) inputs—for robust, discriminative, and efficient action understanding. Recent advances have integrated deep sequence modeling, cross-modal alignment, and semantic knowledge distillation to address limitations of unimodal skeleton representations, yielding state-of-the-art results across recognition, segmentation, and transfer scenarios.

1. Multimodality in Skeleton-Based Action Representation

Multimodal skeleton-based action representation exploits the complementary nature of diverse input sources. Canonical skeleton modalities include (a) joint coordinates, (b) bone vectors, and (c) temporal motion or velocity features; advanced schemes further admit acceleration, rotation-axis direction, joint angular velocity, 2D/3D skeletons, and even cross-modal signals—RGB frames, optical flow, language prompts, and event camera data.

The rationale for multimodality is driven by the recognition that each modality captures distinctive facets of the action manifold. For instance, joint coordinates reflect absolute postures, bone vectors encode kinematic invariance, and motion/velocity highlights dynamic cues. Appearance-based streams supply context and disambiguate visually similar skeleton sequences, while language or high-level prompts can inject semantic regularization. Event data, being inherently sparse, brings power-efficient dynamic sensing.

Crucially, the integration strategy is nontrivial. Early fusion (feature-level aggregation before encoding), late fusion (post-classification or decision fusion), and hybrid self-supervised approaches (cross-modal distillation, agent-driven feedback, contrastive learning) each bear distinctive trade-offs in representational power, inference efficiency, and robustness to missing modalities (Wang et al., 24 Dec 2025, Sun et al., 2023, Song et al., 2020, Liu et al., 2024, Liu et al., 27 Nov 2025).

2. Fusion Architectures and Alignment Strategies

State-of-the-art frameworks deploy sophisticated mechanisms for multimodal fusion and alignment:

• Early Fusion and Unified Encoders: UmURL and Decomposition-Composition frameworks ingest multiple modes (joints, bones, motion) through linear embeddings, followed by sequence encoders (Transformer, GCN). Embeddings are averaged or composed to form a fused representation. Critical supervision involves both intra-modal (decomposition: mapping fused features back to unimodal projections) and inter-modal (composition: aligning aggregate unimodal and fused representations) losses, ensuring the fused space preserves discriminative cues from all modalities. This approach supports missing-modality inference: the shared encoder processes whatever modalities are available without performance collapse (Wang et al., 24 Dec 2025, Sun et al., 2023).
• Cross-Modal Alignment and Compensation: The Modality Compensation Network (MCN) uses residual LSTM blocks on RGB and optical flow sources, regularized via a skeleton-informed adaptation loss. During training, skeleton features (from a frozen auxiliary LSTM) serve as guidance targets at domain, class, or sample level (using MMD or L2 losses). No skeletons are needed at inference; the system "compensates" for missing skeleton information by having trained the remaining modalities to project into the skeleton-informed subspace (Song et al., 2020).
• Knowledge Distillation and Agentic Interaction: Recent models leverage language or multimodal prompts for higher-order supervision. SkeletonAgent creates a feedback loop between a LLM and the skeleton action recognizer. The Questioner agent identifies confused action classes and crafts targeted prompts; the Selector parses the LLM's discriminative descriptions to extract joint constraints and semantic embeddings, which then modulate the recognizer through explicit constraint losses and cross-modal alignment terms (Liu et al., 27 Nov 2025).
• Contrastive and Soft Target Learning: Vision-language frameworks (e.g., C²VL) generate vision-language action concepts using multimodal models like Grounding DINO (vision) and LLaVA (language), then progressively distill this knowledge to the skeleton encoder using a curriculum alternating between intra-modal and inter-modal soft targets. Generative Action-description Prompts (GAP) contrast body-part-wise skeleton embeddings against GPT-generated textual action descriptions using a bidirectional contrastive loss, yielding notable improvements for fine-grained action recognition tasks (Chen et al., 2024, Xiang et al., 2022).
• Hybrid and Hierarchical Spiking Fusion: Spiking Graph Convolutional Networks (SNN-based, e.g., MK-SGN, SNN-driven event fusion) first quantize skeleton and additional dynamic modalities into spike trains, fuse them via multifaceted spiking architectures, and apply knowledge distillation from high-capacity GCN teachers for both accuracy and power efficiency gains (Zheng et al., 2024, Zheng et al., 19 Feb 2025).

3. Modalities and Multi-Stream Variants

The field has standardized around several fundamental and auxiliary skeleton-derived modalities:

Skeleton Form	Definition and Role	Common Use
Joint (J)	Absolute 2D/3D positions	Global postures
Bone (B)	Vector difference between connected joints	Kinematic invariance
Motion (M)	Framewise velocity (Δ coords)	Dynamic change
Acceleration	Δ (velocity)	Fine motion transitions
Rotation Axis	Normalized cross-product of bone pairs	Rotation semantics
Angular Vel.	Rate of change, hinge angle × axis	Articulated dynamics
2D/3D Skeleton	Lifted or native, device-agnostic	Camera or pose estimation
Event Streams	Sparsified camera events	Energy-efficient motion
RGB / Optical	Appearance/texture, context	Scene priors, disambig.
Language	Text descriptions, semantic regularizers	High-level priors

Many systems process these modalities with distinct backbones and ensemble their predictions, while others distill all cues into a shared, robust encoder. Adaptive cross-form and multimodal knowledge distillation enables single-form inference matching multi-form performance, even when only partial modalities are available (Wang et al., 2022, Wei et al., 2023).

4. Efficiency-Effectiveness Trade-Offs and Knowledge Transfer

Efficiency remains a central concern, especially as high-parameter late-fusion or multi-tower models threaten latency and compute budgets. Recent advances have converged on the following strategies:

• Unified Backbones with Decomposition/Composition: Decomposition-Composition learning fuses linear embeddings of all modalities then supervises the fused feature to both reconstruct unimodal projections (Decomposition) and align with late-ensemble multimodal projections (Composition). This achieves near-late-fusion accuracy (e.g., 85.8%/91.8% NTU-60 linear eval) with unimodal computational cost and minimal parameter increases (Wang et al., 24 Dec 2025).
• Implicit Knowledge Exchange: In self-supervised configurations, modules like Implicit Knowledge Exchange Module (IKEM) implicitly aggregate cross-modal features by pulling each stream towards a multimodal anchor, reducing noisy knowledge transfer and enabling additional auxiliary modalities without inference-time cost (e.g., acceleration, rotation axis, angular velocity). Distillation into a three-stream student further streamlines deployment (Wei et al., 2023).
• Spiking Neural Representations, Event Fusion: MK-SGN demonstrates that spiking graph convolutional networks, with modality-specific spike-based encoding and knowledge distillation, can approach the accuracy of continuous GCNs (within 1.4–1.6 pp) at roughly 20% the energy (Zheng et al., 2024). Extending this, SNN-driven frameworks can fuse event and skeleton semantics using spiking information bottlenecks and semantic extraction modules, attaining both superior accuracy and ~8× energy efficiency over ANN baselines (Zheng et al., 19 Feb 2025).

5. Integration with Language, Vision, and Agentic Semantics

Recent trends show tight coupling of skeleton encoders with vision-LLMs or explicit agent-driven semantic priors:

• Language-Augmented Learning: By querying LLMs for action- or part-specific descriptions (SkeletonAgent, GAP), systems expose skeleton encoders to global and joint-level priors not captured by kinematics alone. These descriptions guide attention, provide soft targets for contrastive alignment, or inject global context into classifier outputs, yielding SOTA gains (e.g., +1–3% on FineGYM and UAV-Human) (Liu et al., 27 Nov 2025, Xiang et al., 2022).
• Multi-Modal Co-Learning: MMCL leverages LLM and CNN co-training as auxiliary networks, aligning RGB CNN features with skeleton backbones and refining classifier scores with LLM-instructed features. At inference, only the skeleton GCN remains, preserving strict efficiency while inheriting much of the generality and robustness provided by auxiliary modalities during training (Liu et al., 2024).
• Progressive Distillation from Vision-Language Concept Spaces: C²VL employs LMM-generated action concept spaces (video crops + textual prompts) for cross-modal contrastive pretraining, using a schedule that dynamically balances intra-modal and inter-modal target alignment for robust, task-agnostic representations. Inference requires only skeleton input. Transfer gains of up to 9–12% accuracy have been reported when leveraging such distillation for cross-dataset adaptation (Chen et al., 2024).

6. Empirical Performance and Benchmarking

Multimodal skeleton-based representation methods have converged on several datasets and evaluation paradigms:

• NTU RGB+D / RGB+D 120: Universal test-beds for 2D/3D skeleton, RGB, and cross-modal algorithms. Multimodal models with self-supervised fusion or cross-modal targets routinely surpass unimodal and even late-fusion baselines by 1–4% in accuracy under standard evaluation settings. For instance, Decomposition-Composition achieves 85.8% (x-sub) / 91.8% (x-view) on NTU-60 with a single backbone (Wang et al., 24 Dec 2025); SkeletonAgent yields +0.7% over prior SOTA on NTU RGB+D (x-sub) (Liu et al., 27 Nov 2025).
• UAV-Human and FineGYM: Represent challenging domains requiring fine-grained, robust multimodal reasoning. HDBN, SkeletonAgent, and MMCL report consistent improvements over both Transformer and GCN single-backbone models (e.g., +2–3% on UAV-Human CS splits for HDBN) (Liu et al., 2024, Liu et al., 27 Nov 2025).
• Energy-Constrained Deployments: On standard energy models, spiking GCNs and event-camera driven networks achieve 4–8× energy savings over ANN counterparts, with minor accuracy loss, demonstrating suitability for edge deployment (Zheng et al., 2024, Zheng et al., 19 Feb 2025).
• Self-Supervised and Transfer Tasks: Unified, multimodal backbone approaches demonstrate strong gains on semi-supervised (1–10% labels) and cross-dataset transfer tasks (e.g., USDRL, C²VL, Decomposition-Composition), confirming the utility of multimodal pretraining for data efficiency and generalization (Wang et al., 18 Aug 2025, Chen et al., 2024, Wang et al., 24 Dec 2025).

7. Open Challenges and Future Directions

Current limitations include reliance on high-quality modality alignment during training, the necessity for computationally intensive offline multimodal prompting (in language-augmented learning), and some degradation in extremely fine-grained or modality-degraded conditions (e.g., noisy 3D pose estimation, monocular depth). Ongoing directions highlighted in the literature:

• End-to-end fusion of noisy, in-the-wild modalities (event, depth, RGB, language).
• Adaptive cross-modal attention or gating for dynamic selection of relevant modalities per instance.
• Extension to dense prediction tasks (action segmentation, temporal localization) and broader application in robotics, real-time interactive systems, and neuromorphic computing environments (Wang et al., 24 Dec 2025, Wang et al., 18 Aug 2025, Liu et al., 27 Nov 2025, Zheng et al., 19 Feb 2025).

Multimodal skeleton-based action representation thus constitutes an evolving paradigm integrating cross-modal alignment, semantic regularization, and energy-efficient computation, underpinning highly generalizable and robust human action understanding across diverse deployment settings.