Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements

Abstract: Effective human behavior modeling requires a representation of the human body movement that capitalizes on its compositionality. We propose a hierarchical representation consisting of Action Atoms that capture the atomic joint movements and Action Motifs that are formed by their temporal compositions and encode similar body movements found across different overall human actions. We derive A4Mer, a nested latent Transformer to learn this hierarchical representation from human pose data in a fully self-supervised manner. A4Mer splits a 3D pose sequence into variable-length segments and represents each segment as a single latent token (Action Atoms). Through bottom-up representation learning, temporal patterns composed of these Action Atoms, which capture meaningful temporal spans of reusable, semantic segments of body movements, naturally emerge (Action Motifs). A4Mer achieves this with a unified pretext task of masked token prediction in their respective latent spaces. We also introduce Action Motif Dataset (AMD), a large-scale dataset of multi-view human behavior videos with full SMPL annotations. We introduce a novel use of cameras by mounting them on the feet to achieve their frame-wise annotations despite frequent and heavy body occlusions. Experimental results demonstrate the effectiveness of A4Mer for extracting meaningful Action Motifs, which significantly benefit human behavior modeling tasks including action recognition, motion prediction, and motion interpolation.

• The paper introduces A4Mer, a self-supervised hierarchical framework that decomposes human motion into atomic actions (Action Atoms) and recurring patterns (Action Motifs) using nonlinear segmentation and clustering.
• It employs a JEPA-based masked token prediction in latent space to robustly capture semantic abstractions, even under heavy occlusion conditions.
• The approach outperforms fixed-length methods in action recognition, prediction, and interpolation, and leverages the large-scale AMD dataset for comprehensive evaluations.

Self-Supervised Hierarchical Representation of Human Body Movements with Action Motifs

Introduction

The paper "Action Motifs: Self-Supervised Hierarchical Representation of Human Body Movements" (2604.28173) introduces A4Mer, an unsupervised architecture for learning compositional and semantically meaningful representations of human movement. The approach centers on a hierarchical structure: Action Atoms capture fine-grained atomic movements, while Action Motifs encode recurring temporal compositions of these atomic units, abstracting both movement semantics and contextual variations inherent to daily human activities. In addition, the paper presents the Action Motif Dataset (AMD), a large-scale, multi-view dataset with per-frame SMPL annotations collected under heavy occlusion conditions using novel foot-mounted camera setups.

Hierarchical Movement Representation: Action Atoms and Motifs

A4Mer leverages the compositionality of body movement, capturing basic motions (Action Atoms) and their temporal structures (Action Motifs). By segmenting pose sequences at nonlinear kinematic transitions, the architecture extracts meaningful atomic units using simple trajectory cues. These units are clustered (k-means, $k=512$), with recurrent patterns across sequences discovered using the Generalized Sequential Pattern (GSP) algorithm. Dynamic programming ensures non-overlapping coverage of sequences, yielding variable-length Action Motifs, in contrast to fixed-length segments prevalent in prior approaches.

The hierarchical architecture is built from stacked encoders and latent Transformers. Segment-wise attention strictly limits intra-segment aggregation, facilitating robust and context-independent abstractions. Temporal relationships across segments are modeled by latent self-attention, and the architecture alternates bottom-up segment consolidation and top-down motif reasoning.

Self-Supervised Learning via Masked Token Prediction

A4Mer utilizes a Joint-Embedding Predictive Architecture (JEPA), solving a sequence-wise masked token prediction task in both hierarchy stages. The loss is computed in latent space, enabling the model to focus on semantic abstraction rather than pose reconstruction. To prevent trivial representational collapse, an exponential moving average and stop-gradient operation are employed. The model further decomposes latent tokens into global (sequence-level) and local (segment-wise) components, dynamically prioritizing the local term to suppress sequence biases and promote context-independence.

Hierarchical masking ensures semantic emergence: motifs are masked in the higher stage, with their constituent atoms simultaneously masked, suppressing leakage and enforcing robust learning of compositional structure.

The Action Motif Dataset (AMD): Multi-View, Occlusion-Robust Annotation

AMD is a substantial contribution, providing diverse daily activity sequences in a furnished real home with 24 synchronized room cameras and novel foot-mounted cameras. Leveraging ChArUco markers and calibrated foot camera poses, the authors achieve robust SMPL fitting under frequent severe occlusion, outperforming conventional ceiling-only or MoCap annotations in IoU and qualitative accuracy.

The dataset encompasses 14.2 hours of footage across 50 subjects performing multi-tasked, goal-directed activities in variable furniture arrangements, yielding variable-length, realistic motion sequences suitable for hierarchical representation learning.

Downstream Evaluation: Action Recognition, Motion Prediction, and Interpolation

A4Mer is evaluated on AMD and transferred to Humans in Kitchens (HiK) [38], using both zero-shot and transfer learning protocols. Rigorous comparison with state-of-the-art self-supervised methods (MotionBERT [53], BehaveMAE [34], MacDiff [43], H2T [19], USDRL [41], PUMPS [28]) demonstrates significant gains across major tasks:

• Action Recognition: Zero-shot k-NN classification and Transformer-based transfer learning head show that Action Motifs extracted by A4Mer achieve markedly higher accuracy than all baseline representations, evidencing strong semantic awareness and robust compositional encoding.
• Motion Prediction: Autoregressive latent token prediction and decoding, evaluated over a 3-second future horizon, yield lower MPJPE and semantically meaningful motion forecasts, outperforming prior representations that fail to capture long-term dynamics.
• Motion Interpolation: Given partial sequence observations, interpolation in latent space reconstructs occluded motion, producing non-linear, semantically correct movement recovery—unachievable by linear pose-space interpolation or clip/frame-based baselines.

Ablation studies confirm the necessity of hierarchical segment-wise attention, global-local decomposition, and JEPA latent loss formulation, showing performance degradation when any component is omitted. Fixed-length segment representations collapse semantic boundaries, underscoring the advantage of learned, variable-length motifs.

Implications and Theoretical Contributions

A4Mer's methodology aligns with cognitive models of action interpretation, abstracting multi-scale structure and enabling robust, context-independent behavioral modeling. The variable-length motifs compress pose sequences by an order of magnitude compared to fixed-length approaches, yet are more effective for semantic tasks. AMD's diverse, occlusion-robust sequences foster future developments in daily activity modeling, rehabilitation, monitoring, and generative motion synthesis.

The JEPA approach in latent space is preferable to pose-space losses, yielding Manhattan-like geometry conducive to clustering and compositional pattern mining. The ability to operate with minimal domain-specific augmentation and annotation marks a fundamental step toward scalable, generalizable movement representation learning.

Speculation on Future Directions

The Action Motif paradigm opens avenues for hierarchical generative modeling, cross-modal transfer to video and text, fine-tuned downstream deployment in real-world human-robot interaction, and unsupervised discovery of latent behavioral primitives. AMD will likely serve as a benchmark for occlusion-robust annotation and compositional activity sequence modeling. Future architectures may extend to multi-agent and object-centric interactions, leveraging motif extraction for robust goal inference and reasoning in complex, naturalistic environments.

Conclusion

The paper presents A4Mer, the first fully self-supervised, hierarchical architecture for human movement representation capturing atomic actions and their compositional motifs. Experimentally, Action Motifs outperform fixed-length representations in action recognition, prediction, and interpolation, demonstrating robust abstraction, semantic awareness, and contextual independence. The Action Motif Dataset enables rigorous evaluation and will accelerate progress toward fundamentally compositional, scalable human behavior modeling (2604.28173).