An Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

Abstract: Deep generative models provide flexible frameworks for modeling complex, structured data such as images, videos, 3D objects, and texts. However, when applied to sequences of human skeletons, standard variational autoencoders (VAEs) often allocate substantial capacity to nuisance factors-such as camera orientation, subject scale, viewpoint, and execution speed-rather than the intrinsic geometry of shapes and their motion. We propose the Elastic Shape - Variational Autoencoder (ES-VAE), a geometry-aware generative model for skeletal trajectories that leverages the transported square-root velocity field (TSRVF) representation on Kendall's shape manifold. This representation inherently removes rigid translations, rotations, and global scaling of shapes, and temporal rate variability of sequences, isolating the underlying shape dynamics. The ES-VAE encoder maps skeletal sequences to a low-dimensional latent space incorporating the Riemannian logarithm map, while the decoder reconstructs sequences using the corresponding exponential map. We demonstrate the effectiveness of ES-VAE on two datasets. First, we analyze skeletal gait cycles to predict clinical mobility scores and classify subjects into healthy and post-stroke groups. Second, we evaluate action recognition on the NTU RGB+D dataset. Across both settings, ES-VAE consistently outperforms standard VAEs and a range of sequence modeling baselines, including temporal convolutional networks, transformers, and graph convolutional networks. More broadly, ES-VAE provides a principled framework for learning generative models of longitudinal data on pose shape manifolds, offering improved latent representation and downstream performance compared to existing deep learning approaches.

• The paper presents a geometry-aware VAE embedding skeleton trajectories in Kendall’s shape space, achieving invariance to translation, rotation, scale, and execution speed.
• The paper leverages TSRVF alignment to remove temporal variability, accurately capturing nonlinear motion compared to traditional Euclidean methods.
• The paper demonstrates improved clinical gait analysis and action recognition with interpretable latent dimensions that correlate with real-world biomechanical metrics.

Elastic Shape Variational Autoencoder for Skeleton Pose Trajectories

Introduction and Motivation

This paper presents the Elastic Shape Variational Autoencoder (ES-VAE), a generative model for sequential skeletal data that leverages geometry-aware representation on Kendall’s shape manifold, combined with transported square-root velocity field (TSRVF) alignment. Conventional VAEs, when applied to skeleton pose sequences, allocate significant capacity to nuisance factors—such as translation, rotation, scale, and execution speed—rather than to intrinsic shape dynamics. ES-VAE addresses this by embedding skeleton trajectories in shape space, thereby achieving invariance to these confounding factors and focussing the latent representation on meaningful pose and movement variability. The methodology is validated on clinical gait analysis for stroke assessment and on action recognition from the NTU RGB+D dataset, demonstrating robust gains over both traditional and deep learning baselines.

Geometric Modeling and Alignment

Kendall Shape Space and TSRVF Alignment

Each skeleton is treated as a landmark configuration in $\mathbb{R}^{k \times m}$, where translation is removed via centering, scale through normalization, and rotation by quotienting using the special orthogonal group. The resulting data reside on Kendall's shape manifold, ensuring invariance to rigid body transformations. Sequences are modeled as curves on this manifold, parameterized by time.

Temporal variability caused by execution speed is removed via the TSRVF framework, which employs parallel transport of velocity vectors to a common reference point (Fréchet mean) and subsequent temporal alignment through dynamic programming. This combination yields a rate- and position-invariant representation of entire skeleton trajectories.

ES-VAE Architecture

The ES-VAE implements a Riemannian variational autoencoder:

• Encoder: Inputs tangent vectors (aligned trajectory differences from the mean) and outputs latent codes, parameterized with Gaussian distributions.
• Decoder: Maps latent vectors to tangent space, reconstructs trajectories via the exponential map back onto the shape manifold.
• Loss Function: Uses a Riemannian Evidence Lower Bound (ELBO) with reconstruction defined by squared geodesic distances, regularized by a diagonal Gaussian KL prior.

The architecture explicitly learns nonlinear generative mappings in shape space, eschewing the linearity limitations of methods such as tangent PCA.

Empirical Evaluation

Synthetic Manifold Data

On synthetic datasets sampled from nonlinear submanifolds of the unit sphere $\mathbb{S}^2$, ES-VAE demonstrates superior ability to recover the true underlying curve compared to Euclidean PCA, standard VAE, and tangent PCA. The latter methods are constrained either by Euclidean assumptions or linearity in tangent space, while ES-VAE accurately tracks non-geodesic structure.

Figure 1: ES-VAE recovers nonlinear structure in synthetic $\mathbb{S}^2$ submanifold data more closely than PCA, Euclidean VAE, or Tangent PCA.

Clinical Gait Analysis

On a dataset of 155 subjects (111 healthy, 44 stroke), ES-VAE achieves $R^2 = 0.74$, RMSE $= 2.82$, and Pearson $r = 0.86$ for POMA score prediction, outperforming LSTM ($R^2 = 0.64$) and tangent PCA ($R^2 = 0.70$). Latent dimensions correspond to clinically meaningful motions—stride length, limb stiffness, arm variability—and map strongly to clinical variables. ES-VAE is also the top performer for three-class classification (healthy vs. left/right hemiplegia), attaining macro F1 of 0.83.

Action Recognition

On a challenging NTU RGB+D subset, ES-VAE reaches macro F1 of 0.56, outstripping both classical skeleton encoders and deep learning baselines, with the improvement most pronounced for actions distinguished by shape dynamics rather than static poses.

Latent Space Analysis and Interpretability

ES-VAE’s latent dimensions exhibit interpretable clinical and biomechanical relevance:

• $z_1$ encodes stride length and limb stiffness, separating healthy from stroke gait.
• $z_2$ and $\mathbb{S}^2$0 quantify left arm and right knee variability, paralleling hemiplegic movement.
• $\mathbb{S}^2$1 is linked to lesion laterality, capturing side-specific impairment.

  Figure 2: ES-VAE latent modes: $\mathbb{S}^2$2 (low stride, stiff limbs), $\mathbb{S}^2$3 (left arm variability), $\mathbb{S}^2$4 (left arm/right knee), $\mathbb{S}^2$5 (right arm), $\mathbb{S}^2$6 (elbow subtlety).

Clinical and demographic correlations reinforce that latent dimensions are non-redundant and align with established mobility metrics.

Figure 3: $\mathbb{S}^2$7 and $\mathbb{S}^2$8 inversely correlate with POMA score; $\mathbb{S}^2$9 links to lesion laterality.

Comparative Analysis

Tangent PCA, while benefiting from manifold alignment, is constrained to linear geodesic variability and less effective decomposition of biomechanical features. ES-VAE yields richer, disentangled latent representations and improved downstream prediction. Latent codes are separable by clinical class, supporting practical stratification in real-world datasets.

Figure 4: ES-VAE latent codes separate stroke and healthy subjects; correlation matrix confirms independence across dimensions.

Implications, Limitations, and Future Directions

ES-VAE demonstrates that explicit geometric preprocessing (translation/scale/rotation removal, temporal alignment) substantially enhances model expressiveness and interpretability. The nonlinear manifold prior enables robust, clinically relevant compact representation and supports both regression and classification tasks in skeletal motion analysis.

Limitations include modest cohort sizes and single-site data for clinical evaluation, as well as class imbalance in stroke subtypes. The current model uses only kinematic input and deliberately restricts encoder-decoder complexity for interpretability; extension to multimodal inputs (e.g., EMG) and more expressive architectures is a viable path forward.

Future research directions include scaling ES-VAE to large, multi-center datasets, personalized modeling via conditional VAEs, integration of muscle activation signals, and deployment for longitudinal rehabilitation monitoring.

Conclusion

This work establishes ES-VAE as an effective method for geometry-aware generative modeling of skeleton pose trajectories. By embedding sequences in Kendall shape space and aligning via TSRVF, the model achieves invariance to key nuisance variables, yielding interpretable, compact latent spaces that support robust clinical and action recognition performance. ES-VAE’s approach—removing geometric noise before learning—offers advantages over end-to-end deep sequence models and linear shape analysis, positioning it as a principled framework for manifold-based longitudinal pose modeling in biomedical and activity recognition contexts.