SSD-Poser: Real-Time Pose Estimation

• The paper introduces a novel model that leverages state-space duality, attention mechanisms, and frequency-aware decoding to achieve real-time full-body pose reconstruction with metrics such as 2.67 cm MPJPE and 143 FPS.
• The architecture employs a hybrid encoder combining the Pose State-Space Block and multi-head attention to extract rich spatiotemporal features, ensuring smooth and accurate avatar tracking.
• The frequency-aware decoder adeptly separates low- and high-frequency motion components to minimize jitter and enhance the perceptual realism of avatar movements.

SSD-Poser is a lightweight computational model for real-time full-body pose estimation from sparse signals, specifically head and hand positions commonly available in consumer AR/VR head-mounted displays (HMDs). It leverages recent advances in state-space models and combines them with attention mechanisms and frequency-aware decoding to efficiently reconstruct the full avatar pose with high accuracy and minimal inference latency (Zhao et al., 25 Apr 2025).

1. Background: State-Space Duality in Motion Modeling

State-Space Models (SSMs) underpin SSD-Poser’s theoretical framework. An SSM describes an evolving latent state $h_t$ and observation $y_t$ via coupled equations: $$h_t = A_t h_{t-1} + B_t x_t, \qquad y_t = C_t^\top h_t$$ where $x_t$ is input at time $t$, and $A,B,C$ parameterize the latent transitions and emissions. SSD-Poser adopts the dual formulation (State-Space Duality), wherein the output $y_t$ is computed as a parallelizable sequence of matrix multiplications: $$y_t = \sum_{i=1}^t C_t^\top (A_t \cdots A_{i+1}) B_i x_i$$ This reformulation enables linear computational scaling in sequence length $T$ and constant memory utilization. The SSD operator facilitates parallel scan-style computation, making it suitable for high-throughput inference when reconstructing human motion from sparse temporal signals.

2. Architecture: State Space Attention Encoder (SSAE)

SSD-Poser introduces a hybrid encoder to synthesize the strengths of both SSMs and Transformer architectures. The State Space Attention Encoder (SSAE) consists of two key submodules:

• Pose State-Space Block (PSSB): Extracts low-level spatiotemporal features and imposes efficient state-space recurrences on processed input. The input $V_i$ undergoes normalization and linear projections, with parallel depthwise convolutions and SiLU activations generating multi-stream features. These features are gated and then passed through an SSD-parameterized SSM. The output merges via linear projection and residual connection:

$$Y_i = \mathrm{Linear}\left(\mathrm{LN}(Y_i^{(2)} \odot Y_i^{(3)})\right) + V_i$$

where $\odot$ denotes element-wise multiplication.

• Attention Module (AM): Applies standard multi-head scaled dot-product attention and feedforward network layers to enhance long-range context modeling, following the Transformer paradigm. This yields improved perceptual realism in avatar tracking beyond the capability of pure SSMs.

The model stacks four SSAE blocks, yielding a compact backbone that balances context richness against computational burden.

3. Frequency-Aware Decoder (FAD)

Motion reconstruction fidelity suffers from jitter due to high-frequency fluctuations in sparse sensor input. SSD-Poser’s Frequency-Aware Decoder (FAD) addresses this by decomposing feature activations into distinct temporal bands:

• Low-Frequency Branch: Pointwise ($1\times1$) convolution with SiLU activation emphasizes slow, gross body movements.
• High-Frequency Branch: Temporal ($1\times5$) convolution, also with SiLU, targets rapid, fine-grained pose adjustments associated with jitter.
• Feature Concatenation: The output of all three streams (raw, low-frequency, high-frequency) is concatenated and normalized, then mapped to SMPL joint parameters via a final linear layer.

This separation allows the decoder to treat rapid, noisy fluctuations independently from stable pose components, thereby improving motion smoothness.

4. Training Protocol and Loss Functions

Training utilizes the AMASS dataset, which provides motion-captured data parameterized by SMPL. SSD-Poser is evaluated under two settings: 3 subsets (CMU, BMLrub, HDM05) and 14 subsets for broader generalization. 90% of data is allocated to training, 10% to testing, with sequence length $T=96$.

Supervision is provided via three $L_2$ losses:

• Rotation Loss:

$$L_{\mathrm{rot}} = \frac{1}{N} \sum_{i=1}^N \| R_i - \widehat{R}_i \|_2$$

• Position Loss:

$$L_{\mathrm{pos}} = \frac{1}{N} \sum_{i=1}^N \| P_i - \widehat{P}_i \|_2$$

• Orientation Loss (weighted):

$$L_{\mathrm{ori}} = \frac{1}{N} \sum_{i=1}^N \| O_i - \widehat{O}_i \|_2$$

• Composite Loss:

$$L = L_{\mathrm{rot}} + L_{\mathrm{pos}} + 0.02\,L_{\mathrm{ori}}$$

Optimization employs the Adam optimizer ($1\times10^{-5}$ weight decay, initial learning rate $3\times10^{-4}$ reduced to $3\times10^{-5}$ after 200k iterations, batch size 256) on high-end GPUs.

5. Quantitative Performance on Pose Estimation Tasks

SSD-Poser demonstrates competitive results against established baselines, as shown in the tables below.

Setting 1: CMU, BMLrub, HDM05

Method	MPJRE (°)	MPJPE (cm)	MPJVE (cm/s)	Jitter
AvatarPoser	4.18	4.18	27.70	14.49
AGRoL	3.71	3.71	18.59	7.26
KCTD	3.62	3.62	20.57	10.73
SAGE	3.28	3.28	20.62	6.55
SSD-Poser	3.15	3.15	19.32	8.19

Setting 2: Fourteen AMASS Subsets

Method	MPJRE (°)	MPJPE (cm)	MPJVE (cm/s)	Jitter
AvatarJLM	3.39	3.39	15.75	5.33
AGRoL	3.80	3.80	17.76	10.08
AvatarPoser	3.37	3.37	21.00	10.24
SAGE	2.95	2.95	16.94	5.27
SSD-Poser	2.67	2.67	15.25	6.73

MPJRE: Mean per-joint rotation error (°). MPJPE: Mean per-joint position error (cm). MPJVE: Mean per-joint velocity error (cm/s). Jitter: Motion smoothness metric.

6. Computational Profile

SSD-Poser achieves superior efficiency relative to competing methods:

Method	Parameters (M)	Time (s/seq)	≈FPS
KCTD	12.3	0.005	200
AGRoL	7.48	0.011	91
SAGE	137.4	0.035	29
SSD-Poser	7.34	0.007	143

With only 7.3 million parameters—comparable to basic MLP models—SSD-Poser sustains over 140 FPS on high-end GPUs. Its complexity is dominated by four lightweight attention layers and the linear-time SSM, with effective scaling in both training and inference.

7. Limitations and Prospective Directions

Despite real-time accuracy, SSD-Poser is subject to several limitations:

• The frequency decomposition in FAD is heuristic rather than rigorously spectral; future work may investigate explicit Fourier or wavelet-based separation.
• The model does not enforce physics-based ground contact or interaction priors, reducing plausibility for lower-body actions.
• Finger and facial expression modeling is omitted; extending to SMPL-X would broaden applicability.
• Dynamic transitions in SSD blocks are stationary; adaptive or nonlinear formulations could enhance representation of highly non-stationary motion, e.g., athletics and dance.

A plausible implication is that improvements in these areas will further advance avatar realism and physical fidelity in AR/VR pipelines.

8. Conclusion

SSD-Poser exemplifies the integration of state-space duality, attention mechanisms, and frequency-aware decoding for real-time full-body pose reconstruction from minimal HMD signals. Its architecture achieves high accuracy and perceptual smoothness, maintaining low computational footprint and high inference throughput. These features establish SSD-Poser as an efficient backbone for AR/VR avatar tracking, with open avenues for physics-aware and richer avatar extensions (Zhao et al., 25 Apr 2025).