Part-Aware Masked Autoregression (RoPAR)

• The paper introduces a novel pipeline that decomposes human motion into credible and noisy parts to robustly address occlusions.
• The model employs part-aware variational autoencoding, masked autoregression, and diffusion refinement for high-fidelity, text-conditioned motion synthesis.
• Quantitative ablations confirm that part-level decomposition and diffusion significantly improve motion quality and sample diversity on benchmark tests.

The Part-Aware Masked Autoregression Model (RoPAR) is a motion generation architecture designed to robustly extract, represent, and synthesize human motion sequences from large-scale, noisy video data, particularly in settings where partial occlusion and incomplete observations of the human body are pervasive. RoPAR integrates part-level data credibility assessment, variational autoencoding with shared part representations, and a masked autoregressive sequence model augmented with diffusion post-refinement. The pipeline is engineered to selectively ignore noisy or occluded body parts—marked by low per-part pose confidences—while jointly modeling inter-part dependencies and achieving high-fidelity text-conditioned motion synthesis (Li et al., 14 Dec 2025).

1. Architectural Framework

The RoPAR pipeline is partitioned into three principal stages: part-level decomposition and credibility analysis, part-aware variational autoencoding, and robust masked autoregression augmented with diffusion refinement.

• Decomposition & Credibility Detection: The skeleton is divided into five kinematic parts—torso, left/right arms, left/right legs. ViTPose is applied per frame to obtain $C_j\in[0,1]$ confidence scores for each joint. Average part confidence is computed as $C_{p} = \frac{1}{|J_p|}\sum_{j\in J_p} C_j$. If $C_p > \tau$ (e.g., $\tau=0.6$), the part is considered “credible”; otherwise, “noisy”.
• Part-Aware VAE (P-VAE): Only credible parts are encoded at each frame, yielding a matrix $Z\in\mathbb{R}^{P\times N\times d}$ of latent tokens. The VAE employs a shared MLP encoder and decoder for all parts, driving a unified latent space.
• Robust Masked Autoregression & Diffusion: All noisy tokens are permanently masked. Random masking is applied to credible tokens to achieve a target mask ratio $\alpha$. A masked Transformer autoregressively predicts masked latents conditioned on text, followed by a lightweight diffusion network that further refines predictions before VAE decoding.

The data flow, abstracted as a text-based diagram, is as follows:

input video frames + T │ 1) 2D pose → joint confidences ↓ part decomposition: torso, L/R arms, L/R legs │ 2) credible vs. noisy (Cₚ > τ?) ↓ P-VAE encoder (only credible parts) │ latent tokens Z (P×N) ↓ masked Transformer + diffusion head │ reconstructed Ẑ ↓ P-VAE decoder │ generated full-body motion sequence m̂

This staged pipeline enables selective encoding and robust generative modeling in the presence of prevalent missing data (Li et al., 14 Dec 2025).

2. Part-Aware Variational Autoencoder

Encoder/Decoder Schema

Each part-frame feature vector $m_i^p\in\mathbb{R}^f$ aggregates root linear velocity ($r^x$, $r^z$), root angular velocity ($r^a$), joint positions ($j^p$), velocities ($j^v$), and rotations ($j^r$). All parts are processed by a shared two-layer MLP:

• Encoder: $$m_i^p\xrightarrow[]{\text{MLP}}(\mu_i^p,\log\sigma_i^p)$$, producing $$q_\phi(z_i^p|m_i^p)=\mathcal{N}(z_i^p; \mu_i^p, \operatorname{diag}(e^{2 \log \sigma_i^p}))$$.
• Decoder: $z_i^p\xrightarrow[]{\text{MLP}}\hat{m}_i^p$ reconstructs the input.

Latent Collection and Objective

For each credible part, latent tokens $z_i^p = E_{\text{P-VAE}}(m_i^p)$ are stacked to form $Z$. The VAE’s evidence lower bound, aggregated only over credible parts $P_c$, is:

$$\mathcal{L}_{\text{P-VAE}} = \sum_{p\in P_c}\sum_{i=1}^N \left[ \mathbb{E}_{q_\phi(z_i^p|m_i^p)}\big[-\log p_\theta(m_i^p|z_i^p)\big] + \lambda D_{\text{KL}}(q_\phi(z_i^p|m_i^p)\|\mathcal{N}(0,I)) \right]$$

The reconstruction term is an $\ell_2$ loss, with $\lambda$ adjusting KL regularization strength.

This module constructs a robust, denoised latent representation, crucial for downstream generative modeling in the presence of partially observed data (Li et al., 14 Dec 2025).

3. Masked Autoregressive Generation and Diffusion Refinement

Masking Procedure

Given $Z_{noisy} = \{z_i^p : C_i^p<\tau\}$, the mask probability is:

$$P(z \to [M]) = \begin{cases} 1, & z \in Z_{noisy} \ \alpha - \beta_{\text{seq}}, & z \notin Z_{noisy} \end{cases}$$

where $\beta_{\text{seq}}=|Z_{noisy}|/|Z|$. All noisy tokens are masked, and additional masking is distributed randomly over credible tokens to set the total mask ratio.

Autoregressive Modeling and Training Loss

The masked input $\widetilde{Z} = m \odot Z + (1-m)\odot[M]$ is processed by a causal Transformer. The autoregressive factorization is:

$$p_\psi(Z | m, T) = \prod_{(p,i):m_i^p=0} p_\psi(z_i^p | \{z_{<i}^{<p}\}, m, T)$$

Loss is computed as:

$$\mathcal{L}_{\text{AR}} = -\mathbb{E}_{Z,m,T}\left[ \sum_{p=1}^P \sum_{i=1}^N (1-m_i^p) \log p_\psi(z_i^p | \widetilde{Z},T) \right]$$

In practice, $p_\psi$ predicts the Gaussian mean for each $z_i^p$; negative log likelihood reduces to $\ell_2$ loss.

Diffusion Refinement

Given autoregressively predicted $\hat{Z}$, the diffusion process applies:

$$x_t = \sqrt{\bar{\alpha}_t} \hat{Z} + \sqrt{1-\bar{\alpha}_t}\epsilon,\quad \epsilon\sim\mathcal{N}(0,I)$$

The denoising loss is:

$$\mathcal{L}_{\text{Diff}} = \mathbb{E}_{\hat{Z},\epsilon,t} \|\epsilon - \epsilon_\theta(x_t|t,\hat{Z})\|^2$$

Diffusion improves sample diversity and corrects residual artifacts from the causal decoding process (Li et al., 14 Dec 2025).

4. Training Regime and Inference

Joint Training Protocol

1. Pretrain P-VAE: Train using only credible part frames for $E_1$ epochs, minimizing $\mathcal{L}_{\text{P-VAE}}$.
2. Autoregressive & Diffusion Training: Freeze or lightly fine-tune the encoder. For $E_2$ epochs, encode full sequences, compute masking per Section 3, run the masked Transformer and diffusion head, and optimize $$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{AR}} + \gamma\mathcal{L}_{\text{Diff}}$$.
3. Mask Ratio Scheduling: $\alpha$ can be linearly increased as a curriculum but is effective even when fixed.

Inference Workflow

• Initialize $Z$ as completely masked and supply the text prompt.
• Iteratively autoregressively fill $z_i^p$ positions in raster order.
• Diffusion steps optionally refine $\hat{Z}$.
• Decode tokens with the P-VAE decoder to assemble the motion sequence.

This regime enables RoPAR to generalize to arbitrary missing data at test time, generating plausible and consistent full-body motions from text alone (Li et al., 14 Dec 2025).

5. Handling Incomplete Data and Quantitative Analysis

At test time, RoPAR receives fully masked input tokens and produces full-body predictions by sequentially infilling all parts; diffusion further reduces artifacts. Key ablation results on the K700-M benchmark are summarized below:

Modification	FID	MPJPE	R@1
Full RoPAR	0.21	6.95	0.71
– No part-wise decomposition in P-VAE	1.86	19.83	—
– No shared weights in P-VAE	0.89	9.82	—
– No part-aware decomposition in RoPAR	21.36	—	0.58
– No diffusion head	71.92	—	0.41

Ablations confirm that part-level credible/noisy splitting prevents latent space corruption by occluded parts, masked autoregression robustly models even with high missing data ratios (up to 70%), and the diffusion head is decisive for sample quality and diversity (Li et al., 14 Dec 2025).

6. Context and Significance

RoPAR enables large-scale, web-derived motion dataset utilization by explicitly modeling uncertainties due to occlusions and off-screen captures, a fundamental challenge in character animation. By rigorously separating credible from noisy part data, employing shared-space encoding, and leveraging masked autoregression and diffusion refinement, RoPAR produces high-fidelity, semantically controlled full-body motions. Its efficacy is further validated through superior benchmark performance and detailed ablations illustrating the indispensability of part-level decomposition and joint part-sequence modeling.

A plausible implication is that RoPAR’s methodology of part-aware masking and robust latent modeling can be generally extended to other domains where object-level occlusion or observation gaps are endemic, such as animal motion, multi-agent tracking, or robotics scenarios involving partial perceptions (Li et al., 14 Dec 2025).