ML-RQ: Motion-Lifting Residual Quantized VAE

• The paper introduces ML-RQ, a novel VAE architecture that leverages multi-level residual quantization to lift 2D motion sequences to structured 3D outputs.
• The approach integrates convolutional encoders, residual quantizers, and specialized loss functions to achieve superior fidelity and lower FID scores compared to traditional models.
• Empirical results across systems like Free3D and T2M-HiFiGPT demonstrate ML-RQ’s scalability and effectiveness in both supervised and weakly supervised motion synthesis.

The Motion-Lifting Residual Quantized Variational Autoencoder (ML-RQ) is an architectural paradigm developed for lifting 2D pose or motion sequences to structured 3D motion, leveraging hierarchical residual vector quantization within a VAE framework. It combines convolutional encoders, multi-level residual quantizers, and projection/regularization losses to enable high-fidelity, data-efficient, and generalizable 3D motion generation from limited or weak supervision. ML-RQ modules are empirically validated across several state-of-the-art human motion synthesis systems, including Free3D, T2M-HiFiGPT, and Mogo, ably supporting a wide spectrum of training regimes from fully supervised 3D to projection-only 2D constraints.

1. Architectural Foundations and Modular Structure

The canonical ML-RQ comprises three primary submodules: an encoder $\mathcal{E}$, a multi-level residual quantizer $\mathcal{Q}$, and a decoder $\mathcal{D}$ tasked with sequence reconstruction or lifting. Input domains include single-view 2D keypoint streams (Liu et al., 14 Nov 2025) or temporal 3D pose sequences (Wang, 2023, Fu, 5 Dec 2024). Architecturally, both the encoder and decoder utilize deep stacks of residual 1D convolutions (typically three blocks), often with additional linear projections, normalization, and nonlinearity. The encoder yields a continuous latent $z_e \in \mathbb{R}^{T \times D}$. This embedding is discretized by the quantizer $\mathcal{Q}$, stacking $L$ codebooks of size $K$ each (e.g., $L=6$, $K=512$–$8192$ across implementations) and producing a sum of quantized vectors per frame. The decoder, mirroring the encoder's form, reconstructs output as either dense 3D pose streams or lifted human motion (Liu et al., 14 Nov 2025, Fu, 5 Dec 2024).

2. Hierarchical Residual Vector Quantization Mechanism

The core quantization mechanism performs multi-level nearest-neighbor searches through residual codebooks. For each layer $l$: $$r^1 = z_e, \qquad q^l = \arg\min_{c\in\mathcal{C}^l} \|r^l - c\|_2^2, \qquad r^{l+1} = r^l - q^l, \qquad z_q = \sum_{l=1}^L q^l$$ At each level, the quantizer encodes the current residual into the closest codeword, iteratively refining the latent. Discrete index matrices are recorded for efficient tokenization and subsequent autoregressive modeling. Each codebook $\mathcal{C}^l$ is independently optimized via an exponential moving average rule (Liu et al., 14 Nov 2025, Fu, 5 Dec 2024). The straight-through estimator is employed to ensure effective gradient flow during backpropagation, with stop-gradient operations applied to codeword embeddings in the commitment objective.

3. Loss Functions and Training Objectives

ML-RQ training optimizes a composite objective including:

• Reconstruction loss: Enforces fidelity between the input and reconstructed output, varying by supervision—smooth $L_1$ loss on 3D pose (Wang, 2023), $L_1$ loss for motion sequences (Fu, 5 Dec 2024), and projection-consistent Euclidean loss for 2D->3D lifting (Liu et al., 14 Nov 2025).
• Quantization and commitment loss: Encourages encoder outputs to remain close to assigned discrete codewords, via

$$\mathcal{L}_{\mathrm{VQ}} = \|\mathrm{stopgrad}(z_e) - e\|_2^2 + \beta \|z_e - \mathrm{stopgrad}(e)\|_2^2$$

with $\beta$ typically in the range $0.25$–$2$.

• Additional regularization: In lifting and weakly-supervised scenarios, additional terms include reprojection losses, 3D bone-length or orientation priors, multi-view or latent consistency, and optionally adversarial or kinematic validity objectives (Liu et al., 14 Nov 2025, Wang, 2023). For Free3D, a set of 3D-free regularizers enforce view-consistent projection, orientation coherence, and latent feature consistency.

The total ML-RQ loss is given as

$$\mathcal{L}_{\mathrm{MLRQ}} = \mathcal{L}_{\mathrm{reg}} + \beta \mathcal{L}_{\rm VQ}$$

where $\mathcal{L}_{\mathrm{reg}}$ may encapsulate complex domain- or data-driven constraints.

4. Application Domains and Supervision Regimes

ML-RQ modules are designed for flexible application in motion synthesis pipelines. In fully supervised 3D synthesis, ML-RQ directly reconstructs or codes 3D pose streams with high accuracy (Wang, 2023, Fu, 5 Dec 2024). In motion-lifting, the encoder accepts 2D input (e.g., keypoints or limb vectors), and the decoder is augmented with both 3D-pose and camera-projection heads, optionally embedding camera intrinsics or root-depth tokens for adaptability (Wang, 2023, Liu et al., 14 Nov 2025). Free3D demonstrates that ML-RQ can learn to lift 2D motion to 3D structure with no direct 3D supervision using only geometric and physical regularization (Liu et al., 14 Nov 2025).

ML-RQ also facilitates tokenization for downstream sequence models, such as hierarchical causal transformers and autoregressive networks for generative tasks (Fu, 5 Dec 2024, Wang, 2023). Residual token streams extracted by ML-RQ modules are used as discrete symbolic inputs for these models, enabling high-fidelity, temporally long, and cyclic generation.

5. Empirical Results, Ablations, and Comparative Performance

Empirical ablation studies across datasets (HumanML3D, KIT-ML) indicate that the residual quantization mechanism of ML-RQ confers substantial improvements over non-quantized or shallow-VQ architectures (Liu et al., 14 Nov 2025). For example, discrete VQ-VAE yields FID=0.054 versus FID=0.121 (Gaussian VAE) and FID=0.258 (autoencoder). In Free3D, removing view regularizers or orientation coherence substantially degrades performance (FID rises from 0.054 to >2) (Liu et al., 14 Nov 2025). Similarly, T2M-HiFiGPT observes that RVQ-VAE surpasses prior VQ-VAE models in both accuracy and computational demand for 3D motion (Wang, 2023). In Mogo, ML-RQ-based pipelines achieve FID=0.079 (HumanML3D test set), surpassing models such as T2M-GPT (FID=0.116), AttT2M (FID=0.112), and MMM (FID=0.080) (Fu, 5 Dec 2024). The residual VQ structure enhances sample diversity and multimodality, and ML-RQ token representations are favorable for both autoregressive and masked generative model regimes.

6. Implementation Hyperparameters and Training Strategies

The principal hyperparameters of ML-RQ include codebook size ($K=512$–$8192$), latent dimension ($D=128$–$512$), quantization depth ($L=6$), and convolutional stack depth (3–6 blocks) (Wang, 2023, Fu, 5 Dec 2024, Liu et al., 14 Nov 2025). Batch sizes and learning rates are chosen to ensure stability over long sequences (e.g., batch=256, learning rate $2\times10^{-4}$ for $\sim$100 epochs), with frequent use of warm-up and decay schedules. Stability enhancements include straight-through estimators for codeword lookup and EMA codebook updates. Two-stage training regimes, especially for 2D->3D lifting, freeze the encoder/codebook in a first stage before fitting downstream transformers or decoders.

7. Theoretical and Practical Implications

The ML-RQ module offers strong inductive bias for structure- and distribution-preserving motion synthesis. Its residual quantization effectively improves synthesizability and diversity, supporting both supervised and 3D-label-free learning (Liu et al., 14 Nov 2025). The ability to tokenize complex motion in a data-efficient manner enables integration with powerful generative transformers, facilitating long, cyclic, and semantically coherent motion generation (Fu, 5 Dec 2024, Wang, 2023). The architecture’s success in weak- or unsupervised domains suggests that discrete motion tokenization coupled with geometric priors can supplant full 3D supervision without loss of fidelity. A plausible implication is the broader viability of symbolic token sequence modeling for motion, supporting scalable and generalizable pipelines across motion domains.