Pose-Guided Residual Refinement for Motion

• The paper introduces a dual-latent paradigm that decomposes motion into coarse pose codes and fine residuals, enabling interpretable edits and precise reconstructions.
• It employs hierarchical residual vector quantization to iteratively refine motion details, significantly reducing reconstruction errors compared to tokenization-only methods.
• Empirical results in text-to-motion and video synthesis tasks show marked improvements in fidelity, semantic alignment, and user-controllable outcomes over baseline approaches.

Pose-Guided Residual Refinement for Motion (PGR²M) is a family of computational methods that addresses the limitations of discrete pose-based motion representations by integrating interpretable pose codes with residual refinement through vector quantization or hierarchical estimation. Originally proposed to advance text-based 3D motion synthesis, video generation, and pose estimation, PGR²M methods unify structured control and high-fidelity motion detail across diverse domains by decomposing motion into coarse semantic structure and fine-grained residuals. This dual-latent paradigm supports both interpretable editing and high-quality reconstruction, demonstrating significant improvements over canonical tokenization-only and baseline approaches.

1. Core Principles and Architecture

PGR²M methods formalize the decomposition of complex motion into two distinct representational streams: a primary, interpretable pose (or structure) latent and an auxiliary residual term that refines this base with quantized or learned corrections. In text-to-motion frameworks, the motion sequence $M \in \mathbb{R}^{L \times D}$ is passed in parallel through:

1. Pose Parser ($P$): Identifies active $N$-way pose codes per frame, mapping via a codebook $C \in \mathbb{R}^{N \times D_c}$ to a downsampled pose latent sequence $Z \in \mathbb{R}^{L_d \times D_c}$.
2. Continuous Encoder ($E$): Encodes the input sequence into a continuous latent $H \in \mathbb{R}^{L_d \times D_c}$ at matching resolution.

The initial residual $r^{(0)} = H - Z$ encodes detail not explained by discrete pose codes. A hierarchical residual vector quantization (RVQ) module stacks $L$ quantizers, each learning to encode remaining information in $r^{(l-1)}$, forming quantized residuals $Q^{(l)}(r^{(l-1)})$ per stage. The final latent $F = Z + \sum_{l=1}^L Q^{(l)}(r^{(l-1)})$ is subsequently decoded to reconstruct $M$ at full resolution (Jeong et al., 20 Aug 2025, Jeong et al., 27 Dec 2025). Equivalent residual-refinement pipelines underpin image-to-video and pose-estimation variants, where refinement operates in pixel or pose space (Zhao et al., 2018, Wang et al., 2021).

2. Mathematical Framework for Residual Quantization and Refinement

Residual vector quantization in PGR²M is formulated as a levelwise error-correction process. For $x = H - Z$:

• At each stage $l$ ($l=1..L$):

$e^{(l)} = r^{(l-1)} - Q^{(l)}(r^{(l-1)})$

$r^{(l)} = Q^{(l)}(r^{(l-1)})$

• The quantizer $Q^{(l)}$ selects the nearest codeword in its stage-specific codebook.
• Full reconstruction is achieved as:

$\hat{x} = \sum_{l=1}^L Q^{(l)}(r^{(l-1)})$

These quantized residuals are added atop the pose latent, preserving pose interpretability while enabling detailed reconstruction (Jeong et al., 20 Aug 2025, Jeong et al., 27 Dec 2025).

In image-to-video and pose-estimation domains, residual refinement follows a similar structure: the system first synthesizes a coarse prediction (e.g., a future video frame or pose), then applies a learned or estimated residual (e.g., pixel-wise residuals or $\mathrm{SE}(3)$ pose corrections) to match the target (Zhao et al., 2018, Wang et al., 2021).

3. Model Components, Codebooks, and Learning Strategies

Codebook Configuration

• Pose Codebooks: Typically size $N=392$, dimension $D_c=512$, learned end-to-end with exponential moving average (EMA) updates upon encoder commitment.
• Residual Codebooks: Single or staged codebooks (commonly $M=64$–$512$ codewords) shared across RVQ layers for parameter efficiency, updated using VQ-VAE style EMA and reset.

Residual Dropout

To prevent dependence of the model on residuals (thus maintaining the semantic alignment and editability of the pose codes), stochastic masking (residual dropout) is employed. With probability $\tau$ (e.g., $\tau = 0.1$), only pose codes are used during training; otherwise, both pose and residual codes contribute to reconstruction. This constrains residuals to add only detail, not global semantics (Jeong et al., 27 Dec 2025).

Modular Prediction

• Base Transformer: Autoregressively predicts pose codes given text and keyword embeddings.
• Refine Transformer: Predicts residual codes conditioned on text, pose, quantization stage, and previous residual predictions.

This pipeline supports both generation and editing: global attributes via pose codes, and motion detail or micro-dynamics via residual codes (Jeong et al., 27 Dec 2025).

4. Training Objectives and Optimization

PGR²M models are optimized by a composite loss:

• Reconstruction Loss ($L_{recon}$): $\|\; M - \hat{M} \;\|_1$ to ensure fidelity.
• Velocity Consistency ($L_{vel}$): Penalizes mismatch in framewise velocities.
• Commitment Losses ($L_{commit}^{(l)}$, $L_{rvq}$): Encourage encoder-codebook binding with stop-gradient terms per RVQ stage.
• Entropy Regularization ($L_{ent}$): Balances codebook usage.
• Mask Sparsity and Adversarial Losses (in video): Practical in image-to-video, guiding the model to focus residual refinement on dynamically changing regions.
• Geometric and Photometric Consistency (in depth/pose): Ensures multi-scale structure alignment via reprojection loss and smoothness constraints.

Representative objective:

$$L_{final} = L_{recon} + \beta L_{vel} + \gamma \sum_l L_{commit}^{(l)}$$

with carefully tuned $\beta$ and $\gamma$ (Jeong et al., 20 Aug 2025, Jeong et al., 27 Dec 2025). Curriculum training schedules, where the tokenizer is trained to convergence before transformer modules, are common (Jeong et al., 27 Dec 2025).

5. Quantitative and Qualitative Outcomes

Reconstruction Fidelity and Controllability

On text-to-motion tasks (HumanML3D, KIT-ML), PGR²M demonstrates reduced FID (0.007 in rec. vs. 0.041 for the CoMo baseline), higher Top-1 R-Precision (0.488–0.510), and lower MM-Dist, confirming both fidelity and semantic alignment (Jeong et al., 27 Dec 2025, Jeong et al., 20 Aug 2025). Qualitative analyses reveal sharper velocity changes and more accurate high-frequency detail. Editing studies show that pose code edits alter only the intended global attribute, with residuals preserving nuanced temporal detail.

User Studies

In pairwise comparisons on motion editing, over 75% of responses preferred PGR²M over discrete-only baselines. User studies highlight enhanced naturalness, detail, and structural edit preservation (Jeong et al., 27 Dec 2025).

Efficacy in Video Synthesis and Depth/Pose Estimation

In image-to-video translation, PGR²M outperforms state-of-the-art methods for both facial expression retargeting (ACD-I ~0.18) and pose forecasting (PSNR, MSE), with ablation confirming the essential role of both dense skip connections and a dedicated residual refinement stage (Zhao et al., 2018).

In unsupervised depth and pose, 3D hierarchical refinement yields monocular depth prediction with AbsRel 0.109 (KITTI Eigen split), and visual odometry performance ($t_{rel}=3.37\%/2.76\%$) competitive with geometry-based SLAM systems, surpassing previous self-supervised baselines (Wang et al., 2021).

6. Interpretability, Limitations, and Extensions

PGR²M representations are explicitly designed for interpretability. t-SNE visualization of pose-code embeddings yields distinct clusters by semantic category; pairwise cosine similarity confirms that pose and residual codes are decorrelated, supporting disentangled control (Jeong et al., 20 Aug 2025, Jeong et al., 27 Dec 2025). Residual codes capture high-frequency dynamics, while pose codes encode jointwise attributes.

Known limitations include:

• Possible over-reliance on residuals without dropout.
• Fixed clip lengths and stages in some implementations.
• Focus on single-object/actor scenarios in initial variants.
• Limited explicit long-term memory for sequences.

Suggested extensions include dynamic temporal windows, multi-actor conditioning, and end-to-end training for tightly coupled optimization of all components (Jeong et al., 27 Dec 2025, Zhao et al., 2018).

---

In summary, PGR²M architectures, by fusing pose-guided global structure and residual vector quantization-based detail, set a new standard for controllable, interpretable, and high-fidelity motion modeling. Their conceptual and mathematical rigor enables both user-driven editing and realistic synthesis—a paradigm with applications across text-to-motion, image-to-video, and depth/pose estimation (Jeong et al., 20 Aug 2025, Jeong et al., 27 Dec 2025, Zhao et al., 2018, Wang et al., 2021).