ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation

Abstract: We present ScaleMoGen, a scale-wise autoregressive framework for text-driven human motion generation. Unlike conventional autoregressive approaches that rely on standard next-token prediction, ScaleMoGen frames motion generation as a coarse-to-fine process. We quantize 3D motions into compositional discrete tokens across multiple skeletal-emporal scales of increasing granularity, learning to generate motion by autoregressively predicting next-scale token maps. To maintain structural integrity, our motion tokenizers and quantizers are explicitly designed so that discrete tokens at every scale strictly preserve the skeletal hierarchy. Additionally, we employ bitwise quantization and prediction, which efficiently scale up the tokenizer vocabulary to preserve motion details and stabilize optimization. Extensive experiments demonstrate that ScaleMoGen achieves state-of-the-art performance, establishing an FID of 0.030 (vs. 0.045 for MoMask) on HumanML3D and a CLIP Score of 0.693 (vs. 0.685 for MoMask++) on the SnapMoGen dataset. Furthermore, we demonstrate that our skeletal-temporal multi-scale representation naturally facilitates training-free, text-guided motion editing.

• The paper introduces a multi-scale, autoregressive framework that predicts skeletal-temporal token maps for precise text-driven human motion synthesis.
• It leverages residual quantization and transformer conditioning with RoPE2d encoding to ensure semantic alignment and efficient inference.
• Results show state-of-the-art FID (0.030) and CLIP scores, enabling interpretable, zero-shot motion editing with robust structural control.

ScaleMoGen: Autoregressive Next-Scale Prediction for Human Motion Generation

Introduction and Motivation

ScaleMoGen introduces a new framework for text-driven human motion synthesis by reframing the generation task as a multi-scale, hierarchical process instead of traditional linear temporal expansion. Conventional autoregressive and masked token approaches either rely on sequential framewise prediction or iteratively filling in missing motion tokens. These paradigms often underutilize the inherent multi-scale structure of human motions, leading to limited semantic interpretability and suboptimal alignment with textual prompts. ScaleMoGen leverages this structure by quantizing motion into compositional discrete tokens across progressively finer skeletal and temporal scales, enabling generation via autoregressive next-scale token prediction while preserving the human kinematic hierarchy.

Multi-Scale Skeletal-Temporal Quantization

ScaleMoGen's pipeline encodes a motion sequence into a skeletal-temporal latent grid, which undergoes residual quantization across several scales. At each scale, both temporal resolution and skeletal granularity increase, capturing information from global joint groupings to atomic individual joints.

Figure 1: ScaleMoGen skeletal-temporal multi-scale quantization pipeline, mapping input motion to structured hierarchical motion token maps via residual binary quantization and topology-aware scaling.

This approach yields a hierarchy of token maps, each preserving structural completeness and recursive refinement properties, ensuring semantic consistency across granularity levels. Bitwise quantization expands effective vocabulary size exponentially with latent dimension, allowing fine-grained disentanglement and highly scalable token representations. Through topology-aware upsampling and downsampling, information transfer between scales strictly follows the articulated skeleton hierarchy, maintaining physical realism.

Figure 2: Spatial downsampling of the full-body skeleton into coarser anatomical groups, culminating in a 7-joint atomic topology for stable structural encoding.

Quantized residuals are supervised to reconstruct the original motion, with entropy regularization preventing codebook collapse and enhancing token usage diversity.

Autoregressive Next-Scale Prediction and Transformer Conditioning

ScaleMoGen employs a transformer that autoregressively predicts the next-scale token map conditioned on text embeddings (from T5-base) and accumulated coarser token maps. Motion generation starts at the coarsest scale and progresses through all hierarchy levels in a constant number of steps, making inference efficient. The transformer backbone integrates RoPE2d positional encoding and stochastic bit perturbation to enhance robustness and prevent error propagation across scales. Block-wise causal attention masking enforces hierarchical dependencies, ensuring each scale prediction relies strictly on its prefix.

Figure 3: Text-to-motion and motion editing pipeline: prediction proceeds scale-wise, with editing achieved by masking and blending source and target tokens at arbitrary skeletal-temporal regions.

Zero-Shot Text-Guided Motion Editing

The hierarchical token representation enables zero-shot, structured motion editing. Editing is accomplished by masking specific tokens—globally at coarse scales for structural preservation, or locally at fine scales for targeted articulation changes—and resampling them based on a new textual prompt. Semantic-aware masking allows confidence-based token replacement, ensuring meaningful changes while preserving unrelated behaviors.

Figure 4: Text-driven motion editing: sampled target tokens blend with preserved source tokens to synthesize semantically aligned and structurally consistent edits.

This mechanism is fundamentally training-free and provides high interpretability and control, as reflected in qualitative results and user studies.

Empirical Evaluation

Quantitative Results

ScaleMoGen achieves state-of-the-art FID and CLIP scores on HumanML3D and SnapMoGen benchmarks. Specifically, ScaleMoGen sets an FID of 0.030 (vs. 0.045 for MoMask) and a CLIP Score of 0.693 (vs. 0.685 for MoMask++) on SnapMoGen. These results indicate both superior motion fidelity and improved text-motion alignment, especially for complex, long-form prompts.

Qualitative and Ablation Analyses

Generated motions from descriptive prompts exhibit nuanced sub-action transitions, fine-grained body-part articulation, and precise temporal execution.

Figure 5: Qualitative text-to-motion generation: accurate synthesis of complex actions, fine body-part articulation, and temporally synchronized constraints across diverse prompts.

Ablation reveals that:

• Removing skeletal topology severely degrades performance across all metrics, confirming the necessity of structure-aware embedding.
• Code size influences the fidelity/alignment trade-off: smaller codes facilitate text-conditioned consistency but increase quantization error, while larger codes preserve details but challenge text consistency.
• Scaling the transformer model without dataset expansion leads to overfitting and reduced text-motion generalization, indicating that scaling laws from LLMs do not directly transfer without sufficient data volume.
• The bitwise multi-scale quantization design achieves the best MPJPE and FID for reconstruction, affirming the importance of combining structural hierarchy with scalable tokenization.

  Figure 6: Progressive disentanglement: intermediate token maps show global motion at coarse scales, resolving into independent joint movements and full realism as finer scales are accumulated.

Sampling Efficiency

The autoregressive multi-scale pipeline yields competitive inference speed: only seven generation steps needed, outperforming diffusion and masked models, with 0.071 seconds per sample.

User Study

Relative to diffusion baselines, ScaleMoGen scores highest in preservation, semantic alignment, and overall quality on motion editing tasks, validating its capacity for localized edits and natural synthesis.

Practical and Theoretical Implications

ScaleMoGen's explicit hierarchical representation refines controllability in text-to-motion generation, facilitating interpretable, training-free editing and efficient inference. The multi-scale paradigm is well-aligned with structural priors inherent in visual generation tasks and establishes groundwork for further cross-modal generative modeling. The method's robustness to fine-grained, compositional prompts opens new applications in animation, embodied AI, and interactive content authoring. However, scaling to Internet-level data will be necessary for full realization of scaling laws and further generality. The strong empirical results reinforce the value of discrete, structure-aware tokenization and demonstrate practical viability for real-time, structurally-precise motion generation and editing.

Conclusion

ScaleMoGen represents a principled advancement in text-driven motion synthesis, offering autoregressive next-scale prediction grounded in skeletal-temporal hierarchies. Through structured multi-scale tokenization, efficient conditioning, and robust editing, ScaleMoGen achieves superior motion fidelity, alignment with textual prompts, and highly interpretable motion editing. This framework is poised for extension to richer datasets, more complex interaction modeling, and unified multimodal generative systems (2605.11704).