Next-Scale Autoregressive Models for Text-to-Motion Generation

Abstract: Autoregressive (AR) models offer stable and efficient training, but standard next-token prediction is not well aligned with the temporal structure required for text-conditioned motion generation. We introduce MoScale, a next-scale AR framework that generates motion hierarchically from coarse to fine temporal resolutions. By providing global semantics at the coarsest scale and refining them progressively, MoScale establishes a causal hierarchy better suited for long-range motion structure. To improve robustness under limited text-motion data, we further incorporate cross-scale hierarchical refinement for improving per-scale initial predictions and in-scale temporal refinement for selective bidirectional re-prediction. MoScale achieves SOTA text-to-motion performance with high training efficiency, scales effectively with model size, and generalizes zero-shot to diverse motion generation and editing tasks.

• The paper introduces MoScale, a next-scale autoregressive framework that hierarchically refines motion tokens to achieve state-of-the-art text-to-motion generation.
• It leverages cross-scale and in-scale temporal refinements to correct prediction errors, ensuring both global semantic fidelity and local motion coherence.
• Empirical results on benchmarks like HumanML3D demonstrate superior text-motion alignment, efficiency, and zero-shot editing capabilities.

Hierarchical Next-Scale Autoregressive Modeling for Text-to-Motion Generation

Motivation and Context

Text-to-motion generation necessitates models capable of capturing global semantic structure expressed by natural language, such as repetition, ordering, and sequence-level intent, while also synthesizing locally coherent and physically plausible human motion. Standard causal next-token autoregressive (AR) models excel in natural language processing due to the strong dependency between sequential tokens, but human motion data exhibits highly redundant short-range dependencies that can bias AR models toward exploiting local continuity rather than learning the long-range global semantics specified in text. In contrast, diffusion and masked transformer methods leverage iterative refinement and bidirectional context; however, these approaches often generate locally plausible motion trajectories that lack semantic alignment with the input text, particularly under limited text–motion data. The "Next-Scale Autoregressive Models for Text-to-Motion Generation" (2604.03799) paper introduces MoScale, proposing a hierarchical next-scale AR framework that remedies both the causality mismatch and the short-horizon shortcuts inherent to standard AR modeling, producing state-of-the-art results in text-conditioned motion synthesis and motion editing.

MoScale: Architectural Innovations

MoScale hierarchically encodes motion data into multiscale discrete tokens using a residual VQ-VAE, representing the entire motion at progressively finer temporal resolutions. The core modeling distinction lies in generating motion not token-by-token, but scale-by-scale: at each coarser scale, the model commits to global structure, which is autoregressively refined at each finer scale, constituting a coarse-to-fine hierarchy. This is implemented using a causal transformer that, at each scale, predicts the full sequence of discrete tokens conditioned on the prefix, the accumulated embeddings from all coarser scales, and T5 text embeddings, enforcing strict hierarchical causality via scale-wise attention masking. Rotary position encodings and scale-level embeddings maintain relative temporal relationships across scales.

Figure 1: Overview of MoScale. (a) Multi-scale tokenization of motion. (b) Next-scale autoregressive generation across hierarchical temporal resolutions. (c) In-scale temporal refinement for token quality and consistency.

Two critical refinement mechanisms further differentiate MoScale:

• Cross-Scale Hierarchical Refinement perturbs inputs at each scale during training, exposing the model to corrupted coarser-scale tokenizations and teaching finer scales to correct both quantization and prediction errors, addressing exposure bias and error accumulation in the autoregressive hierarchy.
• In-Scale Temporal Refinement performs iterative, mask-and-repredict refinement within each scale, making use of available bidirectional context to selectively update uncertain tokens. This balances local coherence with the preservation of global causal structure.

This architecture generalizes naturally to zero-shot motion editing, inpainting, and outpainting: arbitrary tokens or regions can be masked at any (or all) scales and repredicted via the same iterative mechanism, enabling controlled modification without contaminating unedited regions.

Experimental Evaluation and Empirical Analysis

MoScale is evaluated on HumanML3D and KIT-ML, two canonical text-to-motion benchmarks. Metrics include Top-k retrieval (R-Precision, for text-motion correspondence), Frechet Inception Distance (FID), Diversity, Multimodal Distance, and Multimodality (motion variance per prompt).

Key results:

• On HumanML3D, MoScale (S=18) achieves Top-1 accuracy of 0.540 and FID of 0.046, outperforming both next-token AR models such as ParCo and masked transformer methods like MoMask++.
• On challenging high-complexity text descriptions, MoScale retains larger absolute and relative performance gains over all baselines, with Top-3 accuracy of 0.775 versus 0.762 for MoMask++.
• User studies show that MoScale is preferred for both text alignment (71.5% of votes) and motion quality (73.3%), corroborating the quantitative superiority.

MoScale scales efficiently with model size: as transformer depth increases, both FID and MM-Dist improve, with only diminishing returns for in-scale refinement iterations beyond 10 at the finest scale. Training and inference are faster than diffusion-based models and competitive with best-in-class AR baselines.

Figure 2: Comparison of Top-1 text alignment and training time for various model sizes on HumanML3D.

Ablation studies demonstrate the principal contribution to semantic fidelity arises from cross-scale hierarchical refinement, not solely from in-scale refinement. Pure hierarchical prediction (without any temporal refinement) already nearly matches the best masked transformer; adding in-scale refinement mainly boosts local coherence. The optimal corruption ratio for cross-scale input perturbation is moderate (γ=0.6).

Analysis using VLM-based fine-grained alignment scores shows that the hierarchical causal framework fundamentally advances semantic text-motion alignment beyond what bidirectional masked transformers or iterative denoising achieve.

Zero-Shot Generalization and Motion Editing

MoScale's mask-and-repredict mechanism directly supports editing, inpainting, and outpainting, both text-conditioned and unconditional. During editing, regions to modify are masked at all relevant scales while preserved areas remain fixed; reprediction then occurs only where modification is desired, preventing semantic drift or unnecessary changes.

Figure 3: Qualitative editing result—MoScale shows improved instruction adherence and precise preservation of unedited motion regions relative to existing methods.

User studies across 25 zero-shot inpainting/outpainting/editing scenarios combined with evaluation of motion quality and consistency consistently favor MoScale (preference rates above 78% across all criteria), highlighting generalization in data-limited regimes.

Implications and Future Directions

MoScale identifies hierarchical next-scale autoregression as a critical design axis for temporal generative modeling with strong textual alignment under limited supervision. This undermines the prevailing assumption that diffusion-based or masked transformer bidirectional modeling is strictly necessary for high alignment, instead highlighting the value of coarse-to-fine causal semantic scaffolds. Zero-shot editing and generalization capabilities position MoScale as a flexible backbone for general motion generation and editing tasks.

Theoretical implications extend toward structured generative modeling of sequences with complex multi-scale dependencies: rather than unidirectional next-token prediction or omnidirectional masked modeling, the results favor explicit, scale-aware hierarchies. Practically, the framework is directly extensible to other temporally and semantically structured spatiotemporal domains, such as video synthesis, audio-driven motion, and multi-agent simulation. Integration with multimodal conditioning (beyond text) and further scaling (with web-scale motion-text datasets) is an immediate and promising direction.

Conclusion

The MoScale framework demonstrates that next-scale, coarse-to-fine autoregressive modeling with integrated hierarchical and temporal refinement achieves state-of-the-art results for text-to-motion generation, maximizing both global semantic fidelity and local motion realism (2604.03799). This paradigm balances efficiency, training stability, and zero-shot flexibility, marking a pivotal step toward controllable, high-resolution motion generation systems with robust language alignment.