Continuous Masked Autoregressive Motion Transformer

• Continuous Masked Autoregressive Motion Transformer is a generative model that uses continuous latent token representations and spatio-temporal masking to create complex motion and video sequences.
• It employs parallel masked decoding and advanced modules such as 3D-RoPE and CanvasViT to enhance prediction accuracy and sampling efficiency across temporal and spatial dimensions.
• Empirical results demonstrate state-of-the-art performance with reduced computational overhead, achieving breakthroughs in video frame and human motion generation tasks.

A continuous masked autoregressive motion transformer is a generative model architecture that leverages continuous-valued latent token representations, spatio-temporal masked autoregression, and transformer-based neural architectures to generate sequences of complex structured data such as human motion, video, or multimodal signals. The approach represents a significant convergence of continuous latent modeling, masked autoregressive prediction, and high-efficiency parallel generation. Various instantiations have emerged in recent years for both unconditional and conditional sequence generation in continuous domains such as video frames and human motion. The following sections provide a rigorous, comprehensive description of the fundamental principles, architectural design, algorithmic mechanisms, and empirical results for continuous masked autoregressive motion transformers, drawing from leading works including VideoMAR (Yu et al., 17 Jun 2025), CanvasMAR (Li et al., 15 Oct 2025), and OmniMotion (Li et al., 16 Oct 2025).

1. Continuous Latent Token Representation

The foundation of continuous masked autoregressive transformers is the representation of structured input data (e.g., videos or motion capture sequences) as sequences of continuous latent tokens. For video, a pretrained continuous autoencoder (usually a VAE) encodes the original pixel array $V \in \mathbb{R}^{T \times H \times W \times 3}$ into a structured tensor $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$, with each $x_{t,i} \in \mathbb{R}^d$ a continuous vector corresponding to a spatial patch at time $t$ (Yu et al., 17 Jun 2025, Li et al., 15 Oct 2025). In human motion or multimodal settings, a similar continuous embedding is produced by a motion autoencoder applied to pose, skeleton, or joint parameterizations (Li et al., 16 Oct 2025).

These continuous tokens serve as the prediction targets for the autoregressive model, in contrast to discrete latent codes (as used in VQ-VAE, MaskGIT, or discrete autoregressive schemes). During training, random masking and/or noise perturbation of a subset of tokens is performed. In VideoMAR, noise is added to selected token subsets via a diffusion-based corruption process

$$x_{t,i}^q = \alpha_q\,x_{t,i} + \sigma_q\,\varepsilon,\quad \varepsilon \sim \mathcal{N}(0,I),$$

where $q$ denotes the diffusion step (Yu et al., 17 Jun 2025). In OmniMotion, continuous tokens are randomly masked by a mask ratio $\gamma(\tau) = \cos\left(\frac{\pi\tau}{2}\right)$ for $\tau \sim \mathcal{U}(0,1)$, supporting flexible random masking and facilitating masked reconstruction (Li et al., 16 Oct 2025).

2. Spatio-Temporal Masked Autoregressive Transformer Architecture

All contemporary continuous MAR frameworks build on multi-layer transformer decoders, equipped with specialized causal attention masks and, in some models, additional guidance or normalization modules.

Core backbone:

• Stacked self-attention layers, hidden dimensions up to $D=1536$, and multi-head attention (e.g., $H=24$ in VideoMAR).
• Token normalization layers, with LayerNorm (VideoMAR, CanvasMAR) or RMSNorm (OmniMotion) (Yu et al., 17 Jun 2025, Li et al., 16 Oct 2025, Li et al., 15 Oct 2025).
• Feed-forward layers (e.g., two-layer MLP, inner width $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$0).
• Optional modules for text or multimodal conditioning (cross-attention on frozen language encoder embeddings) (Yu et al., 17 Jun 2025, Li et al., 16 Oct 2025).

Attention mask designs:

• VideoMAR and CanvasMAR apply a causal mask along the temporal axis (preventing access to future frames) and bidirectional masking within each frame (enabling all patches in a frame to attend to each other) (Yu et al., 17 Jun 2025, Li et al., 15 Oct 2025).
• OmniMotion employs a standard lower-triangular causal mask for autoregression along the temporal (motion) sequence (Li et al., 16 Oct 2025).

Auxiliary modules:

• VideoMAR employs 3D rotary positional embeddings (RoPE) applied over $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$1 positions to facilitate extrapolation in temporally and spatially varying resolutions (Yu et al., 17 Jun 2025).
• OmniMotion incorporates a gated linear attention mechanism and RMSNorm. The gate sharpens model focus on "key action" frames, supporting robust modeling of heterogeneous or multi-action signals (Li et al., 16 Oct 2025).
• CanvasMAR introduces a CanvasViT module for global, blurred prediction guidance (see Section 5).

Model	Tokenization	Masking (time/space)	Normalize	Extra Modules
VideoMAR	VAE	Temporal: casual<br>Spatial: bidi	LayerNorm	3D-RoPE, KV Cache, text cond.
CanvasMAR	Conv encoder	Similar to VideoMAR	LayerNorm	CanvasViT
OmniMotion	AE (ResNet)	Causal (temporal)	RMSNorm	Gated attention, AdaLN, DiT

3. Masked Autoregressive Generation and Training

The core generative process of a continuous MAR transformer is a framewise or sequencewise, masked-prediction procedure factorized by temporal and possibly spatial dimensions.

Framewise Masking and Factorization:

• For videos: at generation step $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$2, all tokens for frame $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$3 are masked. Previous frames ($$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$4) are fully visible, providing causal context. Generation proceeds iteratively over frames.
• Within a frame, all spatial tokens can be jointly modeled due to the bidirectional spatial mask.

Joint Generation Factorization:

$$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$5

where $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$6 are the masked tokens, $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$7 are visible (generated/conditioned) tokens, and $$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$8 is the (optional) conditioning context (Yu et al., 17 Jun 2025).

Losses:

• VideoMAR utilizes a next-frame diffusion loss, where only tokens of the target frame are noised and reconstructed via a diffusion denoising objective aligning with the autoregressive factorization:

$$X = \{x_{t, i}\mid t = 1,\dots,T;\, i = 1,\dots,H'W'\}$$9

with masking $x_{t,i} \in \mathbb{R}^d$0 sampled per frame (Yu et al., 17 Jun 2025).

• OmniMotion applies a masked autoregressive regression loss (L2) only to masked token positions, in addition to VAE reconstruction and diffusion losses:

$x_{t,i} \in \mathbb{R}^d$1

with classifier-free guidance available during training (Li et al., 16 Oct 2025).

4. Sampling, Canvas Mechanisms, and Progressive Strategies

Parallel Masked Decoding:

• Instead of strictly token-by-token decoding, masked autoregressive models sample all masked tokens in a frame (or a set of spatial positions) in parallel using a small number $x_{t,i} \in \mathbb{R}^d$2 of diffusion denoising steps, drastically reducing sampling budget compared to classic next-token prediction ($x_{t,i} \in \mathbb{R}^d$3) (Yu et al., 17 Jun 2025, Li et al., 15 Oct 2025).

Canvas Mechanism (CanvasMAR):

• A separate CanvasViT module generates a global spatial "canvas," a coarse first-guess of each target frame given past frames and an optionally noise-augmented previous frame. During autoregressive spatial sampling, canvas embeddings replace uniform mask tokens, providing stronger global guidance and mitigating slow start and accumulated errors. Canvas-ViT is trained with an L2 frame reconstruction loss and utilized via compositional guidance for further regularization (Li et al., 15 Oct 2025).

Compositional Classifier-Free Guidance:

• CanvasMAR employs compositional guidance where spatial (canvas) and temporal conditions are combined multiplicatively and up-weighted in the Bayes factorization of the likelihood. This separation improves both spatial and temporal structure in the resulting frames (Li et al., 15 Oct 2025).

Noise-Augmented Robustness:

• CanvasMAR and related works inject Gaussian noise into previous frames and/or canvas features during training to force robustness to spatio-temporal prediction errors and prevent error accumulation (Li et al., 15 Oct 2025).

Curriculum Learning and Progressive Scheduling:

• VideoMAR employs temporal short-to-long curriculum (increasing max frame-length phases) and spatial progressive resolution strategies (two-stage VAE compression and upscaling). At inference, temperature annealing suppresses exposure bias in later frames (Yu et al., 17 Jun 2025).

5. Multimodal and Conditional Extensions

OmniMotion generalizes the continuous MAR transformer to multimodal conditional motion generation. Key design aspects include:

• Adaptive LayerNorm (AdaLN): Normalization parameters of transformer layers are modulated by multimodal embeddings (text, speech, music) through small MLPs, enabling contextual adaptation (Li et al., 16 Oct 2025).
• Cross-Attention Injection: Multimodal features are injected into motion transformers either via additional attention heads (speech/music) or text cross-attention blocks (Li et al., 16 Oct 2025).
• Diffusion Transformer (DiT): A lightweight diffusion transformer further refines the masked autoregressive predictions for each token (Li et al., 16 Oct 2025).

This compositional approach enables state-of-the-art performance across text-to-motion, speech-to-gesture, and music-to-dance tasks.

6. Efficiency, Extrapolation, and Empirical Performance

Efficiency Innovations:

• All models exploit parallel masked prediction (as above) and key/value (KV) caching. Once a frame’s tokens are generated, their K/V tensors are reused in all subsequent attention computations, avoiding redundant recomputation and permitting efficient caching scaling as $x_{t,i} \in \mathbb{R}^d$4 per frame (Yu et al., 17 Jun 2025).

Extrapolation Mechanisms:

• 3D rotary embeddings (RoPE) enable continuous MAR transformers to generalize to new (unseen) spatio-temporal shapes or resolutions by encoding position as a three-dimensional rotation function attached to token embeddings (Yu et al., 17 Jun 2025). Because attention depends only on the difference of positions, the model smoothly extrapolates across wider temporal spans or spatial scales.

Empirical Results:

• On VBench-I2V, VideoMAR surpasses the parameter-heavy Cosmos I2V on total and I2V scores while using only $x_{t,i} \in \mathbb{R}^d$5 of parameters, $x_{t,i} \in \mathbb{R}^d$6 of training pairs, and $x_{t,i} \in \mathbb{R}^d$7 of the GPU resources. Inference is $x_{t,i} \in \mathbb{R}^d$8 faster than Cosmos (Yu et al., 17 Jun 2025).
• CanvasMAR attains FVD $x_{t,i} \in \mathbb{R}^d$9 (debiased $t$0) on BAIR with only $t$1–$t$2 AR steps per frame, outperforming previous AR baselines, and FVD $t$3 on Kinetics-600, matching diffusion-based state of the art (Li et al., 15 Oct 2025).
• OmniMotion achieves R-Precision Top-1 of $t$4 and FID $t$5 (HumanML3D), surpassing prior methods such as MotionCraft and FineMoGen, and narrows the gap in music-to-dance and speech-to-gesture settings (Li et al., 16 Oct 2025).

7. Limitations and Variations

Current limitations include potential error accumulation in long sequences (mitigated via canvas mechanisms and noise augmentation) and sensitivity to initialization or multimodal heterogeneity, addressed by architectural choices such as AdaLN and RMSNorm (Li et al., 16 Oct 2025, Li et al., 15 Oct 2025).

Comparisons with alternative discrete masked models and autoregressive baselines show that the continuous MAR approach achieves higher quality and coherence, with tractable scaling and substantial efficiency gains, particularly in high-resolution and long-sequence generation regimes (Yu et al., 17 Jun 2025, Li et al., 15 Oct 2025).

Summary Table: Core Features of Continuous Masked Autoregressive Transformers

Model	Token Type	Mask Strategy	Guidance/Extrapolation	Empirical Gains
VideoMAR	Continuous	Spatio-temporal mask	3D-RoPE, curriculum, KV-cache	SOTA I2V, efficiency
CanvasMAR	Continuous	Canvas + causal mask	Compositional guidance, noise	AR baseline, FVD
OmniMotion	Continuous	Causal (motion seq.)	AdaLN, gated attn, DiT	Multimodal SOTA