Text-Conditioned Diffusion Motion Generation

• Text-conditioned diffusion models are generative frameworks that use an iterative denoising process guided by natural language to synthesize diverse 3D human motions.
• They integrate transformer and U-Net architectures with mechanisms like cross-attention and classifier-free guidance to enable keyframe editing, streaming synthesis, and precise motion control.
• Empirical evaluations demonstrate state-of-the-art improvements in fidelity, R-Precision, and smooth transitions, highlighting their effectiveness in both isolated and streaming motion generation.

A text-conditioned diffusion-based motion generation model is a generative framework that synthesizes motion trajectories—in particular, 3D human motions—by iteratively denoising a random initial state, with each denoising step directly informed by natural language prompts. These models, which dominate recent research in motion synthesis, exploit the stochastic sampling process of the diffusion paradigm to offer state-of-the-art fidelity, diversity, and fine-grained alignment between text and generated motion. Modern variants also support long-term streaming generation, multi-action composition, keyframe editing, and bidirectional tasks such as captioning. This article surveys the technical landscape, foundational architectures, conditioning and control mechanisms, and representative advances in the domain, referencing key developments including MDM (Tevet et al., 2022), MotionDiffuse (Zhang et al., 2022), DART (Zhao et al., 7 Oct 2024), MotionStreamer (Xiao et al., 19 Mar 2025), and FloodDiffusion (Cai et al., 3 Dec 2025).

1. Mathematical Foundations of Diffusion-Based Motion Generation

At its core, a text-conditioned diffusion-based motion generation model adopts a Markovian latent-variable framework. The generation process begins by transforming a structured human motion $x_0\in\mathbb{R}^{T\times D}$—where $T$ is the sequence length and $D$ the per-frame DoF—into a sequence that is gradually corrupted by noise via a forward diffusion process:

$$q(x_t|x_{t-1}) = \mathcal{N}\big(x_t; \sqrt{1-\beta_t} x_{t-1}, \beta_t I\big), \quad t=1,\ldots,T_\mathrm{diff}$$

with $$q(x_{1:T_\mathrm{diff}}|x_0) = \prod_{t=1}^{T_\mathrm{diff}} q(x_t|x_{t-1})$$. The model then parameterizes a reverse process, $p_\theta(x_{t-1}|x_t, c)$, conditioned on the text embedding $c$, learned via denoising score matching (Zhang et al., 2022, Tevet et al., 2022). The network is trained to predict either the original sample (“$x_0$-prediction”) or the additive noise (“$\epsilon$-prediction”), with the empirical advantage of the $x_0$ objective established in MDM (Tevet et al., 2022).

In both discrete (Chi et al., 19 Jul 2024) and continuous (Zhang et al., 2022, Tevet et al., 2022, Zhao et al., 7 Oct 2024, Xiao et al., 19 Mar 2025) frameworks, the loss function typically reduces to

$$\mathcal{L} = \mathbb{E}_{x_0, t, \epsilon} \bigl\| \epsilon - \epsilon_\theta(x_t, t, c) \bigr\|^2$$

for the continuous Gaussian process, or to a cross-entropy of token predictions for vector-quantized discrete diffusion (Chi et al., 19 Jul 2024). Reverse sampling is performed iteratively, optionally employing classifier-free guidance, which linearly combines the unconditional and conditional denoiser outputs to trade off fidelity and diversity (Zhang et al., 2022, Tevet et al., 2022, Xiao et al., 19 Mar 2025).

2. Model Architectures and Text Conditioning Mechanisms

Backbone Architectures

Early models such as MotionDiffuse (Zhang et al., 2022) and MDM (Tevet et al., 2022) popularized transformer-based denoisers with per-frame input tokens and global text conditioning, often using pretrained language representations (BERT or CLIP). The transformer is typically augmented with positional embeddings (timesteps, sequence indices) and, in later work, interaction blocks to process text-motion fusion (Wu et al., 29 Nov 2024). U-Net architectures with down/up-sampling and 1D-convolutions remain prevalent for capturing temporal structure (Azadi et al., 2023, Cohan et al., 17 May 2024).

Discrete-latent approaches apply VQ-VAE (Chi et al., 19 Jul 2024) or causal temporal autoencoders (Xiao et al., 19 Mar 2025), with diffusion acting over codebook indices or continuous causal latents, respectively.

Text Embedding and Multi-level Conditioning

The dominant approach for text embedding is to exploit CLIP, using either the global or token-level features and injecting these via cross-attention, AdaLayerNorm, or multimodal fusion blocks (Zhang et al., 2022, Wang et al., 2023, Wu et al., 29 Nov 2024, Chi et al., 19 Jul 2024). Multi-level fusion architectures such as the LSAM–CAPR pipeline (Wang et al., 2023), temporal injection schemes (Wan et al., 2023, Zhao et al., 7 Oct 2024), and bidirectional text-frame alignments (Cai et al., 3 Dec 2025) are used to ensure allocation of semantic granularity.

To further increase control, models implement classifier-free guidance (randomly drop the condition in training, interpolate both branches in inference), which is critical for obtaining high-fidelity conditional samples (Tevet et al., 2022, Xiao et al., 19 Mar 2025, Wu et al., 29 Nov 2024, Zhang et al., 2022).

3. Extension to Hierarchical, Streaming, and Multi-Segment Generation

Long-term and Streaming Generation

Canonical diffusion models are length-constrained, but recent research has targeted variable-length and streaming synthesis. Autoregressive factorization—with overlapping short motion primitives (Zhao et al., 7 Oct 2024), continuous causal latent spaces (Xiao et al., 19 Mar 2025), or lower-triangular time scheduling (“diffusion forcing”) (Cai et al., 3 Dec 2025)—enables real-time, responsive synthesis with negligible latency growth. FloodDiffusion (Cai et al., 3 Dec 2025) demonstrates that tailored bidirectional attention, noncausal token-wise noise scheduling, and per-frame text injection are essential for matching ground-truth FID in streaming scenarios.

Multi-action and Transition-Smooth Generation

Discrete diffusion frameworks such as M2D2M (Chi et al., 19 Jul 2024) address the generation of contiguous multi-action sequences. The Two-Phase Sampling (TPS) strategy—joint coarse sampling then per-segment refinement—yields both per-segment fidelity and smooth transitions, quantifiably reducing acceleration “jerk” at segment boundaries. DiffusionPhase (Wan et al., 2023) encodes periodicity in the frequency domain, enabling non-degrading long-range composition.

Coherent Sampling and Compositionality

Sampling modifications—such as “past inpainting” and “compositional transition” (Yang et al., 2023)—allow for seamless blending of sequentially described actions. Multi-segment FID, transition-window diversity, and physical plausibility are all tractably improved via these approaches.

4. Fine-Grained Control: Keyframes, Partial Constraints, and Body-Part Editing

Hybrid conditioning, incorporating both textual prompts and explicit spatial constraints, enables precise user control over motion trajectories.

• Keyframe Collaboration: Models like DiffKFC (Wei et al., 2023) and CondMDI (Cohan et al., 17 May 2024) support dual-level guidance by fusing sparse keyframes and text. Masked attention blocks (DMA), imputation throughout the diffusion chain, and DCT-based smoothness priors yield low keyframe error (0.041 at 5% keyframes on HumanML3D) and FID near unconstrained baselines.
• Fine-Grained Linguistic Control: Fg-T2M (Wang et al., 2023) applies GAT-enhanced text encodings and progressive text injection, outperforming prior models on fine-grained splits (R-P@3=0.763 for “Harder-HumanML3D”).
• Body-Part and Temporal Manipulation: MotionDiffuse (Zhang et al., 2022) provides part-specific control via latent mask-based synthesis, and supports time-varied control through interval composition and interpolation.
• Inpainting and Editing: MDM (Tevet et al., 2022) and descendants adapt inpainting procedures for editing joint or interval constraints mid-sampling.

5. Evaluation Protocols and Empirical Findings

Mainstream protocols employ:

• FID (Fréchet Inception Distance): Assesses the statistical match to real motion, measured in a learned motion-feature space.
• R-Precision: Top-$k$ retrieval precision based on a pretrained motion-language embedding.
• Diversity, Multimodality: Mean pairwise sample distance, within and across prompts.
• Transition Jerk, AUJ: Quantify smoothness at concatenation boundaries (Chi et al., 19 Jul 2024, Xiao et al., 19 Mar 2025).
• Keyframe and control errors: L2 deviation from supplied constraints (Wei et al., 2023, Cohan et al., 17 May 2024).

Benchmark results establish that state-of-the-art diffusion-based models can reach FID $\sim 0.054$ (FloodDiffusion (Cai et al., 3 Dec 2025)), R-Precision@3 $>0.8$, with transition smoothness and control precision comparable to real datasets. Robustness to real-time, streaming, and compositional input is maintained when appropriate architectural constraints (e.g., causal masking, dynamic schedules, bidirectional context) are enforced.

6. Practical Extensions and Research Directions

Cutting-edge work is exploring:

• Streaming and real-time synthesis: FloodDiffusion (Cai et al., 3 Dec 2025), DART (Zhao et al., 7 Oct 2024), and MotionStreamer (Xiao et al., 19 Mar 2025) show that autoregressive and diffusion-forcing frameworks can handle interactive or time-varying linguistic streams at sub-50ms latency.
• Discrete-to-continuous hybridization: Discrete token models (VQ-VAE-based) with proximity-aware schedules (Chi et al., 19 Jul 2024) allow for flexible, non-autoregressive multi-segment construction.
• Latent consistency distillation: MLCT (Hu et al., 5 May 2024) leverages consistency training in quantized latent space for few-step inference with near-matching FID/R-Precision, enabling efficient deployment.
• Cross-modal and bidirectional modeling: MoTe (Wu et al., 29 Nov 2024) achieves text-to-motion, captioning, and pseudo-paired joint training within a unified diffusion model architecture, exploiting in-context, cross-attention, and AdaLN interaction mechanisms (R-Precision Top-1 of 0.548 for generation; 0.577 for motion-to-text).
• Mask-based denoising: MMDM (Chen, 29 Sep 2024) shows that masking in time and over body parts enhances the learning of spatio-temporal semantic relations and reduces FID to 0.276.
• Frequency-domain representations: DiffusionPhase (Wan et al., 2023) demonstrates preservation of high-frequency periodic structure and length-robust generation via compact phase encoding.

Research remains active in calibrating model structure against resource needs, scaling to more complex compositional prompts, and closing the gap to real human performance in perceptual studies.

7. Representative Model Feature Table

Model	Text Encoding	Motion Representation	Conditioning	Control Mode	Streaming/Long-Term	FID (HumanML3D)
MDM (Tevet et al., 2022)	CLIP	$\mathbb{R}^{T\times D}$	Token + guidance	Inpainting	No	0.544
MotionDiffuse (Zhang et al., 2022)	CLIP	$\mathbb{R}^{T\times D}$	Cross-attn Transformer	Bodypart, Time-interval	Partial	0.630
DART (Zhao et al., 7 Oct 2024)	CLIP	VAE latent	History + text	RL & Opt, AR	Yes	3.79 (seg-FID)
MotionStreamer (Xiao et al., 19 Mar 2025)	T5, causal TAE	Causal latent	Transformer AR	Streaming	Yes	10.724
FloodDiffusion (Cai et al., 3 Dec 2025)	T5	Latent	Diffusion Forcing	Per-frame, streaming	Yes	0.057
DiffKFC (Wei et al., 2023)	CLIP	$\mathbb{R}^{T\times D}$	Text + Keyframes	DMA, DCT smoothing	Partial	0.111
M2D2M (Chi et al., 19 Jul 2024)	CLIP	VQ-VAE discrete	Cross-attn Transformer	Multi-motion, TPS	No	0.087

The above summarizes the landscape and empirical achievements of contemporary diffusion-based text-conditioned motion generation models, with technical depth conformant to research practice.