DrawMotion: Generating 3D Human Motions by Freehand Drawing

Abstract: Text-to-motion generation, which translates textual descriptions into human motions, faces the challenge that users often struggle to precisely convey their intended motions through text alone. To address this issue, this paper introduces DrawMotion, an efficient diffusion-based framework designed for multi-condition scenarios. DrawMotion generates motions based on both a conventional text condition and a novel hand-drawing condition, which provide semantic and spatial control over the generated motions, respectively. Specifically, we tackle the fine-grained motion generation task from three perspectives: 1) freehand drawing condition. To accurately capture users' intended motions without requiring tedious textual input, we develop an algorithm to automatically generate hand-drawn stickman sketches across different dataset formats; 2) multi-condition fusion. We propose a Multi-Condition Module (MCM) that is integrated into the diffusion process, enabling the model to exploit all possible condition combinations while reducing computational complexity compared to conventional approaches; and 3) training-free guidance. Notably, the MCM in DrawMotion ensures that its intermediate features lie in a continuous space, allowing classifier-guidance gradients to update the features and thereby aligning the generated motions with user intentions while preserving fidelity. Quantitative experiments and user studies demonstrate that the freehand drawing approach reduces user time by approximately 46.7% when generating motions aligned with their imagination. The code, demos, and relevant data are publicly available at https://github.com/InvertedForest/DrawMotion.

• The paper introduces a multi-condition framework combining freehand drawing, trajectory mapping, and text guidance for 3D human motion generation.
• It utilizes a denoising diffusion model with a novel Multi-Condition Module (MCM) to fuse heterogeneous inputs, yielding state-of-the-art spatial and semantic control.
• The approach features training-free intermediate feature guidance for precise trajectory alignment and significantly lower trajectory error compared to previous methods.

DrawMotion: Freehand-Controlled 3D Human Motion Generation via Multi-Condition Diffusion

Introduction

DrawMotion (2605.20955) introduces a multi-condition framework for conditional 3D human motion generation that leverages both textual descriptions and user freehand drawings, enabling semantic and fine-grained spatial control over human motion synthesis. Unlike standard text-to-motion systems that are bottlenecked by text ambiguity and insufficient control granularity, DrawMotion integrates: (1) automatic synthesis of hand-drawn stickman sketches and trajectories, (2) an efficient and expressive Multi-Condition Module (MCM) for fusion of heterogeneous conditions, and (3) training-free spatial guidance that exploits the MCM's intermediate continuous latent space. The paper delineates novel algorithmic contributions and demonstrates state-of-the-art empirical results for controllable motion generation, establishing new practical workflows for both technical and artistic users.

Multi-Condition Input and Data Representation

DrawMotion permits joint semantic and spatial control by ingesting both natural language and hand-drawn guidance. Users provide:

• A 2D trajectory specifying the pelvis path through sequential coordinates, preprocessed via resampling and direct mapping to 3D ground truth coordinates. This trajectory preserves both the motion intent and kinetic cues (i.e., inertia) consistent with hand-drawing and real motion.
• Hand-drawn stickman sketches (any number, at arbitrary temporal positions along the trajectory), representing explicit joint placement and local geometry.
• Optional natural language descriptions for global semantic constraints.

To overcome the lack of annotated freehand stickmen in existing datasets, the authors propose the Stickman Generation Algorithm (SGA), which projects 3D skeleton data onto a canonical 2D view, then simulates realistic sketching artifacts (stroke smoothness, misplacement, body part rescaling) to synthesize representative stickman sketches.

Figure 1: Stickmen auto-generated from motion datasets using SGA, emulating typical human drawing variability and input inaccuracies.

The system employs a compact stickman encoding: each of the six major body parts is encoded as a polyline, transformed into sequence embeddings via a transformer encoder. During model training, a self-supervised autoencoder predicts multiple candidate poses from each stickman to accommodate limb ambiguity and intra-class variance.

Diffusion-Based Motion Generation with Novel Condition Fusion

DrawMotion is built upon the denoising diffusion probabilistic model (DDPM) as the base generative process, adopting DDIM acceleration for inference stability and efficiency. Crucially, the model structure departs from prior mask-based condition fusion by introducing the Multi-Condition Module (MCM):

• Encoders independently map motion, stickman, trajectory, and text into aligned latent spaces.
• The MCM fuses these using heterogeneous cross-attention: standard dot-product attention for locally detailed drawings/trajectories and efficient global attention for textual semantics. Dot-product attention propagates fine-grained spatial cues to individual frames, while efficient attention leverages semantic regularity and reduces computational complexity for text.
• During reverse diffusion, four possible condition combinations are considered at each denoising step. Mixture weights are dynamically adjusted such that early denoising emphasizes trajectory/text, with final refinement exclusively using all available conditioning signals.

  Figure 2: System overview: the diffusion framework is coupled with modular condition encoders and the MCM for spatial and semantic control.

Training-Free Intermediate Feature Guidance

A central contribution is the Training-Free Intermediate Feature Guidance (IFG), which directly addresses spatial misalignment caused by conflicts between text, stickman, and trajectory conditioning. Unlike methods that backpropagate spatial losses through the denoised output (leading to distributional collapse) or solely perturb the initial noise, IFG leverages the uniquely continuous and robust latent feature space formed by the MCM. The authors rigorously analyze and empirically validate:

• Intermediate features in mask-based/self-attention architectures rapidly "collapse" to lower-dimensional manifolds, precluding stable interpolation or semantic editing via gradients.
• The MCM, through cross-modality fusion and architectural design, induces a high-intrinsic-dimensional, continuous latent space. Empirical PCA projections and perturbation studies confirm that feature interpolation (even with large step sizes) does not catastrophically degrade sample quality, unlike conventional models.

  Figure 3: 2D PCA projections of ReMoDiffuse versus DrawMotion’s MCM intermediate features; DrawMotion exhibits substantially higher density and continuity, favoring latent space editing.

• IFG thus performs constrained SGD updates directly in the MCM’s intermediate feature space, minimizing a masked trajectory alignment loss, with Mahalanobis distance-based clipping to restrict updates within the observed training distribution. This procedure is lightweight, training-free, and preserves sample fidelity with minimal bias or overfitting to spatial constraints.

Evaluation and Analysis

Extensive quantitative and user studies validate the approach:

• On HumanML3D and KIT-ML, DrawMotion matches or surpasses dedicated text-to-motion and motion editing frameworks in FID, R-Precision, MultiModality, and uniquely excels in stickman similarity (StiSim) and trajectory error (Traj.Err.), the latter by a significant margin (see Tables 1-2 in the paper).
• IFG yields trajectory errors on HumanML3D ($0.0062$) and KIT-ML ($0.032$) considerably lower than both training-free and training-based prior motion editing methods.
• The framework maintains competitive generation diversity and global semantic fidelity, with specific ablation confirming:
  • Dot-product attention for stickman/trajectory fusion is optimal; efficient attention is best for textual semantics.
  • Increasing stickman sketch granularity (up to 7 per sequence) further refines motion control.
  • MCM dramatically reduces redundancy and computational overhead relative to mask-based condition mixing.

DrawMotion outperforms both StickMotion [wang2025stickmotion] and ReMoDiffuse [zhang2023remodiffuse] in strict user-alignment at much lower user annotation cost/time. User studies further demonstrate that DrawMotion reduces overall human labor, approaching (and in some cases exceeding) the subjective quality of manual animation.

Figure 4: Visual comparison for complex spatial/semantic tasks: DrawMotion achieves strict trajectory adherence and localized pose matching, exceeding StickMotion and ReMoDiffuse.

Limitations and Future Directions

The authors explicitly note that DrawMotion’s flexibility in spatial/pose control introduces user responsibility: physically inconsistent hand-drawn trajectories or conflicting stickmen/text can force the diffusion model to compromise either fidelity or strict alignment. Returning a guidance loss scalar to the user is proposed as a practical feedback channel for tuning input constraints.

Potential future research directions include: (1) end-to-end joint physical/semantic constraint modeling (including differentiable physical priors), (2) dynamic weighting of conflicting conditions via user-in-the-loop optimization, (3) extension to multi-agent/human-object scenarios, and (4) further reduction of inference latency for real-time authoring applications.

Conclusion

DrawMotion (2605.20955) advances the state of controllable 3D human motion generation by introducing a highly flexible multi-condition diffusion architecture, automatic and realistic stickman data synthesis, and robust, training-free latent space control for trajectory alignment. The work demonstrates how cross-modality latent integration and principled gradient editing substantially expand the utility and usability of text-based motion synthesis in creative, virtual reality, and applied graphics contexts.

Figure 5: DrawMotion qualitative visualization: generated 3D motion sequences match both freehand trajectory and local pose sketches over diverse user-specified conditions.