DrawMotion: Diffusion-Based 3D Motion Generation

• DrawMotion is a diffusion-based framework for 3D human motion synthesis that integrates text for global semantic control with hand-drawn 2D trajectories for detailed spatial guidance.
• The system introduces a Stickman Generation Algorithm and a Multi-Condition Module to fuse freehand sketches with textual descriptions, enabling flexible 3D motion generation.
• Empirical evaluations on datasets like HumanML3D and KIT-ML demonstrate enhanced trajectory fidelity, improved user efficiency, and reduced motion specification ambiguity.

Searching arXiv for the primary paper and closely related work. arXiv search: "DrawMotion (Wang et al., 20 May 2026)" DrawMotion is a diffusion-based framework for generating 3D human motions from both text and freehand drawing, introduced to address the ambiguity of text-only text-to-motion generation. Its central interaction paradigm is that a user can draw a 2D trajectory and place stickman poses along it, so that text provides semantic control while drawing provides spatial and pose-level control. In the adjacent literature, “DrawMotion” also appears as a broader label for drawing-driven motion control, including drag-to-control video generation and sketch animation, but the named framework DrawMotion specifically denotes the 2026 system for multi-condition 3D human motion synthesis (Wang et al., 20 May 2026).

1. Conceptual definition and problem formulation

DrawMotion targets the fine-grained motion generation problem in settings where natural language alone does not reliably specify the intended motion. The motivating example given for the underlying ambiguity is that a description such as “a high kick forward” does not fully encode arm pose, body tilt, or trajectory. DrawMotion therefore introduces a hand-drawing condition alongside a conventional text condition, with the explicit goal of making motion specification more visual and less dependent on verbose text (Wang et al., 20 May 2026).

The hand-drawing condition has two components. The first is a 2D trajectory, sampled from a user’s drawing on a web interface as a coordinate sequence $J^t \in \mathbb{R}^{(n,2)}$, then resampled to $\widehat{J}^t \in \mathbb{R}^{(T,2)}$ to match motion length. The second is multiple stickman sketches placed at arbitrary positions along that trajectory. In the system description, the trajectory acts as the target pelvis path and provides global spatial control, while the stickmen provide local pose guidance at selected frames. This design differs from earlier stickman-conditioned systems by allowing multiple stickmen at arbitrary trajectory positions rather than fixing the number and timing of pose insertions (Wang et al., 20 May 2026).

A useful contextual comparison is StickMotion, which also used text and stickman conditions for 3D human motion generation, but restricted users to approximate temporal locations such as start, middle, and end (Wang et al., 5 Mar 2025). DrawMotion generalizes that design toward a more flexible freehand interface, while retaining the basic premise that text supplies global semantics and drawing supplies fine-grained structure (Wang et al., 20 May 2026).

2. Drawing representation and synthetic data construction

A central practical obstacle is that standard motion datasets do not contain hand-drawn stickmen. DrawMotion addresses this with a Stickman Generation Algorithm (SGA) that automatically synthesizes hand-drawn stickman sketches from 3D pose data. The algorithm models the imperfect character of real hand drawings through stroke smoothness, global misplacement, and size variation, and renders stickmen from a front-view projection so that the output resembles a human sketch rather than an arbitrary 3D projection (Wang et al., 20 May 2026).

For encoding, DrawMotion avoids pointwise processing of roughly 200 coordinates. Instead, each stickman is represented by six one-stroke lines corresponding to head, torso, and four limbs. These lines are encoded separately and aggregated with a transformer encoder. The paper further pretrains a stickman autoencoder with a feature-to-pose decoder, and introduces a candidate loss to address ambiguous left/right limb assignments when limbs overlap. The decoder predicts $N$ candidate poses with

$$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$

and the final loss is

$$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$

This formulation encourages multiple plausible pose hypotheses rather than a single brittle reconstruction (Wang et al., 20 May 2026).

The trajectory representation is likewise designed for flexibility. The user-drawn path can be resampled either uniformly or in a density-based manner, so the model can either ignore or preserve the hand’s drawing speed. This suggests that the framework treats drawing not merely as a symbolic hint but as a structured spatial condition whose temporal parametrization can be modulated at preprocessing time (Wang et al., 20 May 2026).

3. Diffusion formulation and multi-condition architecture

DrawMotion is built on a standard diffusion framework. The forward process is

$$q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}\!\left(\mathbf{x}_t;\sqrt{\alpha_t}\mathbf{x}_{t-1},(1-\alpha_t)\mathbf{I}\right),$$

with closed form

$$\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon_t,\quad \bar{\alpha}_t=\prod_{s=1}^{t}\alpha_s.$$

During training, the model minimizes

$$\mathbb{E}_{\epsilon_t,t,x_0}\left[\left\|\epsilon_t-\epsilon_\theta(\mathbf{x}_t,t,L,C(\text{draw}),C(\text{text}))\right\|^2\right].$$

The architecture uses four input encoders: a linear encoder for noisy motion, a 1D CNN for trajectories, CLIP ViT-B/32 for text, and a transformer encoder for stickmen (Wang et al., 20 May 2026).

The core architectural contribution is the Multi-Condition Module (MCM). It explicitly handles the four condition combinations required in classifier-free diffusion: $$(\text{text},\text{draw}),\quad (\text{text},\varnothing),\quad (\varnothing,\text{draw}),\quad (\varnothing,\varnothing).$$ Rather than using masked self-attention over all condition tokens, MCM partitions the batch into four segments, computes only the necessary condition decodings, and reuses the resulting features. The module contains two specialized decoders: a Draw Decoder based on standard dot-product attention, and a Text Decoder based on efficient attention. For the draw branch,

$$e^{kv}=\text{concat}((e^m\oplus e^j), e^s),\quad \mathbf{Q}=FCN_1(e^m),\quad \mathbf{K},\mathbf{V}=FCN_{2,3}(e^{kv}),$$

$\widehat{J}^t \in \mathbb{R}^{(T,2)}$0

For text,

$\widehat{J}^t \in \mathbb{R}^{(T,2)}$1

$\widehat{J}^t \in \mathbb{R}^{(T,2)}$2

The outputs are added to the motion features and re-encoded through a latent encoder, yielding a fusion mechanism reported to have better performance and lower FLOPs than masked self-attention baselines (Wang et al., 20 May 2026).

In comparative terms, this design extends the multi-condition logic already explored in StickMotion, but replaces the earlier start/middle/end control protocol with freehand trajectories and arbitrary stickman insertions (Wang et al., 5 Mar 2025). A plausible implication is that DrawMotion is not simply a condition-added variant of text-to-motion diffusion; it is a redefinition of the control interface around multi-scale drawing inputs.

4. Supervision strategy and intermediate feature guidance

Training uses a unified objective combining trajectory, stickman, and full-motion reconstruction losses: $\widehat{J}^t \in \mathbb{R}^{(T,2)}$3

$\widehat{J}^t \in \mathbb{R}^{(T,2)}$4

$\widehat{J}^t \in \mathbb{R}^{(T,2)}$5

$\widehat{J}^t \in \mathbb{R}^{(T,2)}$6

Here, $\widehat{J}^t \in \mathbb{R}^{(T,2)}$7 enforces global path fidelity, $\widehat{J}^t \in \mathbb{R}^{(T,2)}$8 supervises pose fidelity at selected stickman positions, and $\widehat{J}^t \in \mathbb{R}^{(T,2)}$9 reconstructs the full motion. Random binary masks $N$0 allow arbitrary combinations of stickman insertions during training (Wang et al., 20 May 2026).

A second major component is the training-free method Intermediate Feature Guidance (IFG). The paper’s premise is that the intermediate features inside the MCM occupy a relatively continuous latent space, making them suitable for gradient-based optimization during inference. This is contrasted with the latent spaces used by earlier training-free methods such as OmniControl and DNO, which the paper characterizes as operating at less suitable locations in the diffusion process (Wang et al., 20 May 2026).

The method splits the model around a chosen MCM layer into $N$1 and $N$2. During DDIM reverse sampling, it extracts an intermediate feature $N$3, optimizes it with SGD against a spatial loss on the predicted clean motion,

$N$4

and updates directly in feature space: $N$5 To prevent feature drift, IFG uses Mahalanobis-distance clipping: $N$6 and if $N$7, the update is clipped as

$N$8

The paper also analyzes feature continuity with a perturbation test,

$N$9

where $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$0 is a batch-shuffled feature, and reports that DrawMotion’s MCM maintains stable FID under large perturbations whereas ReMoDiffuse collapses quickly (Wang et al., 20 May 2026).

5. Empirical performance and user studies

DrawMotion is evaluated on KIT-ML and HumanML3D. On HumanML3D, it reports FID $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$1, R-Precision Top-1 $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$2, Top-2 $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$3, Top-3 $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$4, MM Dist $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$5, Diversity $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$6, MultiModality $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$7, and StiSim $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$8. On KIT-ML, it reports FID $$\ell_n = 0.1 \times \lVert \text{limb\_offset}^{gt} - \text{limb\_offset}^{pred}_n \rVert_2^2,$$9, Top-1 $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$0, Top-2 $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$1, Top-3 $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$2, MM Dist $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$3, Diversity $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$4, MultiModality $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$5, and StiSim $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$6 (Wang et al., 20 May 2026).

For motion-editing comparisons, the strongest reported gains are in trajectory fidelity. On HumanML3D, Traj.Err. drops to $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$7, compared with $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$8 for OmniControl and $$\ell^{\text{final}} = 10 \times \ell_k + \sum_{n=1}^{N}\ell_n,\quad k=\arg\min_n \ell_n.$$9 for DNO. On KIT-ML, it reaches $$q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}\!\left(\mathbf{x}_t;\sqrt{\alpha_t}\mathbf{x}_{t-1},(1-\alpha_t)\mathbf{I}\right),$$0, again substantially below earlier methods. Efficiency comparisons at DDIM-50 report about 2,245 MB GPU memory and 24 seconds per batch, compared with 153 seconds per batch for OmniControl and 358 seconds per batch for DNO (Wang et al., 20 May 2026).

The user study recruits 20 volunteers who imagine a 10-second motion and provide either a detailed textual description or a combination of text and hand-drawing. Participants then rate outputs from ReMoDiffuse, StickMotion, and DrawMotion. DrawMotion obtains the highest user score, 9.5, compared with 8.5 for StickMotion and 7.3 for ReMoDiffuse. The paper also reports user times of 33.8 seconds for ReMoDiffuse, 16.4 seconds for StickMotion, and 34.3 seconds for DrawMotion, while making the headline claim of a 46.7% reduction in user time when generating motions aligned with imagination. It further notes that manual 3D stickman animation by professional animators takes around 3 hours per sample (Wang et al., 20 May 2026).

The ablations align with the architectural design. On KIT-ML, the configuration using efficient attention for text and dot-product attention for draw yields FID $$q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}\!\left(\mathbf{x}_t;\sqrt{\alpha_t}\mathbf{x}_{t-1},(1-\alpha_t)\mathbf{I}\right),$$1, R-prec(top3) $$q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}\!\left(\mathbf{x}_t;\sqrt{\alpha_t}\mathbf{x}_{t-1},(1-\alpha_t)\mathbf{I}\right),$$2, StiSim $$q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}\!\left(\mathbf{x}_t;\sqrt{\alpha_t}\mathbf{x}_{t-1},(1-\alpha_t)\mathbf{I}\right),$$3, and Traj.Err. $$q(\mathbf{x}_{1:T}|\mathbf{x}_0)=\prod_{t=1}^T q(\mathbf{x}_t|\mathbf{x}_{t-1}),\qquad q(\mathbf{x}_t|\mathbf{x}_{t-1})=\mathcal{N}\!\left(\mathbf{x}_t;\sqrt{\alpha_t}\mathbf{x}_{t-1},(1-\alpha_t)\mathbf{I}\right),$$4. The third MCM layer is selected as the best trade-off for IFG, since deeper guidance improves trajectory error but can worsen FID (Wang et al., 20 May 2026).

6. Relation to adjacent research and limitations

DrawMotion belongs to a broader family of drawing-conditioned motion systems, but its immediate technical lineage is most clearly visible in 3D human motion synthesis. Relative to StickMotion, it retains diffusion-based multi-condition fusion and automatic stickman generation, but replaces the restricted temporal interface with freehand trajectories and multiple stickmen at arbitrary positions, and adds the training-free Intermediate Feature Guidance mechanism (Wang et al., 5 Mar 2025). In that sense, DrawMotion can be understood as a refinement of drawing-conditioned text-to-motion control from sparse temporal anchors toward continuous spatial authoring (Wang et al., 20 May 2026).

In neighboring areas, related work applies drawing-based control to other media. DragAnything reformulates drag-to-control-motion in video generation from “drag a point” to “drag an entity,” using latent entity features and trajectories for controllable video motion (Wu et al., 2024). SMCD integrates still-image scene conditioning with trajectory-based box control for image-to-video generation (Li et al., 2024). Earlier sketch-based animation systems such as DualMotion use rough trajectory sketches to retrieve and compose motion-capture sequences for global and local character animation design (Peng et al., 2022). These works share a common premise—drawing as a compact motion specification—but differ substantially in output domain, conditioning granularity, and whether motion is generated, retrieved, or edited.

The limitations stated for DrawMotion are primarily about condition coherence. The framework depends on reasonable and consistent user input: if the hand-drawn trajectory or stickman conflicts with the text, or if the request violates human-motion plausibility, the generated result may deviate from the input and fidelity may drop (Wang et al., 20 May 2026). The paper identifies richer drawing primitives, better conflict resolution, more robust handling of ambiguous sketches, and surfacing the final IFG loss to the user as natural extensions. A plausible implication is that DrawMotion’s main unresolved problem is not raw generative capacity but arbitration among semantically valid yet mutually incompatible control signals.

Within the literature summarized here, DrawMotion therefore marks a specific transition point: from text-dominant motion generation with auxiliary pose hints toward a genuinely drawing-centered interface for 3D human motion synthesis. Its distinct contribution is not only that it accepts sketches, but that it organizes global path control, local pose control, efficient multi-condition fusion, and inference-time feature-space optimization into a single diffusion framework (Wang et al., 20 May 2026).