Controllable Motion Generation (CoMo)

• Controllable Motion Generation (CoMo) is a framework that synthesizes motion from explicit control signals—like text, pose codes, and trajectories—to generate semantically meaningful movement.
• CoMo systems use diverse representations, from discrete semantic codes to continuous latent embeddings, to allow precise editing and localized temporal control.
• Architectural patterns in CoMo balance high-quality motion generation with fine-grained control, addressing tradeoffs between realistic motion and editing precision.

Controllable Motion Generation (CoMo) denotes a family of generative formulations in which motion is synthesized under explicit control signals rather than from unconstrained text alone. In the original 3D human-motion formulation, CoMo is a “Controllable Motion generation model” that represents motion as “discrete and semantically meaningful pose codes” and allows a LLM to edit those codes under natural-language instructions (Huang et al., 2024). Subsequent work uses the term more broadly for compositional human motion, continuous latent motion for robot learning, mask-guided video dynamics, compositional motion customization for text-to-video, and other systems in which high-level instructions are mapped to structured control over trajectories, body parts, latents, or video dynamics (Sun et al., 2024, Yang et al., 22 May 2025, Meric et al., 21 May 2026, Xu et al., 27 Oct 2025).

1. Historical development and problem formulation

A central prehistory of CoMo appears in “TrajeVAE: Controllable Human Motion Generation from Trajectories” (Kania et al., 2021), which reformulates “future pose prediction into pose completion in space and time where multiple trajectories are represented as poses with missing joints.” This formulation already establishes a core CoMo theme: observed control signals define only part of the motion state, and the generator completes the remaining degrees of freedom while preserving temporal coherence.

In 2024, several strands of controllability became explicit. “MotionMix: Weakly-Supervised Diffusion for Controllable Motion Generation” separates diffusion denoising into “conditioned coarse synthesis” and “unconditional refinement,” showing that controllability can be retained even when supervision is weak (Hoang et al., 2024). “MotionChain: Conversational Motion Controllers via Multimodal Prompts” frames control as a multi-turn dialogue problem in which text, image, and motion are transformed into a unified token space and used to generate “continuous and long-term human motion through multimodal prompts” (Jiang et al., 2024). “CoMo: Controllable Motion Generation through Language Guided Pose Code Editing” then makes controllability itself the organizing principle of a text-to-motion framework, emphasizing modifying “subtle postures within a motion” and “inserting new actions at specific moments” through language-guided editing of semantic pose codes (Huang et al., 2024).

Later papers broaden the design space rather than converge on a single implementation. CoMA introduces a multi-agent system for “complex human motion generation, editing, and comprehension,” with task planning, body-part-specific generation, text-guided editing, trajectory control, and self-correction (Sun et al., 2024). FineXtrol replaces globally expanded descriptions with “temporally-aware, precise, user-friendly, and fine-grained textual control signals that describe specific body part movements over time” (Shen et al., 24 Nov 2025). COMET targets “real‑time, long‑term controllable human motion generation” through “extended joint targets” and reference-guided feedback (Lee et al., 4 Dec 2025). Kimodo scales a single kinematic diffusion model to “700 hours of optical motion capture data” and supports “full-body keyframes, sparse joint positions/rotations, 2D waypoints, and dense 2D paths” (Rempe et al., 16 Mar 2026). In parallel, video-generation work adopts the same CoMo logic through mask sequences, motion fields, sparse temporal attention, or compositional motion modules (Meric et al., 21 May 2026, Liao et al., 5 Feb 2025, Ling et al., 2024, Xu et al., 27 Oct 2025).

This history suggests that CoMo is not a single control interface but a recurring problem formulation: how to keep generation quality high while exposing motion variables that are temporally localized, semantically legible, and operationally precise.

2. Control signals and internal representations

Across the literature, CoMo systems differ mainly in the choice of control signal and in the internal representation through which control is enforced. Some systems operate on symbolic pose descriptors, some on trajectories or joint targets, some on text segmented by body part and time, and some on latent residuals or optical flow.

System	Primary control signal	Internal representation
TrajeVAE (Kania et al., 2021)	Initial pose + arbitrary number of joint trajectories	Poses with missing joints
CoMo (Huang et al., 2024)	Text + editing instructions	392 semantic pose codes in 70 pose categories
CoMA (Sun et al., 2024)	Global prompt, segment prompts, local edit prompts, trajectory prompts	Four body-part-specific RVQ codebooks and a shared decoder
ECM (Yu et al., 2024)	Initial frame + motion curves per cardiac substructure	Fourier-transformed motion-curve embeddings
CoMo for robot learning (Yang et al., 22 May 2025)	Action-less video pairs $(O_t, O_{t+n})$	Continuous latent motion $Z_t \in \mathbb{R}^{d_z}$
FineXtrol (Shen et al., 24 Nov 2025)	Coarse text + fine-grained temporally-aware text	HCL-trained T5 control embeddings
CoMoGen (Meric et al., 21 May 2026)	Binary mask sequence + input image	Latent residual $\Delta Z$ injected into Motion Layers

The original CoMo uses a hand-crafted semantic codebook rather than a learned VQ codebook. Motion is encoded as K-hot vectors over 392 codes grouped into 70 pose categories, with each code corresponding to thresholded semantics such as “left knee angle,” “distance between left hand and left foot,” or “ground contact of right knee” (Huang et al., 2024). This makes the representation directly readable by a LLM and directly editable at the level of body-part semantics.

CoMA retains discrete structure but shifts from semantic handcrafting to a “Spatially-Aware Residual VQ‑VAE” with four body-part-specific streams—LU, RU, LL, RL—and a shared decoder (Sun et al., 2024). The resulting latent supports masking and inpainting of “specific body-part token streams,” and the control signal itself is factorized into $P_{\text{base}^i}$, $P_{\text{edit}^i}$, and $P_{\text{traj}^i}$. This separates coarse action, local limb behavior, and path control within each temporal segment.

Other CoMo formulations replace symbolic discreteness with continuous or geometric control. CoMo for robot learning uses a continuous latent motion embedding $Z_t$ extracted from frame pairs, motivated by the claim that “continuous motion embeddings” can represent “arbitrarily fine-grained differences” and serve as “continuous pseudo actions” for policy learning (Yang et al., 22 May 2025). FineXtrol keeps control textual but makes it “temporally-aware” and “body part” specific by structuring descriptions at sentence-, snippet-, and sequence-level granularity, separated by <SEP> and optionally <Motionless> tokens (Shen et al., 24 Nov 2025). ECM uses “motion curves” derived from cardiac-structure bounding boxes and transformed by Fourier features, while MotionAgent converts text into “object trajectories” and “camera extrinsics,” then composes them analytically into a unified optical flow (Yu et al., 2024, Liao et al., 5 Feb 2025).

A plausible implication is that the representation choice determines both the ceiling and the ergonomics of controllability. Semantic pose codes are interpretable and editable; body-part RVQ codes support local inpainting; continuous latents favor smooth control and policy learning; geometric motion fields make camera and object motion explicit.

3. Architectural patterns for controllability

A first architectural pattern is symbolic editing over an interpretable token space. In CoMo, a rule-based skeleton parser converts motion into semantic pose codes, a decoder-only Transformer predicts code sequences from text, and GPT‑4 edits code IDs through a three-stage prompt sequence: select frames, select affected pose categories, and rewrite code sequences (Huang et al., 2024). Because the representation is discrete and semantically meaningful, the LLM intervenes directly on the motion representation rather than only rewriting the text description.

A second pattern is residual control on top of a strong generative prior. FineXtrol copies MDM into a frozen lower branch and a trainable upper branch. The lower branch conditions on the coarse prompt $\boldsymbol{p}$, the upper branch adds a fine-grained control embedding $\boldsymbol{e}_c$, and zero-initialized projections $\mathcal{P}_l$ inject residual corrections into each layer (Shen et al., 24 Nov 2025). FlexMotion uses a similar ControlNet-style principle in latent space: a frozen diffusion denoiser is augmented by a trainable control branch and zero-initialized layers so that joint locations, joint actuations, contact forces, or muscle activations can steer generation without retraining the base model (Tashakori et al., 28 Jan 2025). CoMoGen applies the same residual-control logic in a video DiT: a lightweight MaskAdapter computes $Z_t \in \mathbb{R}^{d_z}$0, which is injected only into a subset of “Motion Layers,” while LoRA is also restricted to those layers (Meric et al., 21 May 2026).

A third pattern is factorization of motion into controllable subproblems. CoMA factorizes by body part and by agent role: task planning, motion generation, trajectory editing, and motion review are separated and then recombined in a closed loop (Sun et al., 2024). COMET factorizes control into pelvis intention, orientation intention, and per-joint control intentions, then introduces a reference-guided feedback mechanism that pulls the autoregressive rollout toward a GMM pose manifold (Lee et al., 4 Dec 2025). Kimodo factorizes the denoiser itself into a root stage and a body stage, using a global root representation for the first stage and a local root representation $Z_t \in \mathbb{R}^{d_z}$1 for the second (Rempe et al., 16 Mar 2026). The paper reports that a one-stage model yields substantially worse constraint errors and higher foot skate, which directly ties factorization to controllability (Rempe et al., 16 Mar 2026).

A fourth pattern is agentic reasoning over motion plans. CoMA’s Task Planner uses GPT‑4o to rewrite prompts into dataset-compatible text, segment them temporally, and derive base, edit, and trajectory prompts; the Motion Reviewer captions rendered motion segments with MVC and asks GPT‑4o to compare those captions against the original prompts, producing correction instructions for iterative re-editing (Sun et al., 2024). MotionAgent similarly uses GPT‑4o plus Grounded‑SAM to parse natural-language motion into object-specific trajectories and camera extrinsics before handing the result to a frozen diffusion model (Liao et al., 5 Feb 2025).

4. Temporal composition, editing, and long-horizon control

Temporal structure is the most persistent difficulty in CoMo. Original CoMo addresses it by allowing edits to be localized in specific frame ranges and by representing each downsampled frame as a K-hot set of body-part semantics (Huang et al., 2024). FineXtrol makes temporal locality explicit in the control language itself: snippet-level descriptions are ordered by interval, and <SEP> tokens preserve the interval structure that the contrastive encoder later learns to discriminate (Shen et al., 24 Nov 2025).

CoMA introduces a more explicit temporal pipeline. A prompt $Z_t \in \mathbb{R}^{d_z}$2 is recaptioned into $Z_t \in \mathbb{R}^{d_z}$3, split into $Z_t \in \mathbb{R}^{d_z}$4, and each segment is decomposed into base motion, local editing, and optional trajectory control. The Motion Generator can perform “in-between editing,” “body-part editing,” and “blend editing” through masked prediction over time and parts, while the Motion Reviewer can trigger a self-correction loop for each segment up to $Z_t \in \mathbb{R}^{d_z}$5 iterations (Sun et al., 2024). This is a direct move from one-shot generation to iterative temporal refinement.

Long-horizon control appears in several distinct forms. MotionChain turns continuity into a conversation problem: each assistant response is a sequence of motion tokens that becomes part of the context for the next turn, and token-level concatenation followed by joint decoding yields the best MPJPE, PA-MPJPE, and ACCL among the tested composition strategies (Jiang et al., 2024). COMET frames long-term control as autoregressive frame generation with “extended joint targets,” where any subset of six controllable joints can be assigned target positions and times; reference-guided feedback suppresses drift over long rollouts and also functions as a plug-and-play stylization mechanism (Lee et al., 4 Dec 2025). Kimodo supports full-body keyframes, sparse joint positions and rotations, 2D waypoints, dense 2D paths, and multi-prompt sequencing by overlapping consecutive generations and constraining the overlap with full-body keyframes (Rempe et al., 16 Mar 2026).

Video-generation CoMo treats temporal control differently but with the same underlying goal. CoMoGen modulates mask influence with a reverse cosine schedule,

$Z_t \in \mathbb{R}^{d_z}$6

so that mask control dominates early denoising steps and vanishes at the final step (Meric et al., 21 May 2026). MotionClone identifies motion-driving components in temporal attention maps and restricts guidance to sparse, primary temporal-attention entries, arguing that dense temporal attention includes substantial noisy or very subtle motion (Ling et al., 2024). These designs suggest that temporal controllability often depends not only on richer signals but also on choosing the right stage of the generative process at which those signals are applied.

5. Evaluation paradigms and what they measure

CoMo evaluation is heterogeneous because different systems control different objects. In 3D human motion, the default metrics remain R-Precision, FID, MM-DIST, Diversity, and MModality. The original CoMo reports competitive text-to-motion performance on HumanML3D and KIT-ML while emphasizing that its main advantage is motion editing: in a user study with 54 participants and 20 editing scenarios, CoMo is preferred more than 70% of the time on average, especially for “body part modification” and “action add/delete” (Huang et al., 2024).

When control becomes more fine-grained, evaluation changes. FineXtrol studies “control signal density” from 25% to 100% and compares per-body-part and cross-body-part settings while reporting standard motion metrics. On HumanML3D, FineXtrol improves over MDM and CoMo in FID, R-Precision, Diversity, and MM-Dist, and a user study reports preference for FineXtrol in both naturalness and control matching (Shen et al., 24 Nov 2025). CoMA supplements standard motion metrics with caption-based evaluation and a user study on 40 “challenging, context-rich, compositional prompts,” and introduces Motion Alignment Score,

$Z_t \in \mathbb{R}^{d_z}$7

which increases after editing passes, for example from 28.61 to 29.40 (Sun et al., 2024).

Robot-oriented CoMo requires different diagnostics. CoMo for robot learning introduces LP-MSE, the MSE of a linear probe from latent motion to real actions, and S-PCFC, the cosine similarity between past-to-current and future-to-current motion embeddings. LP-MSE measures action-relevant information, whereas lower S-PCFC indicates less shortcut learning and stronger directionality (Yang et al., 22 May 2025). In video CoMo, CoMoGen uses SSIM, PSNR, LPIPS, FVD, $Z_t \in \mathbb{R}^{d_z}$8, $Z_t \in \mathbb{R}^{d_z}$9, HOTA, and a VQA-based text-to-video alignment metric, while MotionAgent uses VBench’s “Video-Text Camera Motion” and a new “Object Movement Q&A” protocol (Meric et al., 21 May 2026, Liao et al., 5 Feb 2025). Compositional text-to-video CoMo adds a new Crop-and-Compare metric, C$\Delta Z$0C, to penalize motion blending when several learned motions coexist in one scene (Xu et al., 27 Oct 2025).

Taken together, these metrics reveal a methodological point: “controllability” is not a single scalar. It may mean semantic agreement, local limb correctness, trajectory error, action informativeness of a latent, path adherence, camera-motion fidelity, multi-subject disentanglement, or long-horizon stability.

6. Limitations, misconceptions, and open directions

A recurrent misconception is that finer control automatically improves realism. The papers repeatedly show tradeoffs. Original CoMo notes that its pose codes focus mainly on local kinematics and do not explicitly represent “overall speed, style, trajectory path, or repetition patterns,” while the decoder imposes no explicit physical constraints (Huang et al., 2024). FineXtrol’s finer text embeddings improve control but still rely on the base diffusion model’s ability to realize those embeddings in motion space (Shen et al., 24 Nov 2025). FlexMotion shows that stronger physics losses improve penetration, skating, and muscle-limit metrics, but the paper explicitly describes a trade-off between physical fidelity and text-motion metrics (Tashakori et al., 28 Jan 2025).

Another misconception is that control can be solved by adding more modalities without revisiting system design. CoMA reports a “rendering bottleneck,” “over-editing / refinement instability,” and a “non end-to-end” architecture in which LLM, VLM, and SPAM are trained separately (Sun et al., 2024). CoMoGen shows that mask-only control still struggles under “extreme and prolonged occlusion,” “very long videos with many interactions,” and ambiguous masks that specify “where” but not “how” (Meric et al., 21 May 2026). MotionAgent inherits the limits of LLM motion planning and of approximate 3D geometry, especially for complex motion descriptions and extreme camera motions (Liao et al., 5 Feb 2025). Kimodo, despite strong kinematic control, remains “purely kinematic,” and the robotics demonstration still requires a separate physics-based tracking policy (Rempe et al., 16 Mar 2026). COMET reduces long-term drift but also remains kinematic and is limited to a fixed controllable joint set (Lee et al., 4 Dec 2025).

The field therefore appears to be moving toward more unified models, but not by discarding explicit control. CoMA proposes a “single multimodal model” that understands and generates motion from text and video and performs internal self-critique without rendering (Sun et al., 2024). FlexMotion points toward control spaces that include joint locations, torques, contact forces, and muscle activations (Tashakori et al., 28 Jan 2025). Kimodo suggests that scaling data and model size materially improves both quality and control fidelity (Rempe et al., 16 Mar 2026). A plausible implication is that future CoMo systems will combine semantically rich text interfaces, temporally explicit control signals, geometry-aware or physics-aware priors, and evaluation protocols that separate motion quality from motion obedience.