CoMA/CoMAS: Multi-modal Human Motion Synthesis

• The paper introduces hierarchical, agent-based models to decompose human motion into articulated parts using part-specific VQ-VAE codebooks.
• It leverages multi-modal inputs—text, music, and speech—to drive precise motion generation and editing through a unified Transformer framework.
• Empirical results show state-of-the-art performance on benchmarks like HumanML3D, with scalable integration of new modalities and anatomical parts.

Compositional Human Motion Generation with Multi-modal Agents (CoMA/CoMAS) denotes a class of frameworks that consolidate multimodal, part-aware human motion synthesis using collaborative agents and compositional neural architectures. These methods address the need for fine-grained, scalable, and extensible control over articulated 3D motion driven by diverse input modalities (e.g., text, music, speech). Recent approaches eliminate monolithic architectures in favor of hierarchical, agent-based systems and discrete, body part–decomposed motion representations. This enables both highly detailed motion editing and the easy addition of new input modalities or anatomical parts, while achieving state-of-the-art performance in conditional motion generation and editing.

1. Architectural Foundations of CoMA/CoMAS

CoMA/CoMAS architectures employ a modular, agent-based pipeline for context-rich human motion generation, editing, and comprehension (Zhou et al., 2023, Sun et al., 2024). The canonical pipeline consists of:

• Hierarchical Motion Tokenization: The human body is decomposed into distinct parts (e.g., torso, left hand, right hand for (Zhou et al., 2023); left-upper, right-upper, left-lower, right-lower in (Sun et al., 2024)). Each part's motion sequence is encoded via a part-specific vector-quantized variational autoencoder (VQ-VAE) into discrete token streams, stored in dedicated codebooks.
• Multimodal Signal Encoding: Control modalities (text, music, speech) are encoded with large, pretrained encoders—e.g., CLIP for text, MTR/Jukebox for music, HuBERT for speech. Adapters map these embeddings into a shared latent space of fixed dimension.
• Agent-Based Fusion: “Agents” contribute command embeddings corresponding to each input modality. A shared encoder–decoder Transformer processes the concatenation of these embeddings, with a base decoder capturing global spatiotemporal context and head decoders (one per (body part, modality) pair) generating discrete motion tokens in parallel.
• Continuous Reconstruction: Predicted discrete tokens are mapped back to continuous feature vectors through part-specific VQ-VAE decoders, merged at the frame level to produce full 3D poses.

The control flow is compositional and agent-driven: each body part’s generation is handled by a specialized sub-agent, while multimodal commands coordinate global structure across the synthesized sequence.

2. Motion Quantization and Codebook Strategies

Motion quantization forms the backbone of discrete compositionality in CoMA/CoMAS systems (Zhou et al., 2023, Sun et al., 2024). For each anatomical part, continuous pose features ($x \in \mathbb{R}^{T \times c}$) are encoded and quantized:

• Encoding: Each part encoder $\mathcal{E}$ maps the input sequence to pose-space $\mathbb{R}^{T' \times d}$.
• Quantization: For each time step, latent $z_i$ is quantized to the nearest codebook vector $e_j$ using

$$z_i^q = e_j \quad\text{where}\quad e_j = \operatorname{argmin}_{e \in \mathcal{E}} \lVert z_i - e \rVert_2$$

with separate codebooks for each part (e.g., torso codebook size $K_\text{torso}=1024$, hand $K_\text{hand}=512$, $d=512$).

• Hierarchical and Residual VQ-VAE: Some frameworks (e.g., (Sun et al., 2024)) employ multi-stage or residual quantization, with multiple encoder/quantizer layers stacked together. In (Zhou et al., 2023), torso decoding is two-stage: a local decoder reconstructs relative pose, and a U-Net refines the frame-wise output globally.
• Loss Functions: Training optimizes $$\mathcal{L}_\text{VQ} = \mathcal{L}_\text{rec} + \beta_1 \lVert \mathrm{sg}[z] - z^q \rVert + \beta_2 \lVert z - \mathrm{sg}[z^q] \rVert$$, aggregating local and global reconstruction.

This quantization enables compositional control at a part-wise, token-level granularity.

3. Multi-modal and Multi-agent Conditioning

CoMA/CoMAS interprets each input modality as a distinct agent (Zhou et al., 2023):

• Text agent: Processes text prompts via a CLIP encoder and adapter, producing $c_\text{text}$ in the shared latent space.
• Music agent: Encodes audio using MTR or similar models, yielding $c_\text{music}$.
• Speech agent: Leverages HuBERT, mapping to $c_\text{speech}$.
• Fusion: All agents’ embeddings are concatenated (with appropriate padding/special tokens) into a single input sequence for the Transformer encoder, establishing a compositional controller that interprets multi-modal commands.

The agent architecture allows easy integration of new control modalities (e.g., EMG, video) by training an adapter on the new incoming signal, without retraining the base motion generator or existing heads.

4. Token Prediction and Sequence Generation

The motion synthesis core is a Transformer that accepts the fused agent conditioning and predicts body part motion tokens in an autoregressive, parallelized manner (Zhou et al., 2023, Sun et al., 2024):

• Encoder–Decoder Design: An $L$-layer Transformer encoder consumes the agent sequence. The base decoder (6 layers) models common motion context, and separate head decoders (2 layers each per body part–modality pair) specialize in localized articulation.
• Autoregressive Factorization: For each part $b$ and modality $p$, token probabilities are factorized as

$$p(x_{b,p}) = \prod_{i=1}^n p_{\theta, \theta_{b,p}}(x_{b,p,i} \mid x_{b,p,<i}, C_p)$$

with tokens sampled from part- and modality-specific codebooks.

• Objective: The negative log-likelihood is minimized over all part–modality combinations:

$$\mathcal{L} = -\sum_{b,p} \sum_{i=1}^n \log P(x_{b,p,i} \mid x_{b,p,<i}, C_p)$$

• Auxiliary Conditioning: Identity or contextual factors are encoded as learned vocabulary tokens on the encoder side, allowing agent-specific or environmental modifications.

5. Editing, Self-Correction, and Compositional Control

Fine-grained compositional editing and quality refinement is a hallmark of the agent-based approach (Sun et al., 2024):

• Segmented Planning & Editing: Task planner agents decompose input prompts into atomic temporo-spatial instructions, mapping these to base and part-specific edits.
• Masked-token Editing: The motion generator supports inpainting of temporal spans or body parts by masking respective token sequences and sampling new predictions, conditioned on the rest and the agent commands.
• Trajectory Editor: NLP-based chain-of-thought reasoning transforms trajectory prompts into parametric motion curves, which are injected as root-joint targets.
• Self-Correction: A vision-language “motion reviewer” renders and captions generated motion, compares it to the original prompt, and generates further edit instructions for iterative improvement. This loop may be repeated for multiple rounds.
• Metric: The quality of alignment and editability is quantified via scores such as MAS (Motion Alignment Score), calculated by cosine similarity in video-language embedding spaces.

6. Scalability, Modularity, and Extensibility

The CoMA/CoMAS paradigm is inherently scalable:

• Adding Parts: Each new anatomical articulation is handled by training a VQ-VAE and integrating a new head decoder in the Transformer, leaving prior modules untouched (Zhou et al., 2023).
• Adding Modalities: New control modalities require only a pretrained encoder, a lightweight adapter for projection, and input formatting for the shared Transformer sequence. Training effort is focused solely on the new agent.
• Lightweight Integration: This design decouples existing generation capacity from extension, eliminating catastrophic forgetting or architectural redundancy.
• Broader Context: Alternative frameworks employing unified instruction/motion tokenization (e.g., OmniMoGen (Bu et al., 22 Dec 2025)) and learnable model composition with dynamic mixing (MixerMDM (Ruiz-Ponce et al., 1 Apr 2025)) validate the scalability and compositional control offered by such agent-based, multi-modal frameworks.

7. Empirical Results and Benchmarks

CoMA/CoMAS systems exhibit state-of-the-art or competitive results on standard and challenging motion generation benchmarks (Sun et al., 2024, Zhou et al., 2023):

• HumanML3D: The SPAM model (core of CoMA) achieves R-Precision@1 = 0.526, outperforming baselines on compositional tasks.
• Fine-grained Prompts: Performance remains robust even as prompt complexity increases; other methods degrade sharply.
• User Studies: CoMA consistently outperforms strong baselines (e.g., MoMask, ReMoDiffuse) on context-rich, compositional, and long-form prompt collections both qualitatively and quantitatively.
• Ablations: Factorized Transformer spatial attention and single-decoder architectures (as opposed to four separate decoders) lead to superior FID and editing performance.
• Scalability: New modalities or part additions are empirically validated to have negligible negative transfer to prior skills and require minimal retraining time.

These results demonstrate that the compositional and agent-based paradigm provides not only fine-grained control and high diversity in generated motions but also preserves scalability and flexibility in practical multi-modal animation pipelines.

---

References:

• "A Unified Framework for Multimodal, Multi-Part Human Motion Synthesis" (Zhou et al., 2023)
• "CoMA: Compositional Human Motion Generation with Multi-modal Agents" (Sun et al., 2024)
• "OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions" (Bu et al., 22 Dec 2025)
• "MixerMDM: Learnable Composition of Human Motion Diffusion Models" (Ruiz-Ponce et al., 1 Apr 2025)
• "Unified Multi-Modal Interactive & Reactive 3D Motion Generation via Rectified Flow" (Gupta et al., 28 Sep 2025)