--- title: 'CoMA/CoMAS: Multi-modal Human Motion Synthesis' url: https://www.emergentmind.com/topics/compositional-human-motion-generation-with-multi-modal-agents-coma-comas type: topic --- # CoMA/CoMAS: Multi-modal Human Motion Synthesis Compositional Human Motion Generation with Multi-modal Agents (CoMA/CoMAS) denotes a class of frameworks that consolidate multimodal, part-aware human motion synthesis using collaborative agents and compositional neural architectures. These methods address the need for fine-grained, scalable, and extensible control over articulated 3D motion driven by diverse input modalities (e.g., text, music, speech). Recent approaches eliminate monolithic architectures in favor of hierarchical, agent-based systems and discrete, body part–decomposed motion representations. This enables both highly detailed motion editing and the easy addition of new input modalities or anatomical parts, while achieving state-of-the-art performance in conditional motion generation and editing. ## 1. Architectural Foundations of CoMA/CoMAS CoMA/CoMAS architectures employ a modular, agent-based pipeline for context-rich human motion generation, editing, and comprehension [2311.16471, 2412.07320]. The canonical pipeline consists of: - **Hierarchical Motion Tokenization:** The human body is decomposed into distinct parts (e.g., torso, left hand, right hand for [2311.16471]; left-upper, right-upper, left-lower, right-lower in [2412.07320]). Each part's motion sequence is encoded via a part-specific vector-quantized variational autoencoder (VQ-VAE) into discrete token streams, stored in dedicated codebooks. - **Multimodal Signal Encoding:** Control modalities (text, music, speech) are encoded with large, pretrained encoders—e.g., CLIP for text, MTR/Jukebox for music, HuBERT for speech. Adapters map these embeddings into a shared latent space of fixed dimension. - **Agent-Based Fusion:** “Agents” contribute command embeddings corresponding to each input modality. A shared encoder–decoder Transformer processes the concatenation of these embeddings, with a base decoder capturing global spatiotemporal context and head decoders (one per (body part, modality) pair) generating discrete motion tokens in parallel. - **Continuous Reconstruction:** Predicted discrete tokens are mapped back to continuous feature vectors through part-specific VQ-VAE decoders, merged at the frame level to produce full 3D poses. The control flow is compositional and agent-driven: each body part’s generation is handled by a specialized sub-agent, while multimodal commands coordinate global structure across the synthesized sequence. ## 2. Motion Quantization and Codebook Strategies Motion quantization forms the backbone of discrete compositionality in CoMA/CoMAS systems [2311.16471, 2412.07320]. For each anatomical part, continuous pose features ($x \in \mathbb{R}^{T \times c}$) are encoded and quantized: - **Encoding:** Each part encoder $\mathcal{E}$ maps the input sequence to pose-space $\mathbb{R}^{T' \times d}$. - **Quantization:** For each time step, latent $z_i$ is quantized to the nearest codebook vector $e_j$ using $$ z_i^q = e_j \quad\text{where}\quad e_j = \operatorname{argmin}_{e \in \mathcal{E}} \lVert z_i - e \rVert_2 $$ with separate codebooks for each part (e.g., torso codebook size $K_\text{torso}=1024$, hand $K_\text{hand}=512$, $d=512$). - **Hierarchical and Residual VQ-VAE:** Some frameworks (e.g., [2412.07320]) employ multi-stage or residual quantization, with multiple encoder/quantizer layers stacked together. In [2311.16471], torso decoding is two-stage: a local decoder reconstructs relative pose, and a U-Net refines the frame-wise output globally. - **Loss Functions:** Training optimizes $\mathcal{L}_\text{VQ} = \mathcal{L}_\text{rec} + \beta_1 \lVert \mathrm{sg}[z] - z^q \rVert + \beta_2 \lVert z - \mathrm{sg}[z^q] \rVert$, aggregating local and global reconstruction. This quantization enables compositional control at a part-wise, token-level granularity. ## 3. Multi-modal and Multi-agent Conditioning CoMA/CoMAS interprets each input modality as a distinct agent [2311.16471]: - **Text agent:** Processes text prompts via a CLIP encoder and adapter, producing $c_\text{text}$ in the shared latent space. - **Music agent:** Encodes audio using MTR or similar models, yielding $c_\text{music}$. - **Speech agent:** Leverages HuBERT, mapping to $c_\text{speech}$. - **Fusion:** All agents’ embeddings are concatenated (with appropriate padding/special tokens) into a single input sequence for the Transformer encoder, establishing a compositional controller that interprets multi-modal commands. The agent architecture allows easy integration of new control modalities (e.g., EMG, video) by training an adapter on the new incoming signal, without retraining the base motion generator or existing heads. ## 4. Token Prediction and Sequence Generation The motion synthesis core is a Transformer that accepts the fused agent conditioning and predicts body part motion tokens in an autoregressive, parallelized manner [2311.16471, 2412.07320]: - **Encoder–Decoder Design:** An $L$-layer Transformer encoder consumes the agent sequence. The base decoder (6 layers) models common motion context, and separate head decoders (2 layers each per body part–modality pair) specialize in localized articulation. - **Autoregressive Factorization:** For each part $b$ and modality $p$, token probabilities are factorized as $$ p(x_{b,p}) = \prod_{i=1}^n p_{\theta, \theta_{b,p}}(x_{b,p,i} \mid x_{b,p,