Being-M0.5: Controllable Vision-Language Motion Model

• Being-M0.5 is a real-time, fully controllable vision-language-motion model that integrates computer vision, natural language, and 3D human motion.
• It overcomes previous limitations by leveraging a large-scale multimodal dataset and novel part-aware residual quantization for fine-grained motion control.
• The model achieves state-of-the-art performance with real-time inference (≥20 FPS) using an advanced LLaMA-2-chat backbone and efficient, interactive tokenization.

Being-M0.5 is a real-time, fully controllable vision-language-motion model (VLMM) that unifies the domains of computer vision, natural language, and 3D human motion. It addresses key bottlenecks in controllability that have constrained prior VLMMs, including insufficient responsiveness to natural language, poor pose initialization, degraded long-term sequence modeling, generalization challenges for unseen motions, and lack of fine body-part control. The model achieves state-of-the-art performance in motion generation and understanding tasks, leveraging a large-scale curriculum-pretrained foundation and a novel part-aware residual quantization for precise, efficient motion tokenization (Cao et al., 11 Aug 2025).

1. Controllability Challenges in VLMMs

Being-M0.5 was designed explicitly to address five major controllability failures present in previously published VLMMs:

1. Diversity of command response: Insufficient generation quality in response to varied natural language.
2. Pose initialization: Inability to generate motion from arbitrary initial poses.
3. Long-term sequence generation: Subpar performance in synthesizing temporally extended actions.
4. Generalization to unseen motions: Weakness in modeling actions not observed during training.
5. Granularity of control: Lack of fine-grained, part-level manipulation during motion synthesis.

These were resolved through a combination of multi-task, million-scale instruction-data pretraining and a purpose-built tokenizer that enables explicit limb and part-level control. Evaluation demonstrates top-tier accuracy on established and newly curated human-motion benchmarks while preserving real-time performance (≥20 FPS) (Cao et al., 11 Aug 2025).

2. HuMo100M: Foundational Multimodal Motion Dataset

Central to Being-M0.5’s development and evaluation is HuMo100M, the largest and most comprehensive multimodal human motion dataset to date. Its key components include:

• Scale and modality: Over 5 million 3D motion sequences, with sources spanning more than 20 million web videos and public datasets.
• Instructional diversity: 100 million annotated “instruction instances,” structured for five core task categories, encompassing both body-level and part-level tasking.
• Textual annotation: Three tiers—(i) global action captions, (ii) anatomically localized part descriptions, (iii) structured PoseScript representations.
• Sequence construction: Concatenation and learned interpolation for building long-form motion sequences.
• Multi-task instructional templates (editor's term):
  • Instruct-to-Motion (I2M)
  • Motion Prediction & In-Between (MPI)
  • Instruct-to-LongMotion (I2LM)
  • Instruct-to-Unseen (I2U) for held-out categories
  • Instruct-to-PartMotion (I2PM) for explicit limb control

This unprecedented dataset underpins both the learning of complex motion-linguistic associations and the attainment of precise controllability (Cao et al., 11 Aug 2025).

3. Model Architecture and Motion Tokenization

Being-M0.5 extends a LLaMA-2-chat 7B backbone with cross-modal encoders and discrete motion token generators:

• Visual encoder: 400M-param SigLIP backbone, with a 2-layer MLP projector exploiting a slow-fast temporal tokenization strategy.
• Motion tokenizer (“Q”): Maps continuous 3D motion sequences ($m_{1:T}\in\mathbb{R}^{T\times D}$) to discrete tokens, expanding the model vocabulary to include both standard language tokens and structured motion codes.
• Unified inference: Vision, language, and motion inputs are projected into the LLM’s multimodal space. Output is autoregressive, emitting either linguistic or motion tokens, context-conditioned.

Training is performed using next-token negative log-likelihood:

$$\mathcal{L}(\Theta) = -\sum_{j=1}^{L}\log P_\Theta(y_j \mid \mathcal X_Q, \hat y_{1:j-1})$$

This objective ensures unified, context-sensitive cross-modal sequence modeling, essential for both text-conditioned and vision-conditioned motion synthesis (Cao et al., 11 Aug 2025).

4. Part-Aware Residual Quantization (PRQ)

A central innovation of Being-M0.5 is the part-aware residual quantization (PRQ) scheme for motion representation:

• Anatomical decomposition: Each frame’s motion feature $m_i$ is split into five anatomically meaningful parts (left arm, right arm, left leg, right leg, torso).
• Independent quantization: For every part, residual quantization is applied through $K=4$ stacked codebooks ($b^k_{i,j}$):

$$b^k_{i,j}=\mathcal Q(r^k_{i,j}),\quad r^{k+1}_{i,j}=r^k_{i,j}-b^k_{i,j}$$

• Reconstruction: The quantized part codes are decoded and aggregated, averaging shared joints for final $\tilde m_i$.
• Loss formulation: Jointly minimizes

$$\mathcal L_{\rm PRQ} =\sum_{j=1}^5\|m_{j}-\tilde m_{j}\|_1 +\|m-\tilde m\|_1 +\beta\sum_{k=1}^K\sum_{j=1}^5\|r^k_{j}-\mathrm{sg}[b^k_{j}]\|_2^2$$

• Codebook efficiency: Partwise splitting, shared codebooks, and limited stacking prevent codebook explosion, supporting highly granular, limb-level control.

This enables precise manipulation and freezing of specific body parts during inference, providing dramatically enhanced controllability compared to prior approaches (Cao et al., 11 Aug 2025).

5. Real-Time Inference and Fine Control

Being-M0.5 achieves real-time performance and superior flexibility via:

• Frame-by-frame decoding: Each frame’s tokens are emitted as soon as computed, eliminating the need to wait for all quantization layers, sustaining ≥20 FPS on modern accelerators with large (7B-param) backbones.
• Direct feature representation: Uses the HuMo263 feature format—SMPL-based 6D joint rotations, root velocities, and explicit contact bits—removing the need for inverse kinematics post-processing.
• Slow-fast vision: Most frames are subsampled for heavy vision feature computation while retaining keyframes at full rate.
• Selective part control: Inference can freeze or update specific part code tokens, by masking token logits for unchanged parts, enabling robust, interactive modulation of generated motion.

These technical decisions ensure that Being-M0.5 can deploy in interactive, real-world settings where low latency is critical and nuanced, expert-level control is required (Cao et al., 11 Aug 2025).

6. Benchmark Performance and Efficiency

Comprehensive evaluations on nine motion generation and understanding benchmarks establish the model’s superiority:

Task / Benchmark	Being-M0.5	Baseline (ex)	Metric Comparison
HumanML3D (T2M)	FID 0.056, R@1 0.535	FID 0.191 (LMM)	ΔFID ↓, ΔR@1 ↑
HuMo-I2M (diverse instr.)	FID 0.148	FID 0.314 (MotionGPT)	ΔFID ↓
HuMo-Unseen (I2U)	FID 8.65	FID 43.28	ΔFID ↓ w/ dataset ↑
HuMo-I2PM (part control)	76% success	64% (ParCo)	ΔSuccess ↑

Full tables in the cited work detail consistent improvements in Frechet Inception Distance (FID), recall, and part-control metrics. Additionally, the model supports real-time inference speeds (20–28 FPS) on A100/H100 GPUs with 4-bit quantization, and tokenization efficiency of ≈100 tokens/sec (Cao et al., 11 Aug 2025).

7. Architectural Design Principles and Guidelines

Key technical insights distilled from the development and success of Being-M0.5 are:

• Dataset scale and granularity: Million-scale, multi-task datasets with detailed part-level annotations are foundational for controllable VLMMs.
• Tokenization and representation: Part-aware quantization expands representational capacity while keeping codebooks tractable.
• Performance-latency trade-off: The 7B LLM backbone balances linguistic understanding with practical inference latency; smaller models underfit while larger models incur impractical slowdowns.
• Training curriculum: A systematic curriculum—progressing from motion-text to vision-text-motion and finally instruction tuning—incrementally builds model capability.
• Direct feature pathways: Representing motion in rotation and velocity space (HuMo263) supports efficient, accurate generation avoiding external post-processors.

These principles offer a practical blueprint for future designs targeting high-fidelity, real-time, and fully controllable motion-generation models (Cao et al., 11 Aug 2025).