CLAW: Composable Language-Annotated Whole-body Motion Generation

Abstract: Training language-conditioned whole-body controllers for humanoid robots requires large-scale datasets pairing motion trajectories with natural-language descriptions. Existing approaches based on motion capture are costly and limited in diversity, while text-to-motion generative models produce purely kinematic outputs that are not guaranteed to be physically feasible. Therefore, we present CLAW, an interactive web-based pipeline for scalable generation of language-annotated whole-body motion data for the Unitree G1 humanoid robot. CLAW treats the motion modes of a kinematic planner as composable building blocks, each parameterized by movement, heading, speed, pelvis height and duration, and provides two browser-based interfaces -- a real-time keyboard mode and a timeline-based sequence editor -- for exploratory and batch data collection. A low-level whole-body controller tracks the planner's kinematic references in MuJoCo simulation, producing physically grounded trajectories recorded at 50Hz. Simultaneously, a deterministic template-based annotation engine generates diverse natural-language descriptions at multiple stylistic registers for every segment and for the full trajectory. We release the system as open source to support scalable generation of language-motion paired data for humanoid robot learning.

• The paper introduces a modular, planner-agnostic pipeline that synthesizes physically grounded, language-annotated motion for the Unitree G1 robot.
• It employs a four-process system integrating a browser-based frontend, a C++ controller, and the MuJoCo simulator to enable real-time, smooth motion sequencing.
• The methodology uses composable motion primitives and a deterministic, template-based language annotation engine to facilitate scalable, diverse data generation for language-conditioned control.

CLAW: Composable Language-Annotated Whole-body Motion Generation

Motivation and Contributions

The paper introduces dark_gray, a compositional, browser-based pipeline for large-scale generation of language-annotated, physically plausible whole-body motion data specifically targeting the Unitree G1 humanoid robot (2604.11251). The motivation arises from three fundamental bottlenecks observed in existing motion data acquisition paradigms: high cost and limited diversity in motion capture (MoCap), physically infeasible outputs in kinematics-only text-to-motion generative models, and the annotation cost when pairing natural language to each data sample. dark_gray addresses these by shifting to a pipeline where parameterized, composable motion primitives are sequenced in physics simulation, with automated, deterministic multi-style natural language annotation generated in tandem.

The primary contributions are:

• An open-source, planner-agnostic, interactive pipeline capable of physically grounded, language-annotated whole-body motion data synthesis at arbitrary scale.
• Two interactive user interfaces (keyboard mode and sequence editor) for both exploratory and systematic, reproducible data creation.
• A template-based annotation engine generating diverse, style-controlled, deterministic natural language descriptions for both segments and long-horizon trajectories.
• Modular, four-process system architecture suitable for extension and integration with new controllers and planners.

System Design and Architecture

The dark_gray pipeline comprises four loosely coupled processes: a browser-based frontend for human interaction, a protocol translation bridge (WebSocket–ZMQ), a C++ whole-body controller (for kinematic planning and real-time tracking), and the MuJoCo physics simulator (for providing physically realistic execution).

Figure 2: The dark_gray system architecture, detailing the interactions between backend controller, simulation, protocol bridge, and browser frontend.

The architecture is fully planner-agnostic; the frontend communicates structured, high-level commands to the bridge, which sequences these for the controller. The controller handles kinematic reference generation and feeds joint-level targets into the MuJoCo simulator. State feedback flows back throughout, supporting real-time telemetry and visualization. This modularity facilitates both rapid prototyping and systematic, scalable data collection.

Composable Motion Primitives and User Interfaces

dark_gray exposes 25 motion modes, grouped into Locomotion, Squat/Ground, Boxing, and Styled Walking, each parameterized by adjustable dimensions (movement, heading, speed, pelvis height, duration). Users sequence these via:

• Keyboard mode: Free-form, interactive control suitable for exploration and on-the-fly generation of novel motion combinations.

Figure 3: Screenshot of keyboard mode, emphasizing real-time configuration of movement, posture, and style parameters.

• Editor mode: GUI-based sequence editor, enabling reproducible, multi-segment trajectory design, ideal for batch generation and systematic data augmentation.

This compositional abstraction leverages smooth mode switching; transitions between primitives are handled within the planner using current state, with no explicit blending or post-processing required by the pipeline. Most mode pairs yield fluid transitions, though outlier mode switches (e.g., crawl to upright) can produce transient kinematic anomalies.

Figure 5: Examples illustrating both successful and failure cases in motion stitching between distinct behavioral primitives.

The sequence editor enables scalable augmentation through parameter replay—large numbers of semantically similar sequences can be programmatically instantiated by varying continuous parameters.

Language Annotation Engine

The pipeline systematically generates language annotations for every segment and full trajectory, based on the same motion-intent parameters that control the planner. This is achieved via a deterministic, template-driven generator supporting eight stylistic variants:

• Four registers: instruction (imperative), natural, narrative (third-person), and concise (keyword).
• Temporal grounding: each with/without explicit duration.

  Figure 1: Diverse whole-body motion generation capabilities, with each segment and full trajectory automatically annotated in multiple natural language styles.

Annotations exploit synonym banks for verbs, directions, connectives, and tempo adverbs, producing lexical diversity in a reproducible, seed-dependent manner. This method provides consistent, stage-aligned textual labels suitable for supervision of language-conditioned control models.

Figure 4: Example of language annotation for long-horizon, multi-stage trajectories, with per-segment and full-sequence descriptive variants.

The proposed annotation engine balances diversity and determinism, and the architecture allows easy extension to LLM-based generation should paraphrastic richness become a priority.

Experimental Findings

The pipeline supports several strong capabilities:

• Smooth composition of long-horizon, multi-behavior trajectories: Most motion primitive pairs transition fluidly, supporting continuous, complex motion design by simply sequencing primitives (Figure 4).
• Highly scalable, reproducible data generation: Editor recipes are replayable and parameterizable, enabling infinite data synthesis limited only by compute, not real-world capture logistics.
• Deterministic, diverse annotation aligned to motion: The annotation process is tightly coupled with trajectory segmentation, producing perfectly aligned ground truth with no manual effort.

  Figure 6: The data generation pipeline: from user intent to time-aligned trajectory and multi-style annotation output.

No explicit numerical benchmarking is provided in the paper, but the practical utility of infinite, fully aligned trajectory–text data is highlighted for training downstream language-conditioned control policies, where data scale and coverage directly impact policy generalization and robustness.

Implications and Future Directions

The practical implications extend beyond the specific Unitree G1 platform. dark_gray demonstrates a model for tasking high-DOF robots with language-anchored, compositional motion planning without dependence on MoCap or laborious manual annotation. This addresses critical challenges in scaling up RL- and imitation learning-based robotics, where diversity and alignment of supervision are limiting factors.

Theoretically, the approach decouples motion diversity from physical embodiment constraints (no retargeting artifacts), while modular annotation invites research into the balance between template-based and generative language conditioning for robotic learning.

Future research directions include:

• Incorporating LLM-based annotation for richer and less deterministic linguistic supervision.
• Expanding the set of motion primitives (including user-defined or learned skills), further increasing behavioral coverage.
• Automated filtering or correction of low-quality or physically implausible transitions to improve dataset consistency for large-scale learning.
• Extension to real-robot data collection via hardware-in-the-loop, supporting sim-to-real transfer.

Conclusion

dark_gray presents a scalable, modular system for generating whole-body humanoid motion data, paired with diverse, deterministic natural language annotation. Its compositional design, web-based interactivity, and planned extensibility position it as an enabling tool for the language-conditioned control community and for research at the intersection of robotics, interactive simulation, and multi-modal learning. The pipeline circumvents critical bottlenecks in motion diversity, physical feasibility, and labeling alignment, with clear potential for extension into both research and applied domains.