MotionBricks: Real-Time Generative Motion

• MotionBricks is a generative motion framework that integrates a modular latent backbone with smart primitives to translate user commands into precise real-time motion.
• It achieves high performance with a throughput of 15,000 FPS and 2 ms latency, ensuring robust multi-modal control across diverse motion tasks.
• The system employs multi-head tokenization and a progressive in-betweening approach to bridge offline motion synthesis and production-level deployment.

MotionBricks is a production-oriented generative motion framework that treats real-time interactive motion control as a systems problem rather than only an overview problem. It combines a large-scale modular latent generative backbone with “smart primitives,” a high-level control layer that converts navigation and object-interaction intent into unified keyframe constraints. In the reported configuration, a single model is trained on a dataset of over 350,000 motion clips, achieves a real-time throughput of 15,000 FPS with 2 ms latency, and is deployed both in a production-level Unreal Engine 5 demo and on the Unitree G1 humanoid robot (Wang et al., 27 Apr 2026).

1. Problem setting and design objectives

MotionBricks is motivated by a gap between generative motion research and production deployment. The framework argues that current text-to-motion, diffusion-based, and token-based systems can produce plausible motion offline, but that production systems still require properties they do not jointly satisfy: real-time generation, very large skill coverage, fine-grained multi-modal control, robustness under replanning, precise keyframes, and practical integration into engine and robotics stacks (Wang et al., 27 Apr 2026).

The framework identifies two central challenges. The first is real-time scalability: industry applications require one model to cover a vast repertoire of motions without quality collapse or impractical latency. The second is integration: production control must accommodate velocity commands, style selection, precise keyframes, and object-scene interaction in a single interface, whereas many prior systems are text- or tag-driven and therefore too coarse for runtime authoring (Wang et al., 27 Apr 2026).

This design goal is explicitly contrasted with traditional animation graphs and motion graphs. Those systems remain dominant because they provide deterministic transitions and explicit control, but they do so at substantial authoring cost. MotionBricks cites modern AAA pipelines with over 15,000 animations, 5,000 states, and nested graphs up to 12 levels deep, and presents itself as an attempt to replace much of that hand-authored control logic with a generative backbone plus reusable runtime primitives (Wang et al., 27 Apr 2026).

A useful conceptual shorthand is the framework’s own “brick” metaphor: downstream applications are intended to be assembled from modular control components rather than from large state machines. In that sense, MotionBricks is not merely a motion generator; it is a runtime behavior-construction framework.

2. Motion representation and modular latent backbone

MotionBricks represents each motion frame as

$(r_g, r_l, p, q, v, c),$

where $r_g$ are global root values, $r_l$ are local root values, $p$ are joint positions excluding the projected root, $q$ are joint rotations excluding the projected root, $v$ are joint velocities, and $c$ are contact labels (Wang et al., 27 Apr 2026). Two design choices are especially significant. First, the representation keeps both local and global root information rather than canonicalizing heading. Second, joint rotations are represented in global coordinates, which the framework uses to support motions such as crawling and flipping where canonical root heading is ill-defined (Wang et al., 27 Apr 2026).

The generative backbone is modular. It consists of a tokenizer, a root module, a pose module, and a decoder. The tokenizer only discretizes pose: $\left\{z^{t}_e\right\}_{t=1}^{T/4} = \mbox{enc}\left(\left\{p^t, q^t\right\}_{t=1}^T\right),$ so root dynamics are disentangled from pose latents (Wang et al., 27 Apr 2026). The latent is then quantized with a structured multi-head codebook rather than a single large codebook: $$z_q^t = \begin{Bmatrix} z_{q,1}^t \ z_{q,2}^t \ \vdots \ z_{q,K}^{t} \end{Bmatrix},$$ with each head independently choosing its nearest codebook entry (Wang et al., 27 Apr 2026). This multi-head structure is one of the framework’s main scaling devices: it increases effective token capacity while degrading more gracefully when some heads are imperfect.

Above the tokenizer, the root module predicts motion timing and a coarse root trajectory in two stages. The pose module is the largest transformer and predicts latent pose tokens conditioned on local root, global root, and keyframe constraints. The decoder then reconstructs full motion from pose tokens, local root trajectory, and sparse pose constraints. The overall design is explicitly progressive: coarse timing and root behavior are estimated first, then pose detail is filled in, then the decoder refines the final sequence (Wang et al., 27 Apr 2026).

This architecture also adopts an in-betweening formulation rather than a monolithic prompt-to-sequence mapping. Missing constraints are replaced with learnable mask embeddings, allowing the same backbone to interpolate between arbitrary combinations of context frames, target root constraints, and sparse pose targets.

3. Smart primitives and the unified control interface

The second half of MotionBricks is the smart primitive layer. Smart primitives convert user or application commands into a unified keyframe representation

$$\mathcal{T} = \{\mathcal{T}_1, \mathcal{T}_2, \mathcal{T}_3\},$$

where $r_g$0 are local root constraints, $r_g$1 are global root constraints, and $r_g$2 are pose constraints (Wang et al., 27 Apr 2026). Because the backbone is trained to solve arbitrary constrained in-betweening problems, it can consume the same interface for locomotion, style transfer, and object interaction without downstream fine-tuning.

Two smart primitives are defined. Smart locomotion handles velocity and heading commands, including infeasible commands and style changes. Its first stage constructs a naive 1-second root target and then smooths it with a critically damped spring: $r_g$3 yielding a spring-smoothed target $r_g$4 from current position $r_g$5, current velocity $r_g$6, and damping coefficient $r_g$7 (Wang et al., 27 Apr 2026). That trajectory is then refined by the root module and decoder into the final root path. Style control is handled by placing style keyframes—taken from short reference clips, authored poses, or 2D blendspaces—onto the root trajectory, relying on periodic replanning rather than explicit gait-phase alignment (Wang et al., 27 Apr 2026).

Smart object handles interaction with static and portable objects. Each smart object stores intent keyframes and an interaction binding. Keyframe sets carry a drop-frame attribute $r_g$8, where $r_g$9 is a hard constraint and $r_l$0 allows early transition by up to $r_l$1 frames, and each keyframe has a boolean rotation flag $r_l$2 indicating whether it should be rotated around the interaction pivot at runtime (Wang et al., 27 Apr 2026). Interaction binding manages trigger detection, sockets and placement logic, and the anchoring transform used to adapt the same authored interaction to different object positions and orientations.

In the Unreal Engine 5 implementation, these primitives are realized as task actors. A locomotion task actor is always active on the character, scene objects expose smart-object task actors, and a thin StateTree governs high-level mode switching among locomotion, target pose sequence, and falling (Wang et al., 27 Apr 2026). This is why the framework presents itself as plug-and-play: behaviors are composed by swapping primitives and keyframe sets, not by reauthoring the generative model.

4. Data, training, and inference characteristics

MotionBricks is trained and evaluated across four datasets (Wang et al., 27 Apr 2026):

Dataset	Scale	Notes
350k dataset	~700 hours; 315,162 train clips; 35,018 test clips; 27 joints	Primary proprietary dataset
70k dataset	140 hours; 62,132 train clips; 35,018 test clips; 27 joints	Scale study
HumanML3D	28.6 hours; 23,206 train clips after cleaning/segmentation; 2,578 test clips; 22 joints	Open benchmark
LaFAN1-G1	4.6 hours; 2,362 train clips; 262 test clips; 34 joints after retargeting and adding four end-effectors	Robotics benchmark

All datasets are standardized to 30 FPS, and motion segments have length $r_l$3 randomly sampled from 12 to 64 frames in increments of 4 (Wang et al., 27 Apr 2026). The tokenizer is trained with the VQ-VAE objective

$r_l$4

with running-mean codebook updates; the implementation also adds standard foot sliding and velocity losses during tokenizer training (Wang et al., 27 Apr 2026).

Training is staged and large scale. The tokenizer, root module, and pose module are trained on 4 nodes $r_l$5 8 GPUs = 32 GPUs for 2 million updates with Adam, learning rate $r_l$6, cosine schedule, 10k warmup steps, and minimum learning rate $r_l$7. On H100 hardware, the tokenizer takes about 7 days, the root module about 3 days, and the pose module about 7 days (Wang et al., 27 Apr 2026). The released model sizes are 23.5M parameters for the tokenizer, 50M for the root module, and 150M for the pose module (Wang et al., 27 Apr 2026).

At inference time, MotionBricks does not regenerate motion every render frame. It maintains a motion buffer. When commands change or the buffer becomes low, the smart primitive emits keyframe constraints, the root module predicts frame count and root trajectory, the pose module predicts pose tokens, and the decoder reconstructs continuous motion. Generated frames are then buffered and consumed one by one (Wang et al., 27 Apr 2026). This buffering strategy is central to the reported speed.

For deployment, the models are exported through ONNX and executed with TensorRT inside a native C++ Unreal Engine 5 plugin. The paper reports 15,000 FPS throughput with 2 ms latency on a desktop with an RTX 5090 GPU, and about 5 ms latency on Jetson Orin for humanoid deployment (Wang et al., 27 Apr 2026).

5. Empirical performance and applications

MotionBricks is evaluated with speed, distribution, human-evaluation, diversity, smoothness, physical-plausibility, and precision metrics. On the 350k dataset, its headline results are: MMD $r_l$8, FID $r_l$9, win rate $p$0, human score $p$1, joint jitter $p$2, root jitter $p$3, foot skate $p$4, penetration $p$5, foot contact accuracy $p$6, target keyframe error $p$7, target root error $p$8, and reaching success $p$9, alongside the 15,000 FPS and 2 ms runtime figures (Wang et al., 27 Apr 2026).

Across other datasets, the framework continues to report state-of-the-art or near-state-of-the-art results. On LaFAN1-G1 it achieves MMD $q$0, FID $q$1, and reaching success $q$2. On HumanML3D it reports FID $q$3 and reaching success $q$4. On Bones-70k it reports MMD $q$5, FID $q$6, and reaching success $q$7 (Wang et al., 27 Apr 2026).

The ablations are structurally important. They show that the multi-head tokenizer continues to benefit from increased token capacity while a single-head baseline saturates; that dataset scaling improves MotionBricks more consistently than the baseline; and that root-pose disentanglement permits meaningful modification of root trajectories while preserving pose quality (Wang et al., 27 Apr 2026). The framework also reports that discrete latent methods degrade slightly under overly frequent replanning and identifies a practical interval of 3–9 frames, with instant replanning on command changes (Wang et al., 27 Apr 2026).

The production demo is implemented in Unreal Engine 5 and covers locomotion, running, crouching, strafing, stealth locomotion, crawling, expressive movements, acrobatics, falling, sitting, climbing ledges, vaulting, object pickup, and object placement under a single neural backbone (Wang et al., 27 Apr 2026). The paper further states that non-expert users can author smart navigation and smart objects in less than 10 minutes each using standard engine tools and sparse keyframe editing (Wang et al., 27 Apr 2026).

For robotics, MotionBricks is deployed on the Unitree G1 humanoid robot. The same high-level framework is used, but the generated kinematic motion is tracked by the controller from SONIC, running through TensorRT on Jetson Orin with replanning at 10 Hz or on command changes (Wang et al., 27 Apr 2026). This establishes MotionBricks as a kinematic planning layer that can bridge animation and robotics, although the low-level physical execution is delegated to an external controller.

6. Interpretation, limitations, and relation to adjacent literatures

MotionBricks should be understood as a high-performance kinematic motion planner and runtime control interface, not as a full physically grounded control stack. The paper explicitly lists four limitations: dataset coverage remains incomplete even at 350k clips; smart-object execution currently assumes privileged access to clean object geometry and poses; the framework is a kinematic planner and may generate self-collisions or dynamically infeasible motions; and retargeting quality across morphologies remains a significant challenge (Wang et al., 27 Apr 2026).

These limits place MotionBricks in a broader systems landscape. In robotics terms, it occupies the planner side of the planner-controller divide. BRIC, by contrast, addresses the execution discrepancy between diffusion-based kinematic planners and RL-based physics controllers through online test-time adaptation and signal-space test-time guidance (Lim et al., 25 Nov 2025). A plausible implication is that extending MotionBricks from high-quality kinematic target generation to fully robust physical execution would require controller-bridging mechanisms of that general kind.

The title can also be misleading outside motion synthesis. Despite the “brick” metaphor, MotionBricks is unrelated to literal LEGO or masonry generation. That separate literature includes geometry-conditioned buildable brick generation in BrickAnything (Ni et al., 25 May 2026), long-horizon robotic interlocking-brick assembly in BrickCraft (Yu et al., 8 May 2026), real-time snap-fit simulation in BrickSim (Wen et al., 17 Mar 2026), and graph-backed generative brick assembly in BrickNet (Kulits et al., 24 Apr 2026). MotionBricks shares with those systems a modular design philosophy, but its domain is humanoid motion rather than physical brick construction.

Taken as a whole, MotionBricks is best characterized as a modular latent motion system that attempts to replace brittle animation-graph complexity with a unified, keyframe-driven neural runtime. Its principal contribution is not only state-of-the-art generative quality, but the claim that large-scale real-time generative control can be packaged as an authorable production system.