---
title: 'MotionBricks: Real-Time Generative Motion'
url: https://www.emergentmind.com/topics/motionbricks
type: topic
---

# MotionBricks: Real-Time Generative Motion

MotionBricks is a production-oriented generative motion framework that treats real-time interactive motion control as a systems problem rather than only a synthesis problem. It combines a large-scale modular latent generative backbone with “smart primitives,” a high-level control layer that converts navigation and object-interaction intent into unified keyframe constraints. In the reported configuration, a single model is trained on a dataset of over 350,000 motion clips, achieves a real-time throughput of 15,000 FPS with 2 ms latency, and is deployed both in a production-level Unreal Engine 5 demo and on the Unitree G1 humanoid robot [2604.24833].

## 1. Problem setting and design objectives

MotionBricks is motivated by a gap between generative motion research and production deployment. The framework argues that current text-to-motion, diffusion-based, and token-based systems can produce plausible motion offline, but that production systems still require properties they do not jointly satisfy: real-time generation, very large skill coverage, fine-grained multi-modal control, robustness under replanning, precise keyframes, and practical integration into engine and robotics stacks [2604.24833].

The framework identifies two central challenges. The first is **real-time scalability**: industry applications require one model to cover a vast repertoire of motions without quality collapse or impractical latency. The second is **integration**: production control must accommodate velocity commands, style selection, precise keyframes, and object-scene interaction in a single interface, whereas many prior systems are text- or tag-driven and therefore too coarse for runtime authoring [2604.24833].

This design goal is explicitly contrasted with traditional animation graphs and motion graphs. Those systems remain dominant because they provide deterministic transitions and explicit control, but they do so at substantial authoring cost. MotionBricks cites modern AAA pipelines with over 15,000 animations, 5,000 states, and nested graphs up to 12 levels deep, and presents itself as an attempt to replace much of that hand-authored control logic with a generative backbone plus reusable runtime primitives [2604.24833].

A useful conceptual shorthand is the framework’s own “brick” metaphor: downstream applications are intended to be assembled from modular control components rather than from large state machines. In that sense, MotionBricks is not merely a motion generator; it is a runtime behavior-construction framework.

## 2. Motion representation and modular latent backbone

MotionBricks represents each motion frame as
\[
(r_g, r_l, p, q, v, c),
\]
where \(r_g\) are global root values, \(r_l\) are local root values, \(p\) are joint positions excluding the projected root, \(q\) are joint rotations excluding the projected root, \(v\) are joint velocities, and \(c\) are contact labels [2604.24833]. Two design choices are especially significant. First, the representation keeps both local and global root information rather than canonicalizing heading. Second, joint rotations are represented in global coordinates, which the framework uses to support motions such as crawling and flipping where canonical root heading is ill-defined [2604.24833].

The generative backbone is modular. It consists of a tokenizer, a root module, a pose module, and a decoder. The tokenizer only discretizes pose:
\[
\left\{z^{t}_e\right\}_{t=1}^{T/4} = \mbox{enc}\left(\left\{p^t, q^t\right\}_{t=1}^T\right),
\]
so root dynamics are disentangled from pose latents [2604.24833]. The latent is then quantized with a structured multi-head codebook rather than a single large codebook:
\[
z_q^t =
\begin{Bmatrix}
z_{q,1}^t \\
z_{q,2}^t \\
\vdots \\
z_{q,K}^{t}
\end{Bmatrix},
\]
with each head independently choosing its nearest codebook entry [2604.24833]. This multi-head structure is one of the framework’s main scaling devices: it increases effective token capacity while degrading more gracefully when some heads are imperfect.

Above the tokenizer, the **root module** predicts motion timing and a coarse root trajectory in two stages. The **pose module** is the largest transformer and predicts latent pose tokens conditioned on local root, global root, and keyframe constraints. The decoder then reconstructs full motion from pose tokens, local root trajectory, and sparse pose constraints. The overall design is explicitly progressive: coarse timing and root behavior are estimated first, then pose detail is filled in, then the decoder refines the final sequence [2604.24833].

This architecture also adopts an in-betweening formulation rather than a monolithic prompt-to-sequence mapping. Missing constraints are replaced with learnable mask embeddings, allowing the same backbone to interpolate between arbitrary combinations of context frames, target root constraints, and sparse pose targets.

## 3. Smart primitives and the unified control interface

The second half of MotionBricks is the **smart primitive** layer. Smart primitives convert user or application commands into a unified keyframe representation
\[
\mathcal{T} = \{\mathcal{T}_1, \mathcal{T}_2, \mathcal{T}_3\},
\]
where \(\mathcal{T}_1\) are local root constraints, \(\mathcal{T}_2\) are global root constraints, and \(\mathcal{T}_3\) are pose constraints [2604.24833]. Because the backbone is trained to solve arbitrary constrained in-betweening problems, it can consume the same interface for locomotion, style transfer, and object interaction without downstream fine-tuning.

Two smart primitives are defined. **Smart locomotion** handles velocity and heading commands, including infeasible commands and style changes. Its first stage constructs a naive 1-second root target and then smooths it with a critically damped spring:
\[
r(t) = e^{-\gamma t}\left((r_0 - r_{g,1}) + (v_0 + \gamma(r_0 - r_{g,1}))t\right) + r_{g,1},
\]
yielding a spring-smoothed target \(r_{g,2}\) from current position \(r_0\), current velocity \(v_0\), and damping coefficient \(\gamma\) [2604.24833]. That trajectory is then refined by the root module and decoder into the final root path. Style control is handled by placing style keyframes—taken from short reference clips, authored poses, or 2D blendspaces—onto the root trajectory, relying on periodic replanning rather than explicit gait-phase alignment [2604.24833].

**Smart object** handles interaction with static and portable objects. Each smart object stores **intent keyframes** and an **interaction binding**. Keyframe sets carry a drop-frame attribute \(\tau\), where \(\tau=0\) is a hard constraint and \(\tau>0\) allows early transition by up to \(\tau\) frames, and each keyframe has a boolean rotation flag \(\omega\) indicating whether it should be rotated around the interaction pivot at runtime [2604.24833]. Interaction binding manages trigger detection, sockets and placement logic, and the anchoring transform used to adapt the same authored interaction to different object positions and orientations.

In the Unreal Engine 5 implementation, these primitives are realized as task actors. A locomotion task actor is always active on the character, scene objects expose smart-object task actors, and a thin StateTree governs high-level mode switching among locomotion, target pose sequence, and falling [2604.24833]. This is why the framework presents itself as plug-and-play: behaviors are composed by swapping primitives and keyframe sets, not by reauthoring the generative model.

## 4. Data, training, and inference characteristics

MotionBricks is trained and evaluated across four datasets [2604.24833]:

| Dataset | Scale | Notes |
|---|---:|---|
| 350k dataset | ~700 hours; 315,162 train clips; 35,018 test clips; 27 joints | Primary proprietary dataset |
| 70k dataset | 140 hours; 62,132 train clips; 35,018 test clips; 27 joints | Scale study |
| HumanML3D | 28.6 hours; 23,206 train clips after cleaning/segmentation; 2,578 test clips; 22 joints | Open benchmark |
| LaFAN1-G1 | 4.6 hours; 2,362 train clips; 262 test clips; 34 joints after retargeting and adding four end-effectors | Robotics benchmark |

All datasets are standardized to 30 FPS, and motion segments have length \(T\) randomly sampled from 12 to 64 frames in increments of 4 [2604.24833]. The tokenizer is trained with the VQ-VAE objective
\[
L= \sum_{t=0}^{T/4}\sum_{k=1}^K\|sg(z_{q,k}^t) - e_k\|^2_2 + \beta\|z_{q,k}^t - sg(e_k)\|^2_2,
\]
with running-mean codebook updates; the implementation also adds standard foot sliding and velocity losses during tokenizer training [2604.24833].

Training is staged and large scale. The tokenizer, root module, and pose module are trained on 4 nodes \(\times\) 8 GPUs = 32 GPUs for 2 million updates with Adam, learning rate \(5 \times 10^{-5}\), cosine schedule, 10k warmup steps, and minimum learning rate \(2 \times 10^{-6}\). On H100 hardware, the tokenizer takes about 7 days, the root module about 3 days, and the pose module about 7 days [2604.24833]. The released model sizes are 23.5M parameters for the tokenizer, 50M for the root module, and 150M for the pose module [2604.24833].

At inference time, MotionBricks does not regenerate motion every render frame. It maintains a motion buffer. When commands change or the buffer becomes low, the smart primitive emits keyframe constraints, the root module predicts frame count and root trajectory, the pose module predicts pose tokens, and the decoder reconstructs continuous motion. Generated frames are then buffered and consumed one by one [2604.24833]. This buffering strategy is central to the reported speed.

For deployment, the models are exported through ONNX and executed with TensorRT inside a native C++ Unreal Engine 5 plugin. The paper reports 15,000 FPS throughput with 2 ms latency on a desktop with an RTX 5090 GPU, and about 5 ms latency on Jetson Orin for humanoid deployment [2604.24833].

## 5. Empirical performance and applications

MotionBricks is evaluated with speed, distribution, human-evaluation, diversity, smoothness, physical-plausibility, and precision metrics. On the 350k dataset, its headline results are: MMD \(0.1056\), FID \(1.054\), win rate \(86.5\%\), human score \(4.06\), joint jitter \(3.38\), root jitter \(1.82\), foot skate \(0.003\), penetration \(0.008\), foot contact accuracy \(92.6\%\), target keyframe error \(0.076\), target root error \(0.023\), and reaching success \(99.6\%\), alongside the 15,000 FPS and 2 ms runtime figures [2604.24833].

Across other datasets, the framework continues to report state-of-the-art or near-state-of-the-art results. On LaFAN1-G1 it achieves MMD \(0.2835\), FID \(0.891\), and reaching success \(61.1\%\). On HumanML3D it reports FID \(0.914\) and reaching success \(98.9\%\). On Bones-70k it reports MMD \(0.1050\), FID \(1.090\), and reaching success \(99.6\%\) [2604.24833].

The ablations are structurally important. They show that the multi-head tokenizer continues to benefit from increased token capacity while a single-head baseline saturates; that dataset scaling improves MotionBricks more consistently than the baseline; and that root-pose disentanglement permits meaningful modification of root trajectories while preserving pose quality [2604.24833]. The framework also reports that discrete latent methods degrade slightly under overly frequent replanning and identifies a practical interval of 3–9 frames, with instant replanning on command changes [2604.24833].

The production demo is implemented in Unreal Engine 5 and covers locomotion, running, crouching, strafing, stealth locomotion, crawling, expressive movements, acrobatics, falling, sitting, climbing ledges, vaulting, object pickup, and object placement under a single neural backbone [2604.24833]. The paper further states that non-expert users can author smart navigation and smart objects in less than 10 minutes each using standard engine tools and sparse keyframe editing [2604.24833].

For robotics, MotionBricks is deployed on the Unitree G1 humanoid robot. The same high-level framework is used, but the generated kinematic motion is tracked by the controller from SONIC, running through TensorRT on Jetson Orin with replanning at 10 Hz or on command changes [2604.24833]. This establishes MotionBricks as a kinematic planning layer that can bridge animation and robotics, although the low-level physical execution is delegated to an external controller.

## 6. Interpretation, limitations, and relation to adjacent literatures

MotionBricks should be understood as a high-performance **kinematic motion planner and runtime control interface**, not as a full physically grounded control stack. The paper explicitly lists four limitations: dataset coverage remains incomplete even at 350k clips; smart-object execution currently assumes privileged access to clean object geometry and poses; the framework is a kinematic planner and may generate self-collisions or dynamically infeasible motions; and retargeting quality across morphologies remains a significant challenge [2604.24833].

These limits place MotionBricks in a broader systems landscape. In robotics terms, it occupies the planner side of the planner-controller divide. BRIC, by contrast, addresses the execution discrepancy between diffusion-based kinematic planners and RL-based physics controllers through online test-time adaptation and signal-space test-time guidance [2511.20431]. A plausible implication is that extending MotionBricks from high-quality kinematic target generation to fully robust physical execution would require controller-bridging mechanisms of that general kind.

The title can also be misleading outside motion synthesis. Despite the “brick” metaphor, MotionBricks is unrelated to literal LEGO or masonry generation. That separate literature includes geometry-conditioned buildable brick generation in BrickAnything [2605.26182], long-horizon robotic interlocking-brick assembly in BrickCraft [2605.07605], real-time snap-fit simulation in BrickSim [2603.16853], and graph-backed generative brick assembly in BrickNet [2604.22984]. MotionBricks shares with those systems a modular design philosophy, but its domain is humanoid motion rather than physical brick construction.

Taken as a whole, MotionBricks is best characterized as a modular latent motion system that attempts to replace brittle animation-graph complexity with a unified, keyframe-driven neural runtime. Its principal contribution is not only state-of-the-art generative quality, but the claim that large-scale real-time generative control can be packaged as an authorable production system.

Source: https://www.emergentmind.com/topics/motionbricks