CrossBFM: Distilling a Shared Latent Behavior Space Across Humanoid Embodiments

Abstract: Behavior Foundation Models (BFMs) give humanoids a promptable policy over a latent behavior space, enabling one single vector to represent a motion to imitate, a pose to reach, or a reward to maximize. Forward-Backward representations successfully produce such spaces, but at the cost of hundreds of GPU-hours for a single robot. Moreover, when the training process is repeated for a second robot, it produces a second space unrelated to the first, resulting in embodiment-specific latents that do not unify or transfer. We address these problems with CrossBFM, treating the latent space as the transferable asset for various embodiments. As retargeting provides frame-level cross-embodiment correspondence, we propose a unified encoder architecture with no robot-specific parameters for distilling the behavior space to address all training embodiments simultaneously in less than a GPU-hour. Following this encoder, latent-conditioned trackers turn the distilled latent into whole-body control in a conventional PPO training manner in just 10 more GPU-hours. On three distilled humanoids, all three prompting modes transfer: motion tracking with latent-conditioned policy losing only $0.025$ rad to its joint-conditioned counterpart, smooth goal reaching between poses with no falls, and reward optimization for all $41$ reward prompts. Our experiments further reveal that 1) regressing the encoder on a quarter of the motion corpus costs only $5\%$ of tracking performance and 2) training the encoder on a subset of robots and evaluating on an unseen one recovers up to $89\%$ of the tracking performance of seen robots, demonstrating cross-embodiment generalization to morphologically similar robots. We also verify the pipeline on real robots across all three prompting modes and with flow-based generated latents. Project website: https://dotandung.github.io/crossbfm/

• The paper presents a method for transferring a shared latent behavior space across multiple humanoid embodiments by using a unified transformer encoder.
• Encoder training for the CrossBFM method takes only less than one GPU-hour. Despite its small size, the encoder offers exceptional agreement and tracking on different humanoids. When examined, every technique recalls the closest around 90% of the entirely educated embodied control.
• CrossBFM preserves reward-to-latent mappings, improving performance by approximately 2.35 times on average across three target robots. Under unseen motions, it accurately reconstructs the geometry of source space and sustains the performance

CrossBFM addresses a specific bottleneck in promptable humanoid control: Forward–Backward representations provide a value-functional latent behavior space in which a single vector can encode a motion, goal state, or reward, but the space is normally learned independently for each embodiment. Consequently, retraining is required for every new robot, and independently learned latent coordinates have no guaranteed semantic correspondence. The paper proposes to treat the latent space of a pretrained BFM as a transferable coordinate system and to distill it onto multiple humanoids through retargeted motion correspondence (2609.38087).

Problem formulation and central claim

The method starts from a BFM trained on the Unitree G1 and retains its backward map, which maps states or reward functions into latent vectors. In an FB representation, the backward map is the component that gives the latent its prompting semantics: a goal state, motion frame, or reward can be converted into a vector that conditions the downstream policy. CrossBFM therefore does not attempt to relearn the complete successor-measure factorization on each target robot. It regresses target-robot proprioception directly onto the frozen source backward-map outputs.

The key assumption is that temporally aligned retargeted motions constitute a sufficiently accurate correspondence oracle. If a source G1 frame and a target-robot frame are generated from the same motion and timestamp, the source latent is used as the target label. Under this construction, cross-embodiment transfer becomes supervised regression rather than target-side unsupervised RL, adversarial representation learning, or state-space alignment. The implication is substantial computationally: the paper reports that encoder distillation takes less than one GPU-hour, whereas training the source backward map requires approximately 10 GPU-hours and complete BFM pretraining can exceed 100 GPU-hours.

This approach differs from prior multi-embodiment policies whose shared representations primarily encode morphology or task-specific control variables. CrossBFM instead inherits a value-functional latent space. The intended result is that motion tracking, goal reaching, and reward optimization remain available after transfer, rather than requiring a separate shared representation for each task.

Unified cross-embodiment encoder

The encoder uses a fixed-width canonical representation of humanoid proprioception. It contains 33 canonical joint slots, eight key-body slots, root information, velocities, and binary presence masks, for a total input width of 163 dimensions. Missing joints and bodies are zero-padded and explicitly masked. Joint-name mappings, sign conventions, and robot-specific scale normalization are handled outside the learned network. Thus, robot identity enters through configuration-level index maps rather than robot-specific parameters.

A four-layer pre-LayerNorm transformer processes a 64-frame window, corresponding to approximately 2.13 seconds at 30 Hz, and produces one 256-dimensional latent per frame. The output is radially projected onto the source BFM’s latent sphere. Training minimizes framewise cosine distance to the frozen source latent. The projection is incorporated into the computational graph rather than applied only after prediction, preventing the encoder from producing vectors outside the manifold on which the latent-conditioned tracker was trained.

The architectural choices are functionally important. The authors report that post-LayerNorm variants collapsed to an approximately constant latent, which cosine regression could not recover. The unified model has 3.27 million parameters, compared with 38.5 million parameters for three separate 512-width per-robot encoders. Despite its smaller size, the unified encoder achieves comparable source-latent alignment while producing substantially higher cross-robot agreement: pairwise agreement increases from approximately 0.867–0.913 for independently trained encoders to 0.938–0.965 for the unified encoder (2609.38087). This supports the claim that joint training is not merely a parameter-sharing device; it imposes a useful consistency constraint on the transferred coordinate system.

The paper also evaluates causal attention. Relative to bidirectional attention, causal attention reduces validation cosine alignment on the full corpus from 0.8695 to 0.8420, but closed-loop joint error slightly improves from 0.2068 to 0.2043 rad. With only 25% of the corpus, causal and bidirectional alignment are nearly identical, while causal attention achieves lower tracking error. This result matters for deployment because the bidirectional encoder uses future reference frames, whereas causal attention supports online reference streaming without a measurable degradation in closed-loop performance.

Latent-conditioned control and prompt transfer

After distillation, each target robot receives a separate PPO tracker conditioned on the distilled latent and current proprioception. Unlike the joint-conditioned baseline, the actor does not receive the per-frame target joint configuration. It must infer the behavior from the latent vector. The critic remains privileged during simulation and receives future reference information, but these observations are not available at deployment.

The experiments use three target robots: Inhouse M3, Booster T1, and Fourier N1. They differ in joint count, topology, body count, wrist structure, and waist configuration. The source is a G1 BFM with a 256-dimensional latent space. Across the three target robots, latent-conditioned motion tracking remains close to joint-conditioned tracking:

Robot	Joint-conditioned MAE	Latent-conditioned MAE	Difference
M3	0.1777 rad	0.2024 rad	+0.0247 rad
T1	0.1844 rad	0.1901 rad	+0.0057 rad
N1	0.1362 rad	0.1391 rad	+0.0029 rad

The largest degradation is only 0.0247 rad. This is a strong result because the joint-conditioned policies receive the exact frame-level joint target, whereas the latent-conditioned policies receive only a compact behavior representation. The practical implication is not simply comparable imitation accuracy: latent conditioning also permits interpolation between behavior states, enabling smooth goal reaching. Goal-reaching errors relative to the tracking baselines are at most 0.0321 rad, with no reported falls.

The three original BFM prompting modes transfer differently in implementation. Motion tracking maps the retargeted reference window through the encoder. Goal reaching encodes a window ending at the desired pose. Reward optimization requires an approximation to the target robot’s state distribution, because the target robot has not undergone the original FB training and therefore lacks the source model’s visitation distribution. CrossBFM constructs this distribution from retargeted target-robot data and forms a reward-weighted latent average.

The reward transfer mechanism includes percentile-based threshold transport. Absolute source-robot thresholds, such as a 1.0 m hand-height condition, can be physically inappropriate for a shorter or differently proportioned humanoid. CrossBFM maps source thresholds through empirical feature quantiles, preserving relative pose difficulty rather than absolute dimensions. This correction is not optional engineering: the paper shows that applying the source reward formulation with the source visitation distribution performs substantially worse than target-side reward construction. The target-side method improves normalized return by 3.50 times on M3, 1.14 times on T1, and 2.95 times on N1, with a mean improvement of 2.35 times.

The method succeeds across all 41 reward prompts in the reported suite. These prompts include standing, locomotion, rotation, arm raising, locomotion with arm configurations, combined rotation and arm behaviors, crouching, and sitting. The result supports the stronger claim that the distilled space preserves not only trajectory-level semantics but also the reward-to-latent structure required for zero-shot value-functional prompting.

Generalization to unseen motions

The encoder is evaluated on held-out motion clips from the same LAFAN behavior categories. On validation motions, latent inference increases key-body error by only 1.4–6.7 mm relative to the corresponding training or oracle conditions, while maintaining stable trajectories. The latent trajectories also preserve the geometry of the source space: held-out clips fall into the same behavioral clusters as their source counterparts rather than forming isolated or intermediate regions.

The paper reports an important distinction between latent reconstruction and control utility. Exact cosine agreement with the source latent is not always necessary for successful tracking. The tracker can tolerate some latent displacement if the predicted vector remains within a behavior cluster that the policy has learned to exploit. Consequently, geometric alignment is a useful diagnostic but not a complete proxy for downstream behavior.

Data scaling results reinforce this distinction. Reducing the distillation corpus to 25% increases closed-loop joint MAE from 0.2132 to 0.2241 rad, a 5% degradation despite a fourfold reduction in data. Performance begins to deteriorate sharply at 10%, where error reaches 0.2706 rad, although this remains substantially better than the random-latent control at 0.4411 rad. The result indicates that temporal continuity and behavioral coverage matter more than exhaustive frame-level coverage: one quarter of the corpus is sufficient when all behavior categories remain represented.

Generalization to unseen embodiments

Cross-embodiment generalization is tested by training the unified encoder on two robots and evaluating it on the third. The results show a clear dependence on morphological proximity.

Held-out robot	Validation cosine	Closed-loop MAE	Signal gap recovered
M3	0.6251	0.1829 rad	89.3%
N1	0.5861	0.2237 rad	55.5%
T1	0.2533	0.2886 rad	12.8%

The most important result is the M3 case: training on the other two robots recovers 89.3% of the tracking performance obtained by an encoder trained on all three. By contrast, the T1 case recovers only 12.8%, reflecting its smaller body, wrist-less configuration, and greater morphological distance from the available training embodiments.

These results qualify the paper’s cross-embodiment claim. The encoder generalizes effectively when the unseen robot lies near the training morphologies, but the evidence does not establish robust extrapolation across substantially different humanoid designs. Indeed, the reported cosine similarities show that representation-level proximity can be low even when control remains usable, while severe morphological mismatch degrades both alignment and closed-loop behavior. The relevant regime is therefore interpolation among related embodiments, not unrestricted embodiment independence.

Real-robot validation and flow-based latent generation

The authors deploy the latent-conditioned policies on real M3 and T1 hardware. On M3, the real-world motion-tracking MAE is 0.2206 rad for latent conditioning and 0.1985 rad for joint conditioning. Goal-reaching error is 0.2675 rad, compared with 0.2345 rad in simulation, and reward optimization achieves a normalized return of 0.562 versus 0.508 in simulation. The modest sim-to-real degradation for tracking and goal reaching, together with the comparable or higher reward result, indicates that the transferred latent interface survives deployment under the stated domain-randomization and hardware conditions.

CrossBFM also trains a single rectified-flow model over latent chunks. The generator is conditioned on behavior modes, latent history, and optionally frame-aligned motion features. It produces latent trajectories directly, without a robot-specific backward map or kinematic decoder at inference. Because all target trackers consume the same latent coordinate system, one generated latent stream can command multiple embodiments. The reported demonstrations include walking, running, and dancing, as well as the original prompt modes. However, the paper does not provide a comprehensive quantitative comparison between generated and manually inferred latents; the flow-based component is therefore best interpreted as a compatibility demonstration rather than a fully evaluated generative control system.

Limitations and open questions

The method depends centrally on the quality of motion retargeting. Retargeted frames must preserve both geometric correspondence and temporal alignment. The corruption experiments show that non-systematic Gaussian noise with standard deviation 0.05 is largely tolerable, and systematic geometric bias remains relatively harmless up to approximately four times the tested baseline. At eight times the baseline, alignment falls from 0.8916 to 0.7647 and tracking error increases from 0.1655 to 0.1913 rad; at sixteen times the baseline, performance becomes substantially worse.

Timing errors have a different structure. A global temporal offset can be recovered by a one-dimensional phase search, even for offsets as large as 32 frames. In contrast, per-clip timing offsets without a recoverable global pattern degrade training and inference. Thus, the “retargeting as correspondence oracle” formulation is valid only when synchronization errors are either small or identifiable.

The evaluation is also constrained to one source BFM, one LAFAN motion corpus, three target robots, and humanoids with broadly related morphology. The unseen-robot experiment shows that generalization is poor for the most dissimilar target, so the proposed fixed canonical vocabulary does not by itself solve topology mismatch. Further, reward transfer requires manually selected key-body features and percentile-based transformations; rewards involving detailed joint configurations are discarded. The reported success on 41 prompts therefore does not establish general transfer for arbitrary reward functions.

Finally, the target trackers still require approximately 10 GPU-hours per embodiment. CrossBFM removes the expensive target-side BFM training, but it does not eliminate target-specific policy training. The paper leaves open how much of this cost could be reduced through a single shared latent-conditioned control policy, how the latent geometry scales with richer motion data, and whether a substantially larger and more diverse embodiment set can convert the observed interpolation behavior into reliable generalization beyond nearby morphologies.

Conclusion

CrossBFM presents a supervised distillation procedure for transferring the latent coordinate system of a frozen humanoid BFM across embodiments. Its central technical device is a unified transformer encoder trained on frame-aligned retargeted motions, with a fixed canonical proprioceptive representation and explicit morphology masks. The resulting latent-conditioned trackers approach joint-conditioned tracking accuracy, support smooth goal reaching, preserve reward prompting across 41 tasks, operate on real robots, and can be driven by a shared flow-based latent generator.

The strongest evidence concerns computational efficiency and transfer among morphologically similar humanoids: encoder training takes under one GPU-hour, one quarter of the motion corpus retains near-full-data tracking quality, and held-out M3 transfer recovers 89.3% of the all-robot signal. The central qualification is equally clear: generalization deteriorates for substantially different embodiments, and the method remains dependent on accurate retargeting and target-specific tracker training. Within these boundaries, the paper establishes retargeted motion correspondence as a practical mechanism for inheriting the prompting semantics of a value-functional behavior space rather than relearning that space for every robot.