Cross-Embodiment Action Codebook

• Cross-Embodiment Action Codebook is a shared representation layer enabling behavior transfer across robots with distinct kinematics and control interfaces.
• It employs diverse methods including discrete vector-quantized tokens, language-defined actions, and continuous latent spaces to decouple intent from execution.
• The framework improves transfer efficiency by adapting shared action semantics to robot-specific decoders through progress-aware alignment and adversarial invariance.

Searching arXiv for papers on cross-embodiment action representations and codebooks. Cross-Embodiment Action Codebook denotes a shared representation layer that enables policies, world models, or action decoders to transfer behavior across robots with heterogeneous kinematics, control interfaces, and morphology. In the recent literature, the term covers both discrete and continuous constructions: discrete, embodiment-agnostic “action motifs” in MOTIF (Zhi et al., 14 Feb 2026); a vector-quantized universal action space of atomic behaviors in UniAct (Zheng et al., 17 Jan 2025); natural-language action strings in LAP (Zha et al., 11 Feb 2026); phase-manifold embeddings in PHASOR (Kim et al., 1 Jun 2026); geometry-aware continuous latents in OPFA (Mu et al., 15 Mar 2026); motion-focused latent tokens learned from human egocentric video (Xu et al., 17 Jun 2026); and continuous intent-token interfaces in BLM$_1$ (Tan et al., 28 Oct 2025). Across these formulations, the common objective is to decouple reusable action semantics from embodiment-specific actuation, so that “what to do” can be shared while “how to execute it” remains robot-dependent.

1. Definitions, scope, and problem formulation

The core problem is that each robot embodiment $e$ has its own action space $A_e$, such as joint torques, velocities, gripper commands, or end-effector controls, and a policy trained on source embodiments can produce physically infeasible actions on a target embodiment with different joint limits, link lengths, or controller interfaces (Zhi et al., 14 Feb 2026). The literature therefore frames a cross-embodiment action codebook as a single, embodiment-agnostic representation that different robots can interpret and execute without per-embodiment retraining, or with only lightweight adaptation (Zha et al., 11 Feb 2026).

In MOTIF, the codebook consists of “action motifs,” defined as “statistically significant trajectory subsequences that represent pure spatiotemporal patterns, independent of task semantics or robot embodiment” (Zhi et al., 14 Feb 2026). UniAct formalizes the analogous object as a discrete universal action space $U \in \mathbb{R}^{N \times D}$ whose entries encode generic atomic behaviors and can be translated back to heterogeneous actionable commands by embodiment-specific heads (Zheng et al., 17 Jan 2025). OPFA instead treats the codebook as a continuous Geometry-Aware Latent Representation, learned from forward-kinematics-derived hand or gripper geometry and decoded by a unified latent retargeting decoder without embodiment-specific tuning (Mu et al., 15 Mar 2026).

Other systems broaden the concept beyond explicit quantization. BLM$_1$ states that it does not use a discrete action token codebook; instead, it uses a continuous shared intent representation extracted from a frozen MLLM and compressed into a fixed set of $K$ intent tokens that condition a shared Diffusion Transformer policy across embodiments (Tan et al., 28 Oct 2025). PHASOR similarly defines a universal action embedding space induced by a phase manifold $P(\tau)$ plus pooled pose context, so that semantically similar actions across humanoids trace similar orbits in a shared space (Kim et al., 1 Jun 2026). This suggests that “action codebook” in current cross-embodiment work is best understood as a family of shared action interfaces rather than a single discrete-token design.

A recurrent distinction is between embodiment-invariant intent and embodiment-specific realization. Several papers describe the transferable object as atomic behaviors, spatiotemporal patterns, or motion intent, while delegating execution to robot-specific decoders, controllers, or policy heads (Zhi et al., 14 Feb 2026). The same division also appears in Demo-JEPA, which replaces action correspondence with target-compatible latent subgoals in a shared predictive representation space, thereby treating demonstrations as specifications of future goals rather than instructions for copying source actions (He et al., 20 May 2026).

2. Representational families

Current work organizes cross-embodiment action codebooks into several representational families.

Discrete vector-quantized codebooks appear explicitly in MOTIF, UniAct, and the motion-focused latent-action framework trained from egocentric video. MOTIF learns a VQ-VAE codebook $C=\{c_k\}_{k=1}^K$ over canonicalized short-horizon end-effector segments and assigns latent tokens by nearest-neighbor vector quantization (Zhi et al., 14 Feb 2026). UniAct learns a discrete universal action space with $N=256$ and $D=128$ in UniAct-0.5B, using Gumbel-Softmax to select universal actions conditioned on observation and goal (Zheng et al., 17 Jan 2025). The motion-focused human-video method learns an action codebook of size 16, represents each 1-second frame pair by 4 action tokens, and separates those motion tokens from a distinct background codebook of size 16 (Xu et al., 17 Jun 2026).

Language-defined codebooks are represented by LAP, which converts continuous end-effector actions into deterministic natural-language descriptions such as “move forward 5 cm; tilt forward 10 degrees; rotate clockwise 20 degrees; close gripper,” quantized to integer centimeters and degrees and expressed under a fixed coordinate convention (Zha et al., 11 Feb 2026). LAP argues that this removes embodiment-specific action tokenizers and aligns action supervision with the pretrained VLM’s native input-output distribution, yielding an embodiment-agnostic action codebook in text form (Zha et al., 11 Feb 2026).

Autoregressive token blocks with shared semantics appear in ET-VLA. There, the codebook is realized as a shared token vocabulary inherited from OpenVLA, with seven tokens per arm per timestep, fixed semantics corresponding to end-effector Cartesian position, orientation, and a gripper command, and a fixed ordering convention in the bimanual case (Li et al., 3 Nov 2025). The paper emphasizes that alignment is achieved implicitly through shared token semantics, Synthetic Continued Pretraining, and embodiment-specific decoders $e$0, rather than through explicit embedding-alignment losses (Li et al., 3 Nov 2025).

Continuous latent or intent spaces dominate several recent systems. BLM$e$1 uses a Perceiver to compress MLLM hidden states into a fixed-size set of continuous intent tokens $e$2 shared across embodiments (Tan et al., 28 Oct 2025). SCAR learns a continuous latent action $e$3 from visual transitions via an inverse dynamics model and conditions a forward dynamics model on those latents, with KL regularization and GRL-based invariance to suppress embodiment and environment leakage (Liu et al., 13 May 2026). Tenma normalizes state and action vectors into fixed canonical slots with masks and maps them into a shared latent space of embedding dimension $e$4, though it does not define an explicit discrete codebook by default (Davies et al., 15 Sep 2025).

Geometry-aware and morphology-aware spaces are central to OPFA and X-DiffVLA. OPFA learns a geometry-aware continuous codebook from reachable-state point clouds derived by forward kinematics and point sampling, and retargets latents through a universal decoder and fixed selection matrices (Mu et al., 15 Mar 2026). X-DiffVLA instead uses a unified diffusion action head over a unified action space sized to the maximum dimensionality across embodiments, with zero-padding, masking, embodiment descriptors, and soft prompts, and strengthens transfer by Embodiment Forcing and Morphological Tree Diffusion (Li et al., 24 May 2026).

State-centric and whole-body codebooks appear in HEX and PHASOR. HEX provides a humanoid-aligned universal state representation with nine body-part slots and uses that shared latent substrate to derive a canonical set of whole-body action primitives operable across heterogeneous humanoids (Bai et al., 9 Apr 2026). PHASOR’s codebook is the universal action embedding space induced by the phase manifold $e$5, where periodic structure is explicitly factorized into amplitudes, frequencies, and phase shifts, and a small amount of pooled pose context disambiguates non-periodic configuration (Kim et al., 1 Jun 2026).

3. Learning mechanisms and alignment strategies

A central technical issue is how a shared codebook is made invariant enough to transfer, while remaining informative enough to decode into executable actions.

MOTIF combines three ingredients. First, it canonicalizes short-horizon end-effector segments by expressing them in a frame anchored at the initial pose and normalizing by workspace, with motif inputs given by

$e$6

Second, it trains a VQ-VAE with encoder $e$7, learned codebook $e$8, and decoder $e$9, using

$A_e$0

with $A_e$1 (Zhi et al., 14 Feb 2026). Third, it explicitly enforces invariance with progress-aware alignment and an embodiment adversarial loss. Progress alignment uses a soft weight

$A_e$2

and a soft-weighted InfoNCE objective; embodiment invariance is imposed by a GRL-based discriminator over latent tokens (Zhi et al., 14 Feb 2026). The overall Stage I objective is

$A_e$3

with $A_e$4 and $A_e$5 (Zhi et al., 14 Feb 2026).

SCAR reaches a related objective through self-supervised inverse-forward dynamics rather than explicit action supervision. It learns latent actions $A_e$6 from visual transitions, regularizes the posterior toward a standard Gaussian with

$A_e$7

and adds GRL-based adversarial invariance

$A_e$8

yielding

$A_e$9

with $U \in \mathbb{R}^{N \times D}$0 and $U \in \mathbb{R}^{N \times D}$1 (Liu et al., 13 May 2026). This suggests an alternative route to a codebook: learn a continuous invariant latent first, then discretize it post hoc.

UniAct uses a discrete but differentiable selection mechanism. Its universal action policy is

$U \in \mathbb{R}^{N \times D}$2

and selection is trained with Gumbel-Softmax rather than a straight-through estimator, because the latter caused codebook collapse (Zheng et al., 17 Jan 2025). OPFA instead grounds alignment in geometry: forward kinematics and point sampling generate point clouds, KPConv extracts local features, a geometric transformer adds coordinate and semantic positional embeddings, and global pooling yields the shared latent $U \in \mathbb{R}^{N \times D}$3 (Mu et al., 15 Mar 2026).

PHASOR explicitly structures the latent space. For each body part $U \in \mathbb{R}^{N \times D}$4, motion is approximated by

$U \in \mathbb{R}^{N \times D}$5

with phase-circle embeddings

$U \in \mathbb{R}^{N \times D}$6

Human-pretrained periodic encoders are frozen and robots are anchored to that shared manifold through lightweight adapters plus pairwise, clip-level, speed, and shape consistency losses (Kim et al., 1 Jun 2026). Unlike purely geometric alignment, this anchoring uses motion periodicity as the invariant.

BLM$U \in \mathbb{R}^{N \times D}$7 and ZR-0 place the shared codebook above the action level. In BLM$U \in \mathbb{R}^{N \times D}$8, alignment is imposed by action standardization, a shared DiT, and a future-prediction cosine objective between DiT future tokens and MLLM-derived future intent tokens, while the MLLM backbone remains frozen during policy training (Tan et al., 28 Oct 2025). ZR-0 uses dense Embodied Chain-of-Thought to align perception, progress reasoning, planning, and sub-task decomposition inside the VLM; the DiT-based action expert then cross-attends only to prompt features, not generated reasoning tokens, so ECoT can be omitted at inference with no loss in performance (Li et al., 29 Jun 2026).

4. Decoding, control interfaces, and few-shot transfer

Once a shared codebook is learned, it must be grounded into embodiment-specific actions. The literature offers several decoder patterns.

A common design is a shared latent plus lightweight embodiment-specific heads. In MOTIF, a Perceiver-based predictor maps frozen DINOv2 and T5 features to continuous motif tokens,

$U \in \mathbb{R}^{N \times D}$9

which are quantized through the learned codebook and fused with state and noisy action tokens inside a DiT. The resulting motif-conditioned flow-matching policy optimizes

$_1$0

Few-shot transfer reuses $_1$1, $_1$2, and $_1$3 frozen, collects $_1$4 demonstrations on the target embodiment, and fine-tunes only the state encoder $_1$5, action encoder $_1$6, and action decoder (Zhi et al., 14 Feb 2026).

UniAct uses lightweight embodiment-specific MLP heads $_1$7 to translate universal action tokens back to executable controls,

$_1$8

and adaptation to a new robot freezes both the universal codebook and the universal extractor while training only a 4M-parameter head for about 10K steps on roughly 100 demonstrations (Zheng et al., 17 Jan 2025). This decoder-centric transfer pattern is echoed in OPFA, where the universal decoder predicts a universal joint vector $_1$9 and a fixed selection matrix $K$0 yields the embodiment-specific action $K$1 without embodiment-specific tuning (Mu et al., 15 Mar 2026).

LAP’s execution layer separates representation from runtime control. Although the VLM is trained to predict language-actions autoregressively, real-time execution uses a continuous action expert trained with flow matching and run at 25 Hz on an RTX 4090 (Zha et al., 11 Feb 2026). The paper states that a generic per-robot controller applies the predicted delta end-effector pose and gripper commands via inverse kinematics or an EE delta controller while reusing global quantile normalization and canonical frame conventions across robots (Zha et al., 11 Feb 2026).

ET-VLA and X-VLA instead preserve a shared backbone while adapting interface layers. ET-VLA uses a shared vocabulary, fixed seven-token block semantics per arm, and embodiment-specific decoders $K$2 that map discrete token blocks into continuous controls; Synthetic Continued Pretraining ensures the autoregressive model emits structurally valid fourteen-token bimanual sequences before real fine-tuning (Li et al., 3 Nov 2025). X-VLA uses a shared continuous end-effector action space with xyz, Rotate6D orientation, and binary gripper state, while embodiment-specific soft prompts and minimal input/output projections absorb hardware and camera heterogeneity with only about $K$3 of the total parameters unshared (Zheng et al., 11 Oct 2025).

Several papers push the decoder toward planning or world modeling rather than direct inverse mapping. Demo-JEPA translates source visual motion into target-compatible latent subgoals $K$4 in a shared JEPA latent and then plans under target dynamics using CEM to minimize a latent discrepancy objective (He et al., 20 May 2026). The embodiment-equivariant VLA framework proposes an analytical decoder $K$5 that maps configuration-invariant camera-frame relative actions into embodiment-specific base-frame commands while guaranteeing equivariance under frame changes (Chen et al., 18 Sep 2025). This suggests a broader principle: the “decoder” of a codebook can be a controller, inverse-kinematics stack, planner, or diffusion policy, not merely an MLP.

5. Empirical evidence and comparative behavior

Recent results indicate that cross-embodiment codebooks improve transfer, robustness, or data efficiency, though the gains depend on the representation family and evaluation regime.

MOTIF reports that it “significantly outperforms strong baselines in few-shot transfer scenarios by 6.5% in simulation and 43.7% in real-world settings,” with simulation results including 1-shot Transfer SR of 36.00% versus 33.33% for TTO, 21.67% for GR00T N1, and 10.00% for HPT, and 5-shot Transfer SR of 54.33% versus 45.67% for TTO and 35.00% for GR00T N1 (Zhi et al., 14 Feb 2026). Real-world Transfer Avg is 67.50% versus 23.75% for Diffusion Policy and 21.25% for GR00T N1 (Zhi et al., 14 Feb 2026). Ablations show that removing kinematic canonicalization drops transfer SR by 10.33%, removing progress-aware alignment by 4.66%, removing adversarial invariance by 2.66%, and removing motif guidance degrades 1-shot transfer by 5.33% (Zhi et al., 14 Feb 2026).

LAP reports over 50% average zero-shot success across three previously unseen embodiments and six real-world manipulation tasks, describing roughly a 2x improvement over the strongest prior VLAs under identical architectures and data mixtures (Zha et al., 11 Feb 2026). It also reports lower held-out prediction error on unseen embodiments, with best error 0.151 versus 0.168 for $K$6-replicated and 0.189 for $K$7-replicated, and states that all tested open-source VLAs collapse to near-zero success on unseen robots (Zha et al., 11 Feb 2026).

UniAct shows that a 0.5B model can outperform 14X larger embodied foundation models in some evaluations, with LIBERO gains of +17.2% average accuracy over OpenVLA-7B and +33.6% over Octo, and reports that at least 40% of the 256 universal actions decode to semantically consistent behaviors across disparate robots, viewpoints, and sim-to-real gaps (Zheng et al., 17 Jan 2025). OPFA reports that cross-embodiment co-training can improve success rates by more than 50% compared to single-source training, and that adding only eight demonstrations from a new embodiment can reach performance comparable to a well-trained model with 72 demonstrations (Mu et al., 15 Mar 2026).

For continuous latent approaches, SCAR improves future-prediction metrics relative to shared latent baselines on both Procgen and Robotwin and reduces embodiment leakage, while sequence-level action-to-latent adaptation with fine-tuning can surpass shared ground-truth action conditioning on Franka (Liu et al., 13 May 2026). PHASOR reports strong cross-embodiment retrieval, with R@1 of 90.3 for human-to-robot, 90.5 for robot-to-human, and 84.8 for robot-to-robot under its best “Soft Coarse+Fine” alignment setup, outperforming both an unstructured MLP baseline and a VQ codebook baseline (Kim et al., 1 Jun 2026).

Whole-body and high-DoF cases show the same trend. HEX reports 79.8% average success across seven in-distribution real-robot tasks, exceeding GR00T N1.5 at 70.2% and $K$8 at 71.8%, and 61.8% under eight distribution-shift variants, surpassing $K$9 at 44.3% and GR00T N1.5 at 41.0% (Bai et al., 9 Apr 2026). ZR-0 reports 97.8% average success on LIBERO, 88.70%/87.98% on RoboTwin 2.0 Clean/Randomized, and 69.3% on RoboCasa GR-1 Tabletop, while ablating ECoT lowers LIBERO average from 97.8% to 95.7% (Li et al., 29 Jun 2026).

6. Limitations, controversies, and open directions

Despite consistent progress, the literature is explicit about remaining limits. Large kinematic disparities, underactuation, reachability mismatches, and contact-rich timing can still reduce executability of a shared code on a new embodiment (Zhi et al., 14 Feb 2026). LAP notes that chunked net deltas have not been evaluated at scale for fine-grained deformable manipulation or force/torque regulation, and that unusual interfaces such as underactuated hands may require additional low-level controllers even if the codebook itself remains agnostic (Zha et al., 11 Feb 2026). BLM$P(\tau)$0 observes that continuous intent spaces may be less interpretable than symbolic codebooks and that PlaceSphere on WidowX AI achieves only 2%, illustrating morphology sensitivity (Tan et al., 28 Oct 2025).

Interpretability versus precision is an active trade-off. Discrete or language-grounded codebooks are easier to inspect, cluster, or retrieve, but may sacrifice fine control resolution or rely on quantization choices. Continuous spaces preserve detail and often integrate naturally with diffusion or world models, but they can be harder to audit or stabilize. The motion-focused latent-action paper notes that a small discrete action codebook is effective for intent but insufficient for the most fine-grained manipulation requiring high-precision control, and proposes future work on multi-scale latent representations (Xu et al., 17 Jun 2026). PHASOR similarly performs best on rhythmic, locomotion-like behavior, while highly aperiodic or reactive tasks are less naturally captured by a periodic phase prior (Kim et al., 1 Jun 2026).

Another unresolved issue is whether alignment should happen at the action, state, intent, or reasoning level. Demo-JEPA argues that action-level correspondence is brittle and instead translates demonstrations into target-compatible latent goals (He et al., 20 May 2026). ZR-0 moves the shared interface upward into dense embodied reasoning, effectively turning chain-of-thought supervision into a semantic motor vocabulary (Li et al., 29 Jun 2026). X-Diffusion proposes a different interpretation altogether: the forward diffusion hierarchy itself acts as an implicit codebook, where high-noise levels encode coarse, embodiment-invariant guidance primitives and low-noise levels encode embodiment-specific execution details (Pace et al., 6 Nov 2025). This suggests that future codebooks may be hierarchical across semantic level, time scale, and control precision rather than purely discrete or continuous.

Several extension directions recur across papers: hierarchical or compositional motif libraries; nonparametric or language-conditioned codebooks; stronger domain invariance such as invariant risk minimization; integration of force or tactile signals; explicit planner layers over retrieved primitives; and hybrid systems that combine discrete high-level codebooks with continuous low-level refinement (Zhi et al., 14 Feb 2026). A plausible implication is that the field is converging toward multi-layer action interfaces: a shared semantic or geometric substrate, an intermediate transferable action representation, and an embodiment-specific execution module. In that sense, the cross-embodiment action codebook is becoming less a single module than an organizing principle for how robotic policies partition transferable structure from morphology-specific control.