Robota-Embed: Unified VLA Embedding Model
- Robota-Embed is a unified vision-language-action (VLA) embedding model designed for retrieving task- and execution-stage-appropriate robot experiences, leveraging joint encodings of language, visual states, and actions.
- The model encompasses a retrieval mechanism that prioritizes predicting future video continuations and aligned action sequences, adapting to various source-specific action interfaces for consistent performance.
- Evaluations on the Robota-Embed-Bench, a meticulously curated benchmark, demonstrate remarkable improvements over baselines, especially in episode-level retrieval where Robota-Embed achieves 81.77% R@1 compared to 48.30% by the highest external benchmark.
Robota-Embed is a unified vision-language-action (VLA) embedding model for retrieving task- and execution-stage-appropriate robot experience. It jointly encodes language instructions, visual interaction states, and native actions in current-state queries, while encoding recorded future video continuations and aligned action sequences as retrieval targets. Its predictive retrieval objective is designed to make historical continuations useful for robot in-context learning and task-local context evolution. The system is evaluated on Robota-Embed-Bench, a benchmark containing 4,000 queries and an 8,000-entry gallery, and is applied through the Embed2ICL and Embed2Evolve frameworks (Liu et al., 14 Oct 2026).
1. Motivation and retrieval formulation
Robota-Embed addresses the problem of retrieving useful physical experience for a robot whose current observation may correspond to multiple stages of the same task. Language specifies task intent, vision describes the current interaction state, and native action conveys motion, execution phase, and control state. A retrieval method based only on semantic or visual similarity may therefore select an experience with the correct objects and instruction but an incompatible continuation.
The model represents a current query as
where is the instruction, is an interaction-region image at time , is the synchronized native action, and identifies the source domain. The corresponding retrieval target is a future continuation,
where is a four-second video clip and is its aligned action sequence. The future interval begins strictly after the accepted query timestamp.
This asymmetry distinguishes Robota-Embed from ordinary similarity learning. The query describes the robot’s present state, whereas the target describes what happened next. The model consequently learns current-to-future predictive retrieval rather than similarity between simultaneous descriptions.
Query and target encoders produce
where 0 is a source-specific action projector. Candidate targets are ranked by cosine similarity,
1
Because the vectors are normalized, inference is equivalent to maximum inner-product search:
2
The source-specific projectors allow a shared model to ingest heterogeneous native actions from human hand trajectories, single-arm robot controls, simulated robot actions, and dual-arm actions without requiring an identical action representation across sources.
2. Architecture, preprocessing, and training corpus
Robota-Embed is initialized from Cosmos3-Edge 4B, whose 28-layer Transformer has hidden width 2,048. The query and target endpoints share the Transformer, video tokenizer, and retrieval projection, but use separate residual endpoint adapters 3 and 4. The video VAE is frozen.
At action-token positions after the generation RMSNorm following layer 28, the model obtains fused multimodal states
5
The global endpoint vector is constructed by temporal averaging, a shared LayerNorm/projection 6, an endpoint adapter 7, and 8 normalization:
9
Actions are normalized per source using training-partition coordinate quantiles. They are not clipped and are zero-padded to 64 dimensions. The native action interfaces are:
| Source | Action interface |
|---|---|
| EgoDex | 57 dimensions at 30 Hz |
| DROID | 10 dimensions at 15 Hz |
| LIBERO | 10 dimensions at 10 Hz |
| RoboMIND–Franka-dual | 20 dimensions at 10 Hz |
Queries use the instruction, synchronized current action, and a SAM3-localized interaction crop. Targets use the instruction, 16 video frames from the next four seconds, and native-rate future actions. SAM3 proposals are filtered by detection score, image coverage, and zoom ratio. Qwen3-VL-8B-Instruct verifies that the crop contains the correct, visible, separable, and sufficiently unoccluded interaction object. If verification fails, an earlier synchronized timestamp, crop, and action are resampled.
The final filtered training corpus contains 1,695,684 current-state/future-continuation anchors:
| Source | Anchors |
|---|---|
| EgoDex | 1,345,160 |
| DROID | 26,563 |
| LIBERO | 136,671 |
| RoboMIND–Franka-dual | 187,290 |
| Total | 1,695,684 |
Training and evaluation partitions are disjoint at the episode, session, and content-identity levels, although task, object, and embodiment categories may recur. Sources are sampled equally during optimization despite their unequal numbers of filtered anchors.
3. Predictive objective and negative construction
The training objective is
0
The main alignment term is bidirectional masked InfoNCE. The exact future clip 1 is the positive for query 2. A different clip identifier is not automatically treated as a negative; only candidates with explicit semantic conflicts are included in the denominator. For a batch 3 and valid conflict set 4,
5
The reverse target-to-query term is computed independently, and
6
The global batch contains 512 entries before masking, while each denominator retains only the exact positive and valid semantic negatives. The inherited temperature 7 is optimized within 8.
Semantic conflicts include task conflicts, object conflicts, action conflicts, and joint conflicts. Task conflicts preserve object roles while changing operation, relation, or direction. Object conflicts preserve the task relation while changing the interaction-object category. Action conflicts preserve task and objects while changing the current action primitive or future action phase. Joint conflicts change both task and objects. Action conflicts are constructed from discretized current-action primitives and future-motion descriptors, including direction, peak-motion timing, gripper state, and sampled statistics. Temporal overlaps with intersection-over-union above 9 are excluded as near duplicates.
The grounding term introduces text-only and image-only auxiliary queries. Text-only queries align with targets sharing task identity, while image-only queries align with targets sharing interaction-object identity. Standalone action-only alignment is deliberately disabled because a single current action generally does not uniquely determine a future trajectory.
Modality-swap and hard-negative hinge constraints encourage the fused representation to distinguish valid from conflicted inputs. For a swapped query 0 and positive target 1,
2
Margins are 3 for text swaps, 4 for image swaps, and 5 for action swaps. Additional hard-task, hard-object, and hard-action terms select the highest-similarity candidate within each conflict class. A fused-dominance term encourages the complete VLA query to outperform each unimodal query by a 6 margin.
The regularization term promotes centered, isotropic, non-collapsed embeddings through variance floors, participation-ratio rank constraints, and direct query and target centering penalties. Objective weights are: alignment 7; text auxiliary 8; image auxiliary 9; text/image/action swaps 0; hard task/object/action 1; fused dominance 2; isotropy 3; and query/target centering 4.
Optimization uses AdamW with 5, gradient clipping at 6, learning rates 7 for the Transformer and 8 for the head and adapters, weight decay 9, global batch size 512, 1,314 warm-up steps, cosine decay, and 26,275 total optimizer steps.
4. Robota-Embed-Bench and retrieval performance
Robota-Embed-Bench contains 4,000 held-out queries and a shared gallery of 8,000 candidates. Each of the four sources contributes 1,000 queries from 800 episodes. Each source contributes 1,000 paired future clips and 1,000 nearby same-episode clips, producing 2,000 gallery entries per source and 3,200 episodes in total.
At the clip level, only the uniquely paired future clip is relevant. At the episode level, every candidate from the same source-scoped episode is relevant. Episode metrics are averaged within episodes, then equally over the 800 episodes per source and the four sources. Retrieval uses cosine similarity without reranking.
Robota-Embed obtains:
| Evaluation level | R@1 | R@5 | R@10 | MRR |
|---|---|---|---|---|
| Clip | 47.33% | 91.90% | 96.95% | 67.05% |
| Episode | 81.77% | 94.91% | 98.06% | 86.96% |
The difference between clip-level and episode-level retrieval indicates that identifying the correct episode is easier than identifying the exact continuation. Robota-Embed’s episode-level versus clip-level R@1 is 87.06% versus 36.70% on EgoDex and 99.88% versus 74.10% on DROID.
The strongest action-free external comparison is Gemini Embedding 2:
| Model | Clip R@1 | Episode R@1 |
|---|---|---|
| Gemini Embedding 2 | 25.24% | 48.30% |
| Omni-Embed-Nemotron-3B | 24.5% | 48.6% |
| Robota-Embed | 47.33% | 81.77% |
Robota-Embed exceeds Gemini Embedding 2 by 22.09 percentage points at clip level and 33.47 percentage points at episode level. Other action-free baselines include CLIP ViT-L/14, Perception Encoder B16, InternVideo2, Qwen3-VL-Embedding-2B, VLM2Vec-V2, and WeMM-Embedding-2B; their aggregate clip R@1 values range from 1.2% to 24.5%, compared with 47.3% for Robota-Embed.
Candidate-language masking reduces clip R@1 from 47.33% to 12.78%, indicating that target language contributes substantially. However, restricting the gallery to candidates sharing the same instruction leaves R@1 at 47.62%, showing that retrieval is not explained solely by language matching. Within same-instruction galleries, clip R@1 is 38.60% on LIBERO and 40.10% on RoboMIND-FD, compared with random rates of 1.90% and 0.24%, respectively.
5. Modality analysis and downstream in-context learning
Robota-Embed’s modality studies evaluate the contribution of native action, vision, and language. In matched DROID retraining, the full VLA model obtains 78.7% clip R@1. Removing action reduces this to 59.8%, a decrease of 18.9 percentage points. Removing vision produces 76.8%, a decrease of 1.9 points, while removing language produces 20.2%, the largest decrease.
With the final model and gallery fixed, masking only the query modality produces:
| Query input | Clip R@1 |
|---|---|
| Full VLA | 47.33% |
| No action | 20.08% |
| No vision | 23.00% |
| No language | 7.13% |
On the controlled DROID checkpoint, masking action reduces R@1 from 78.7% to 50.7%, whereas masking image reduces it to 60.9%. These results support the narrower conclusion that native action supplies execution-stage information beyond language and vision and is more consequential than vision for predictive continuation retrieval under the reported experiments. They do not imply that vision is unimportant: retraining without vision allows adaptation to the remaining modalities, whereas fixed-model vision masking introduces an input-distribution shift.
Retrieval training is essential. A frozen Cosmos3-Edge readout obtains 0.65% clip R@1 and 1.17% episode R@1. On DROID, head-only adaptation produces 13.8% R@1, whereas adapting the backbone and head produces 78.7%. Objective ablations yield 75.6% for alignment alone, 77.0% for alignment plus regularization, and 78.7% for the complete objective. The additional objectives primarily improve top-rank ordering, since R@10 is already 100% in the controlled DROID runs.
Embed2ICL
Embed2ICL uses retrieved continuations as policy context through training-free and training-based interfaces.
For frozen DreamZero-DROID, a retrieved four-second video initializes the policy’s rolling visual key-value cache. The policy is not updated. A GPT-5.6 Sol reranking agent may select among retrieved candidates using task relevance and completion evidence. The reference remains fixed for the evaluation trials, while live observations subsequently update the cache.
| Reference condition | Average success |
|---|---|
| No warm-up | 47.0% |
| Random reference | 36.0% |
| Omni-Embed top-1 | 43.3% |
| Omni-Embed + reranking | 51.0% |
| Robota-Embed top-1 | 50.3% |
| Robota-Embed + reranking | 56.3% |
The per-task Robota-Embed-plus-reranking results are 65% for Cube 0 Bowl, 24% for Can 1 Mug, and 80% for Banana 2 Bin. On the real YAM robot, the same type of visual-cache warm-up increases Banana 3 Bowl mean progress from 21.67 to 40.00 on a 0–100 staged-progress scale and increases success from 0% to 16.7% in 12 rollouts.
For Fast-WAM, the target encoder’s fused action-token states are converted into 32 cached Context Tokens. The global retrieval vector is used for search, while the context tokens are consumed by the policy. The selected context is projected to Fast-WAM’s width and appended to native text and proprioceptive conditioning. The embedding encoder remains frozen while Fast-WAM is trained to use the context.
On a five-task difficult LIBERO subset with 50 trials per task, official operation without in-context learning obtains 86.4% average success, random context tokens obtain 88.0%, and retrieved context tokens obtain 92.8%. The reported per-task changes are:
- Black bowl 4 plate: 5.
- Bowl 6 top drawer: 7.
- Wine bottle 8 rack: 9.
- Two mugs 0 plates: 1.
- Mug–pudding arrangement: 2.
Continuing the no-ICL policy for the same 21,700 updates still gives 86.4%, so the improvement is not explained simply by additional training. A smaller frozen-backbone side-branch integration obtains 88.0%, below the 92.8% achieved through native Context-Token prefilling. On real robots, Fast-WAM progress increases from 75.00 to 79.17 on Pick Fruit and from 66.67 to 87.50 on Split Cup. These progress values are partial credit over four independently scored substeps and are not directly comparable to binary LIBERO success.
6. Embed2Evolve, limitations, and scope
Embed2Evolve uses Robota-Embed to construct and refine a task-specific context harness around a frozen GPT-6 Astra policy. GPT-6 Astra, Robota-Embed, the robot controller, and the policy action interface remain fixed. The system evolves the task-local memory bank, retrieval count 3, refresh cadence 4, retained history length 5, and success and efficiency bonuses 6.
Completed development attempts, including successes and ordinary failures, contribute indexed continuations of up to four seconds. Each memory anchor contains RGB, commands or actions, robot state, feedback, and timestamps. Retrieval excludes the current seed and selects at most one anchor per source episode. For normalized query and anchor vectors 7, the ranking score is
8
where 9 is episode success and 0 is control ticks. Failed episodes receive no bonus or penalty and remain retrievable by similarity. The efficiency bonus rewards successful episodes that use fewer ticks.
Context is absent at decision zero. Retrieval begins at decision one, refreshes according to the selected cadence and when the active arm changes, and is cached between refreshes. GPT-6 Astra receives current RGB views, robot state, instruction, execution feedback, recent history, and retrieved historical context. It outputs end-effector targets and gripper commands, which are validated and executed through fixed inverse-kinematics and proportional-derivative control.
Development uses ten trials per round and selects lexicographically by success first:
1
Development stops after frontier stagnation or an eight-round cap. The selected memory and context recipe are then frozen for fresh test initializations.
Five ManiSkill tasks are evaluated with 20 fresh test initializations per condition:
| Task | GPT-6 Astra Direct | Random history | Evolve + Omni-Embed | Embed2Evolve |
|---|---|---|---|---|
| Poke Cube | 45% | 15% | 45% | 80% |
| Stack Cube | 10% | 20% | 20% | 60% |
| Lift Peg Upright | 30% | 40% | 55% | 65% |
| Pull Cube with Tool | 10% | 10% | 20% | 50% |
| Two-Robot Stack Cube | 25% | 15% | 25% | 85% |
| Average | 24% | 20% | 33% | 68% |
Embed2Evolve averages 0.518 million policy tokens per test rollout, compared with approximately 2.531 million for the independently evolved Omni-Embed-Nemotron-3B harness, corresponding to approximately 79.5% fewer tokens while achieving 68% versus 33% success. Against Direct, it uses 43.7% fewer tokens per rollout, falling from 920,979.76 to 518,216.15, and reduces estimated reference API cost by 41.8%, from $V_i^+$25.7980. On the 19 shared-success initializations, policy decisions decrease by 64.1% and observed client execution time decreases by 41.4%, corresponding to an average task-wise ratio of approximately $V_i^+$3. These wall-time figures include API waits and were not obtained under fully controlled concurrency; they are therefore observed efficiency measurements rather than isolated causal speedups.
The reported limitations include the fixed four-second target horizon, dependence on source-specific action projectors, and limited embodiment diversity. Action interfaces and the ManiSkill endpoint-error action proxy are not uniformly native across downstream settings. Episode-level success does not establish that every retrieved local action was correct, and failed attempts can complicate memory quality.
Robota-Embed-Bench contains held-out episodes, but task, object, and embodiment categories may overlap between training and evaluation. External baselines use their supported action-free interfaces rather than matched VLA inputs, while controlled modality conclusions rely on additional matched retraining and masking studies. Real-robot results are small, consisting of 12 rollouts per task and condition, and use task-specific partial-progress scores. The Embed2Evolve comparison with Omni-Embed-Nemotron-3B compares independently evolved systems rather than a common fixed final harness.
The demonstrated scope is therefore a predictive, action-aware VLA retrieval space that identifies task- and stage-appropriate future continuations and makes them usable for robot in-context learning and task-local context evolution around frozen policies. Proposed extensions include broader embodiment and action-space coverage and variable-length retrieved continuations instead of a fixed four-second window.