Robota-Embed-Bench: Benchmark for Vision-Language-Action Embeddings
- .Robota-Embed-Bench is a benchmark that assesses vision-language-action (VLA) embeddings by retrieving the appropriate future continuation of robot or human manipulation trajectories, focusing on episodic and temporal continuation accuracy.
- The benchmark uses 4,000 held-out queries, 8,000 gallery entries, and data from 4 domains (EgoDex, DROID, LIBERO, RoboMIND–Franka-dual) to evaluate how well representations combine language, vision, and action for accurate future agent actions.
- Robota-Embed, the associated model, significantly outperforms other models in both clip and episode retrieval tasks, demonstrating the importance of including native action data in the embeddings.
Robota-Embed-Bench is a benchmark for evaluating vision-language-action (VLA) embeddings that retrieve the appropriate future continuation of a robot or human manipulation trajectory. Introduced with Robota-Embed, it tests whether a representation jointly uses language, visual interaction state, and native action to distinguish not only the correct task or episode, but also the particular execution stage that should follow a query observation (Liu et al., 14 Oct 2026). The benchmark contains 4,000 held-out queries, a shared gallery of 8,000 entries, 3,200 episodes, and data from EgoDex, DROID, LIBERO, and RoboMIND–Franka-dual.
1. Purpose and conceptual scope
Robota-Embed-Bench addresses predictive retrieval for embodied systems. A retrieval query represents the robot’s or human operator’s current state, while each gallery item represents a strictly subsequent segment of recorded experience. The benchmark therefore differs from conventional image-text or video-text retrieval, where semantic similarity to a complete task or scene may be sufficient.
The benchmark treats retrieval as a two-level problem:
- Episode identification: determining whether the retrieved item belongs to the appropriate interaction episode.
- Temporal continuation localization: identifying the particular future clip that follows the query anchor.
The second problem is more demanding. Multiple clips can belong to the same episode and share the same task, objects, and instruction while corresponding to different execution stages. A representation must consequently distinguish reaching, grasping, lifting, transporting, placing, and withdrawing, even when visual and linguistic context is similar.
The query contains three principal modalities:
- Language: the intended task or goal.
- Vision: the current scene and interaction state.
- Native action: the current motion, gripper state, direction, and execution phase.
The gallery entry contains the same instruction together with a future video clip and its synchronized native action sequence. The benchmark’s central hypothesis is that native action supplies predictive information that cannot be recovered reliably from language and vision alone.
Robota-Embed-Bench is not an end-to-end robot-success benchmark. It evaluates retrieval representations independently of downstream policy execution. Its associated applications, Embed2ICL and Embed2Evolve, examine whether retrieved experience can improve policy conditioning and task-local memory, but their results are downstream outcomes rather than benchmark retrieval metrics.
2. Formal retrieval task
For sample from source domain , the query is represented as
where is the natural-language instruction, is the current interaction-region image, is synchronized native action at time , and identifies the source domain and selects the corresponding action projector.
The paired gallery entry is
where is a four-second future video clip, 0 is the native action sequence aligned with the clip, and the clip begins strictly after the query anchor.
The query and gallery encoders produce normalized embeddings:
1
where 2 is the source-specific native-action projector. Retrieval uses cosine similarity,
3
because both embeddings are L2-normalized. Candidates are ranked in descending order of similarity, without reranking in the benchmark.
A clip is a sampled four-second future window containing 16 video frames and synchronized native actions. An episode is a complete interaction trajectory or recorded session. A future continuation is the window immediately following a current-state anchor. A nearby same-episode clip is temporally offset from the query’s paired continuation and is deliberately included as a difficult distractor.
The benchmark therefore evaluates predictive compatibility rather than simple semantic identity. A retrieved clip can be correct at the episode level while incorrect at the clip level if it comes from the appropriate trajectory but represents the wrong future stage.
3. Data sources, domains, and curation
Robota-Embed-Bench combines four sources:
- EgoDex: egocentric human dexterous-manipulation video with tracked human motion.
- DROID: real-robot manipulation trajectories.
- LIBERO: simulated robot manipulation.
- RoboMIND–Franka-dual: simulated or recorded Franka dual-arm manipulation data.
The sources span human experience, simulated robot experience, single-arm robots, dual-arm robots, and different native action spaces and sampling rates.
| Source | Domain ID | Native action dimension | Rate |
|---|---|---|---|
| EgoDex | 3 | 57 | 30 Hz |
| DROID | 8 | 10 | 15 Hz |
| LIBERO | 5 | 10 | 10 Hz |
| RoboMIND–Franka-dual | 12 | 20 | 10 Hz |
Actions are normalized per coordinate using training-partition first and ninety-ninth percentiles:
4
Values are not clipped. Action tensors are zero-padded to 64 dimensions, and the domain ID routes them through the appropriate projector.
The cleaned training corpus contains 1,695,684 current-to-future anchor pairs:
| Source | Filtered training rows |
|---|---|
| EgoDex | 1,345,160 |
| DROID | 26,563 |
| LIBERO | 136,671 |
| RoboMIND–Franka-dual | 187,290 |
| Total | 1,695,684 |
Although the sources contribute unequal numbers of cleaned rows, training samples them equally in a 5 source ratio.
Curation is designed to reduce crop noise, invalid temporal pairs, and train-test contamination. Complete episodes, sessions, and content identities are assigned to disjoint partitions before dense anchor sampling. Candidate anchors are retained only when sufficient future frames remain. SAM3 is prompted with phrases naming the instruction’s physical interaction objects, and detections must satisfy thresholds for proposal score, final score, image coverage, and zoom ratio. Crops are recomputed at 6 and verified deterministically by Qwen3-VL-8B-Instruct. A crop is retained only when the interaction object is visible, instruction-consistent, separable from other instances, and sufficiently unoccluded. If verification fails, an earlier neighboring timestamp is resampled and the synchronized image, action, and anchor time are updated.
The target begins strictly after the accepted anchor and contains the next four seconds of video and actions. Benchmark episodes are excluded from training, although task, object, and embodiment categories may recur across partitions. The split is therefore designed to prevent episode, session, and content-identity leakage rather than to enforce complete task or object novelty.
4. Benchmark composition and relevance definitions
The evaluation set contains 4,000 held-out queries, 3,200 episodes, and 8,000 gallery entries. Each source contributes 1,000 queries drawn from 800 episodes, 1,000 paired positive future clips, and 1,000 nearby same-episode clips. Each source therefore contributes 2,000 gallery entries, while every query searches the same shared 8,000-entry gallery rather than a source-specific gallery.
The source distributions differ substantially in instruction diversity. EgoDex contains 591 distinct task-instruction strings, DROID 800, LIBERO 39, and RoboMIND–Franka-dual 5. This produces a broad range from highly varied human instructions to smaller instruction vocabularies in robot and simulation datasets.
Clip-level retrieval
Each query has exactly one relevant clip:
7
A gallery item is a clip-level negative if it is not the exact paired future clip. Even a different clip from the same episode is treated as negative at this level.
Episode-level retrieval
Any gallery clip from the same source-scoped episode is relevant:
8
A nearby same-episode clip is consequently positive for episode-level retrieval but negative for clip-level retrieval. This difference makes it possible to measure whether a model recognizes the correct trajectory without accurately localizing the next continuation.
Training negatives
During training, a different clip ID is not automatically considered a valid negative. The negative set contains candidates with explicit semantic conflicts, including task, object, action, or joint task/object conflicts. Compatible, uncertain, potentially duplicated, or temporally overlapping candidates are ignored. Same-episode future windows with temporal intersection-over-union greater than 0.25 are excluded as near-duplicates. Cross-episode action negatives must also differ in future gripper phase.
The principal query-to-target masked InfoNCE loss is
9
where 0 is the minibatch, 1 is the valid semantic-negative set, and 2 is a learned temperature constrained to 3. A reverse target-to-query loss is computed with an independently constructed validity mask:
4
The full objective combines predictive current-to-future alignment, auxiliary grounding, and regularization:
5
The grounding terms include text/image alignment, modality-swap penalties, hard task/object/action negatives, and fused-modality dominance. The regularization terms address centering, isotropy, variance, and effective rank.
5. Metrics and evaluation protocol
For query 6, let 7 denote the rank of the relevant item under the specified retrieval protocol. Recall at 8 is
9
For clip retrieval, 0 is the rank of the unique paired clip. For episode retrieval, it is the rank of the highest-ranked clip belonging to the correct episode. Mean reciprocal rank is
1
Clip-level metrics are averaged equally across the four sources. Episode-level metrics are first averaged over queries within each episode, then equally over the 800 episodes in each source, and finally equally across sources. This prevents episodes with more sampled queries from dominating the aggregate.
The benchmark reports R@1, R@5, R@10, and MRR for both clip- and episode-level retrieval. All 8,000 candidates are ranked using cosine similarity without benchmark reranking.
The distinction between metrics is substantive rather than cosmetic. High episode-level recall with lower clip-level recall indicates that the representation identifies the appropriate task trajectory but fails to determine the exact continuation. This episode-versus-clip gap is the benchmark’s principal temporal-localization diagnostic.
6. Robota-Embed architecture and results
Robota-Embed uses Cosmos3-Edge 4B as its base model. Its architecture contains 28 Transformer layers, hidden width 2,048, a shared Transformer, video tokenizer and retrieval projection, separate query and target residual adapters, source-specific native-action projectors, a frozen video VAE, and 1,024-dimensional normalized endpoint embeddings.
For endpoint 2, action-token hidden states 3 are temporally pooled and projected:
4
Queries use the instruction, current native action, and localized interaction crop. Targets use the instruction, 16 future frames, and future native actions. Training uses AdamW, 5, gradient clipping at 1.0, a Transformer learning rate of 6, head and adapter learning rate of 7, weight decay 0.05, global batch size 512, 26,275 optimizer steps, 1,314 linear warm-up steps, cosine decay, equal source sampling, and the final-step checkpoint for evaluation.
Aggregate results
| Model | Clip R@1 | Clip R@5 | Clip R@10 | Clip MRR | Episode R@1 | Episode R@5 | Episode R@10 | Episode MRR |
|---|---|---|---|---|---|---|---|---|
| CLIP ViT-L/14 | 1.2 | 5.8 | 9.8 | 4.06 | 3.2 | 7.7 | 11.6 | 5.98 |
| Perception Encoder B16 | 2.0 | 8.1 | 12.8 | 5.52 | 5.0 | 10.1 | 15.2 | 8.01 |
| InternVideo2 | 1.6 | 7.3 | 11.4 | 4.82 | 4.1 | 9.1 | 13.2 | 6.94 |
| Gemini Embedding 2 | 25.2 | 52.3 | 57.5 | 37.68 | 48.3 | 58.2 | 64.7 | 53.69 |
| Omni-Embed-Nemotron-3B | 24.5 | 54.5 | 60.1 | 38.10 | 48.6 | 59.1 | 64.4 | 53.84 |
| Robota-Embed | 47.33 | 91.90 | 96.95 | 67.05 | 81.77 | 94.91 | 98.06 | 86.96 |
Relative to Gemini Embedding 2, Robota-Embed improves clip R@1 by 22.09 percentage points and episode R@1 by 33.47 percentage points. Relative to Omni-Embed-Nemotron-3B, the corresponding improvements are approximately 22.83 and 33.18 percentage points.
Performance varies by source:
| Source | Clip R@1 | Clip R@5 | Clip R@10 | Clip MRR | Episode R@1 | Episode R@5 | Episode R@10 | Episode MRR |
|---|---|---|---|---|---|---|---|---|
| EgoDex | 36.70 | 96.60 | 99.30 | 62.22 | 87.06 | 98.94 | 99.44 | 90.98 |
| DROID | 74.10 | 99.00 | 100.00 | 85.27 | 99.88 | 100.00 | 100.00 | 99.94 |
| LIBERO | 38.50 | 87.00 | 96.60 | 60.33 | 67.62 | 91.81 | 98.19 | 77.42 |
| RoboMIND-FD | 40.00 | 85.00 | 91.90 | 60.39 | 72.50 | 88.88 | 94.62 | 79.50 |
| Average | 47.33 | 91.90 | 96.95 | 67.05 | 81.77 | 94.91 | 98.06 | 86.96 |
The episode/clip gap is largest for EgoDex, where episode R@1 is 87.06% but clip R@1 is 36.70%. The gap is also substantial for LIBERO and RoboMIND–Franka-dual. The overall median rank is 2 for clip retrieval and 1 for episode retrieval; the mean rank is 2.72 for clips and 1.88 for episodes.
7. Ablations, applications, and limitations
Modality ablations
In matched DROID retraining experiments, the full vision-language-action system achieves 78.7% clip R@1. Removing action reduces performance to 59.8%, a decline of 18.9 percentage points. Removing vision produces 76.8%, a decline of 1.9 points, while removing language produces 20.2%, a decline of 58.5 points. Thus, language is dominant overall, but native action contributes more than vision in the retrained comparison.
| Input | Clip R@1 |
|---|---|
| Vision + language + action | 78.7 |
| Vision + language | 59.8 |
| Language + action | 76.8 |
| Vision + action | 20.2 |
In fixed-model query masking across all four sources, full vision-language-action input produces 47.33% clip R@1 and 81.77% episode R@1. Masking action reduces these to 20.08% and 37.36%, while masking vision reduces them to 23.00% and 39.50%. The different effects of retraining ablation and fixed-model masking reflect the distinction between allowing the representation to adapt and imposing an input-distribution shift at inference.
Replacing gallery instructions with empty prompts reduces clip R@1 from 47.33% to 12.78% and episode R@1 from 81.77% to 23.17%. However, restricting retrieval to candidates with the same instruction leaves aggregate performance almost unchanged: clip R@1 rises from 47.33% to 47.62%, and episode R@1 from 81.77% to 82.53%. This indicates that performance is not explained solely by instruction-string matching.
Embed2ICL
Embed2ICL uses retrieved experience to condition a downstream policy. In a training-free DreamZero interface on three DROID-Sim tasks, Robota-Embed with reranking reaches 56.3% average success, compared with 47.0% without warm-up, 43.3% for Omni-Embed top-1, and 51.0% for Omni-Embed with reranking.
On the YAM real robot, mean Banana-to-Bowl progress increases from 21.67 to 40.00, while binary success increases from 0.0% to 16.7%. In a training-based Fast-WAM interface on a difficult LIBERO subset, official performance is 86.4%, compared with 92.8% using retrieved Embed2ICL context tokens.
These results depend on the downstream policy, reranking, and context-injection mechanism. They are not direct substitutes for Robota-Embed-Bench retrieval scores.
Embed2Evolve
Embed2Evolve uses Robota-Embed within task-local memory and context management around a frozen GPT-6 Astra policy. Historical successes and failures are indexed as anchors, and candidate selection combines embedding similarity with success and execution-efficiency terms.
Across five ManiSkill tasks, Embed2Evolve reaches 68% average success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B. It uses approximately 79.5% fewer policy tokens per test rollout than the latter system. The reported per-task success rates are 80% for Poke Cube, 60% for Stack Cube, 65% for Lift Peg Upright, 50% for Pull Cube with Tool, and 85% for Two-Robot Stack Cube.
Embed2Evolve is not a pure encoder-swap experiment: the embedding model affects memory selection, retrieval count, refresh cadence, retained history, success and efficiency bonuses, and the final evolved harness.
Limitations
Robota-Embed-Bench has several limitations:
- The evaluation gallery contains 8,000 entries, although scaling experiments study precomputed galleries up to 15 million entries.
- Targets have a fixed four-second duration, whereas different tasks may require different temporal horizons.
- Task, object, and embodiment categories can recur across splits; the benchmark is not a strict task-zero-shot or compositional-generalization evaluation.
- Weak semantic labels are generated through structured parsing and deterministic rules rather than human annotation.
- Source-balanced aggregation does not necessarily reflect real-world deployment frequencies.
- Most external baselines are action-free, use uncropped images, and follow their supported interfaces, so their inputs are not identical to Robota-Embed’s.
- Modality masking introduces distribution shift.
- Episode-level relevance may credit a temporally inappropriate clip.
- Downstream policy results depend on policy architecture, reranking, context injection, and training.
- Statistical significance intervals and repeated-seed variance are not reported.
- Generalization to entirely unseen robot embodiments and action spaces remains limited.
Robota-Embed-Bench’s principal contribution is the explicit separation of episode retrieval from future-continuation retrieval. Its results show that identifying the correct task trajectory is substantially easier than identifying what should happen next. By incorporating native action into both query and gallery representations, the benchmark evaluates a form of embodied predictive alignment that conventional semantic retrieval metrics do not capture.