Robota-Embed: Unified VLA Embeddings for Robot In-Context Learning and Evolution
Abstract: Retrieving useful context for a robot requires matching its task, execution stage, and the continuation that should follow. We introduce Robota-Embed, to our knowledge the first vision-language-action (VLA) embedding model to jointly encode all three modalities in both current-state queries and recorded future continuations. Predictive retrieval training aligns these endpoints, enabling historical continuations to serve as context at inference time. We construct a multi-source corpus of 1,695,684 cleaned training anchors and introduce Robota-Embed-Bench, with 4,000 queries and a shared 8,000-entry gallery. Robota-Embed achieves 47.33% clip-level and 81.77% episode-level R@1, exceeding action-free Gemini Embedding 2 by 22.09 and 33.47 percentage points. Controlled retrieval studies show that removing native action degrades retrieval more than removing vision. We study two complementary applications. Embed2ICL uses retrieved experience for training-free and training-based policy conditioning, increasing success from 47.0% to 56.3% in DROID-Sim and from 86.4% to 92.8% on the LIBERO difficult subset, while improving task progress on real robots. Embed2Evolve uses execution feedback to grow task-local memory and refine context use around a frozen GPT-6 Astra policy. Across five ManiSkill tasks, it achieves 68% success, compared with 24% for GPT-6 Astra Direct and 33% for a harness independently evolved with Omni-Embed-Nemotron-3B under the same protocol. Relative to the latter, it uses approximately 79.5% fewer policy tokens per test rollout, counting both successes and failures. These results show that a unified, action-aware retrieval space can support robot in-context learning and task-local context evolution.
First 10 authors:
Paper Prompts
Sign up for free to create and run prompts on this paper.
Explain it Like I'm 14
1. What is this paper about?
This paper introduces Robota-Embed, a system that helps robots find useful past experiences.
Imagine a robot trying to put a cup on a table. Instead of figuring everything out from scratch, it could search through videos and records of earlier robot actions to find a similar example. The robot could then use that example to decide what to do next.
Robota-Embed searches for useful examples using three kinds of information:
- Vision: What the robot currently sees
- Language: What the robot has been told to do
- Action: What the robot is currently doing, such as moving its arm
The paper argues that all three are important. A picture and an instruction may look similar in two situations, but the correct next movement may be different.
2. What questions did the researchers ask?
The researchers wanted to answer several main questions:
- Can a robot find the right past experience? For example, can it find a recording that shows what should happen after a robot reaches for a bowl?
- Does information about movement help? Is knowing what the robot is already doing more useful than only looking at pictures and reading instructions?
- Can retrieved examples help a robot perform tasks better? The researchers tested whether selected examples could improve robot performance in simulations and on real robots.
- Can a robot improve its own memory over time? They studied whether a robot could save experiences from earlier attempts and learn which memories are most useful.
3. How did the researchers do the study?
Building a shared “experience map”
The researchers trained Robota-Embed to turn information into mathematical objects called embeddings. An embedding is like a location on a large map.
On this map, experiences that are useful for similar situations should be placed close together. For example:
- A robot currently reaching toward a cup
- An earlier recording showing the robot successfully grasping a similar cup
should have nearby locations on the map.
The system creates one embedding from the robot’s current situation and another embedding from a possible future recording. It then compares them using cosine similarity, which is a mathematical way of measuring how closely two directions point on the map.
Training with current situations and future actions
The researchers used nearly 1.7 million examples from four sources:
- Human movement recordings
- Simulated robot tasks
- Real robot demonstrations
- Other robot-control datasets
For each example, they gave the system:
- An instruction
- A picture of the current situation
- The robot’s current movement
They paired this with a video showing the next four seconds and the actions that followed.
This teaches the system to answer a question like:
“Given what the robot sees, what it was told, and what it is doing now, which past recording shows the most useful continuation?”
Testing retrieval ability
The researchers created Robota-Embed-Bench, a test containing:
- 4,000 questions
- 8,000 possible video clips
The system had to find the correct future clip from the possible choices.
They compared Robota-Embed with other systems that use images and language but do not include the robot’s actions.
Testing robots with retrieved examples
The researchers also used the retrieved examples in two applications:
- Embed2ICL: The robot receives a useful example before performing a task.
- Embed2Evolve: The robot stores examples from repeated attempts and improves how it chooses and uses them.
These systems were tested in simulations and on real robots.
4. What did the researchers find?
Robota-Embed found better examples
Robota-Embed correctly found the exact matching future clip first about 47% of the time. It found a clip from the correct overall task episode first about 82% of the time.
The strongest comparison systems found the exact clip first only about 25% of the time.
This means Robota-Embed was much better at finding not only a similar task, but also the correct next part of the task.
Robot actions were especially important
One of the most important findings was that action information helped more than expected.
When the researchers removed information about the robot’s movements during retraining, performance dropped by about 19 percentage points. Removing vision caused a much smaller drop in that experiment.
This suggests that two scenes can look almost the same but require different actions depending on what the robot is already doing. For instance, reaching toward an object and pulling away from it may involve the same object and instruction, but they need different continuations.
Retrieved examples improved robot success
In one simulated experiment, giving the robot a carefully selected example increased average success from:
- 47.0% without a helpful example
- To 56.3% with a retrieved example
Random examples made performance worse, showing that simply giving the robot more information is not enough. The example must be relevant.
In another set of difficult tasks, retrieved context increased success from 86.4% to 92.8%.
The method also improved progress on real-robot tasks. For example, performance on a “split cup” task increased from 66.67% to 87.50%.
The robot’s memory could improve over time
In the Embed2Evolve experiments, the robot saved successful and unsuccessful attempts in a task-specific memory. It then learned how many examples to retrieve and how much past history to show its controlling system.
Across five simulated tasks:
- A normal GPT-6 Astra robot succeeded 24% of the time.
- A system using another embedding model succeeded 33% of the time.
- Embed2Evolve with Robota-Embed succeeded 68% of the time.
It also used fewer words, or tokens, when communicating with the control system. This is important because fewer tokens can mean lower cost and faster operation.
5. Why is this research important?
Robots often fail because they do not understand what should happen next, even when they recognize the object or task. Robota-Embed gives robots a better way to search their past experiences by considering:
- What the robot sees
- What the robot is trying to do
- What movement it is already making
The research suggests that future robots may not need to learn every task from the beginning. Instead, they could:
- Search a large library of human and robot experiences.
- Find an example that matches the current situation.
- Use that example to guide their next actions.
- Save the result and improve their memory through repeated attempts.
This could make robots more flexible and useful in homes, factories, laboratories, and other places. However, the research was tested on particular datasets, tasks, and robot systems. More work is still needed to see whether the method works reliably in messy real-world environments with unfamiliar objects and unexpected problems.
Overall, the paper’s main message is simple: for robots, remembering the right movement—not just recognizing the right object—can make a major difference.
Practical Applications
Immediate Applications
- Robot experience retrieval for deployment and debugging — Robotics, manufacturing, logistics.
Deploy
Robota-Embedas an action-aware vector index over demonstrations from human operators, simulations, and prior robot episodes. Given the robot’s current instruction, visual state, and motion, the system can retrieve the most relevant future continuation rather than merely a visually similar scene. This can support demonstration search, failure analysis, operator review, and recovery planning. Dependencies: synchronized vision, language, and native-action data; compatible action projectors for each robot embodiment; reliable interaction-region detection; sufficient coverage of the target task distribution. - Retrieval-augmented robot control without policy retraining — Industrial robotics, service robotics, warehouse automation.
The
Embed2ICLtraining-free interface can retrieve a relevant video and use it to warm up a policy’s visual cache. This offers a practical workflow for adapting a frozen robot policy to new object arrangements or task instances. The paper reports improved DROID-Sim performance, from 47.0% without warm-up to 56.3% with selected demonstrations, and improved real-robot task progress. Dependencies: the control policy must accept video or visual-cache context; retrieved demonstrations must be temporally and behaviorally relevant; safety monitoring is required because a retrieved example is not a guarantee of successful execution. - Action-aware demonstration libraries — Robotics research, robot manufacturers, education. Build searchable libraries in which demonstrations are indexed by task, scene, execution stage, and motion—not only by text or image similarity. Researchers and engineers could query for examples such as “withdraw after placing an object,” “stabilize with one arm while separating with the other,” or “recover after a failed grasp.” The paper’s results indicate that native action is central to this distinction: removing action reduced matched retrieval performance by 18.9 percentage points. Dependencies: action logs must be calibrated and represented consistently; cross-robot motion differences may require embodiment-specific normalization or projectors.
- Robot policy conditioning with cached context tokens — Robotics software infrastructure. For policies such as Fast-WAM, precompute and cache retrieved demonstrations’ context tokens, masks, and embeddings. At runtime, a robot only needs to encode its current query, perform cosine search, and load the selected context. This can reduce online computation and make retrieval-augmented control compatible with production inference pipelines. Dependencies: the policy must be trained or adapted to consume the context-token interface; cached representations become stale if the embedding model, action encoding, or data distribution changes.
- Simulation-to-real development workflow — Robotics R&D and digital twins. Use simulation episodes, human demonstrations, and real-robot trajectories in a shared retrieval system to select behaviorally relevant references during policy development. The reported improvements on LIBERO and real-robot tasks suggest a workflow in which simulation supplies broad experience while real-robot data validates embodiment-specific execution. Dependencies: simulation actions and observations must be sufficiently aligned with reality; domain gaps in friction, camera geometry, latency, and actuator behavior may limit transfer.
- Task-local robot memory for repeated operations — Manufacturing, inspection, agriculture, domestic robotics.
Implement a memory service that stores short continuation clips, actions, outcomes, and feedback after each attempt. On later attempts, the robot retrieves examples from the same task and execution stage to guide control. This is directly supported by
Embed2Evolve, which adds successful and failed attempts to task-local memory and retrieves RGB, actions, and feedback. Dependencies: memory must be scoped by task and environment; failed trajectories should be labeled carefully so that the system does not imitate unsafe behavior; retention and refresh policies need monitoring. - Token-efficient context management for robot agents — Robotics platforms and embodied-agent software. Use embedding-guided retrieval to select a small number of high-value historical episodes instead of passing the full interaction history to a large policy. The paper reports substantial token reductions relative to an independently evolved retrieval harness. This can lower inference cost and latency for cloud- or edge-hosted robot agents. Dependencies: token savings must not remove critical safety or state information; the reported efficiency depends on the specific policy, prompt format, memory size, and evaluation protocol.
- Benchmarking and regression testing for physical-experience retrieval — Academia and industrial QA. Adopt the paper’s benchmark design principles: separate episode-level retrieval from exact continuation retrieval, include nearby same-episode distractors, and evaluate clip-level and episode-level recall. A company could use such a benchmark to test whether a new embedding model retrieves the correct next behavior rather than merely the correct task or object. Dependencies: benchmark data must contain reliable temporal alignment and meaningful hard negatives; performance on the four-source benchmark should not be assumed to generalize to unseen robots or tasks.
- Human-in-the-loop demonstration search and teleoperation support — Teleoperation, training, maintenance. During teleoperation or manual robot programming, retrieve comparable prior executions to help an operator choose a motion strategy, inspect likely failure modes, or select a recovery sequence. Action-aware retrieval is especially relevant when identical objects require different motions depending on the current stage. Dependencies: the interface must present uncertainty and provenance; operators should be able to reject unsuitable references; privacy and consent are required for human-motion datasets such as EgoDex.
- Educational tools for robot learning — Academia and robotics education. Create interactive systems in which students issue a task instruction and inspect retrieved human, simulated, and robot demonstrations. Students could compare how language, visual state, and action change the retrieved continuation, making the system useful for teaching imitation learning, multimodal representation learning, and robot control. Dependencies: demonstrations need clear annotations and safe simulators; educational conclusions should not rely solely on retrieval metrics or small task suites.
Long-Term Applications
- General-purpose experience operating systems for fleets of robots — Warehouse, healthcare, agriculture, construction, and home robotics. A fleet-wide action-aware memory could aggregate experiences across robot embodiments, sites, and tasks. Robots could retrieve compatible continuations from other machines and use them for adaptation, recovery, or planning. Source-specific projectors in the paper provide an initial mechanism for heterogeneous action formats. Dependencies: robust cross-embodiment alignment, standardized action representations, access control, continual-learning safeguards, and extensive validation under distribution shift are required. Human demonstrations may not be directly executable by robots without geometric and control translation.
- Autonomous recovery and failure-aware planning — Safety-critical robotics and industrial automation. Extend the retrieval target from ordinary future clips to labeled recovery continuations: failed grasp, object slip, occlusion, collision avoidance, or partial completion. A robot could retrieve what historically followed a similar failure state and choose among recovery actions. Dependencies: large, accurately labeled failure datasets; explicit uncertainty estimation; formal safety constraints; and a controller capable of verifying that a retrieved recovery action is appropriate in the current physical state.
- Continual robot learning without repeated policy fine-tuning — Long-lived service and industrial robots.
Combine task-local memory growth with periodic evaluation of retrieval and context policies. Robots could improve through accumulated experience while keeping the core policy frozen, reducing the cost and risk of frequent parameter updates. The paper’s
Embed2Evolveresults provide an early demonstration of this approach. Dependencies: protection against memory poisoning, catastrophic retrieval errors, nonstationary environments, and feedback bias; the system must distinguish genuine improvement from overfitting to development episodes. - Cross-robot skill transfer and embodiment translation — Robotics platforms and robot manufacturers. Use the shared VLA representation to retrieve a human or robot demonstration and then translate its continuation into the target robot’s action space. This could enable knowledge transfer between robot arms, mobile manipulators, and dual-arm systems. Dependencies: retrieval similarity alone is insufficient for executable transfer. The system would require kinematic retargeting, object-centric representations, dynamics adaptation, collision checking, and validation on the target hardware.
- Large-scale autonomous data curation — Robotics laboratories and industrial data platforms. Use the embedding space to select informative episodes for labeling, simulation generation, policy fine-tuning, or human review. Examples that are temporally ambiguous, behaviorally novel, or associated with failures could be prioritized for additional annotation. Dependencies: novelty and usefulness must be calibrated separately from embedding distance; automated curation can amplify dataset bias if underrepresented tasks or embodiments are not explicitly monitored.
- Embodied-agent planning with multimodal episodic memory — General-purpose AI and autonomous systems. Integrate action-aware retrieval into agents that plan through repeated interaction. Instead of storing only text reflections, the agent could retrieve visual states, motor trajectories, outcomes, and short future continuations. This may support manipulation, navigation, tool use, and other embodied tasks where the same goal requires stage-specific behavior. Dependencies: the current evidence is concentrated on manipulation and selected simulation environments; broader navigation, locomotion, and long-horizon tasks require new datasets, temporal abstractions, and evaluation protocols.
- Safety and governance systems for embodied AI — Policy, regulation, and enterprise compliance. Maintain provenance-linked records of the demonstrations that influenced each robot action. A retrieval log could support audits, incident investigation, dataset governance, and certification by showing which historical experiences were selected and why. Benchmarks could also require testing against adversarially similar but unsafe demonstrations. Dependencies: similarity scores are not explanations and should not be treated as causal evidence; governance systems need interpretable metadata, immutable logs, privacy protections, and independent safety validation.
- Adaptive assistants in healthcare and assisted living — Healthcare robotics and eldercare. In the longer term, a robot could retrieve stage-appropriate demonstrations for tasks such as object handover, meal assistance, or room organization, adapting to individual preferences and physical environments. Human-motion data may help represent natural interaction patterns, while robot trajectories provide executable action context. Dependencies: strict privacy, consent, clinical validation, robust human detection, fail-safe interaction, and certification are essential. The paper does not establish suitability for medical or physical-contact tasks.
- Energy-efficient edge inference for robot fleets — Embedded systems and energy-constrained robotics. Precompute gallery embeddings and context tokens, use approximate nearest-neighbor search, and transmit only compact retrieved context to edge robots. The reported scaling results—15 million entries requiring approximately 37.47 ms for top-10 search on an A100—suggest a path toward large but operationally manageable indexes, although edge hardware will require further optimization. Dependencies: quantization, hardware-specific acceleration, network reliability, cache freshness, and latency budgets must be studied; A100 measurements cannot be directly generalized to embedded processors.
- Standardized physical-experience retrieval infrastructure — Academic and industrial ecosystem. The paper could motivate common formats for synchronized language, video, proprioception, native actions, continuation windows, and outcome labels. Such standards would enable interoperable robot-memory services, comparable retrieval benchmarks, and shared datasets across laboratories and manufacturers. Dependencies: agreement on privacy, action semantics, coordinate systems, temporal sampling, and evaluation definitions; the field must also address licensing and ownership of human and industrial demonstrations.
Glossary
- Action diffusion: A generative process that iteratively denoises representations to produce robot actions. “at every action-diffusion denoising step”
- Action-native: Adapted to the original action representation of a particular data source or robot. “Source-specific projectors map native actions into the backbone.”
- Action-aware metric learning: Metric-learning methods that incorporate behavioral or action information when learning similarity representations. “action-aware metric learning connects perception or language with behavior”
- Adapter: A trainable module inserted into a largely shared neural architecture to specialize or modify its behavior. “with separate adapters ”
- Additive scoring: An evaluation scheme that sums independently earned partial-credit components. “Pick Fruit and Split Cup award 0.25 independently for each completed sub-step”
- Anchor: A sampled point in an interaction trajectory used to construct a query and its corresponding future continuation. “We split episodes before sampling anchors”
- Backbone: The main feature-extraction network underlying a machine-learning model. “with backbone, data, and training budget matched on DROID”
- Bidirectional InfoNCE: A contrastive-learning objective that evaluates query-to-target and target-to-query matching directions. “Predictive retrieval alignment ($\mathcal{L}_{\mathrm{align}$) uses bidirectional InfoNCE”
- Cross-attention: An attention mechanism in which one sequence attends to the representations of another sequence. “native cross-attention reads it in all 30 video-expert and 30 action-expert layers”
- Cosine similarity: A measure of vector similarity based on the cosine of the angle between two vectors. “we rank candidates by the cosine similarity between the current-state query”
- Context adapter: A trainable projection module that transforms retrieved representations into a policy’s conditioning format. “The context adapter inserts cached tokens into native conditioning”
- Context harness: The combined memory and configuration that determines how retrieved context is supplied to a policy. “This memory and context recipe form a harness”
- Context Tokens: Intermediate learned representations extracted from retrieved multimodal states and supplied to a robot policy. “Cached Context Tokens from its multimodal temporal states condition the policy”
- Contrastive learning: Representation learning that pulls compatible examples together and pushes incompatible examples apart in an embedding space. “action-aware metric learning connects perception or language with behavior”
- Discrimination objective: A training objective that encourages representations to distinguish semantically different inputs. “Multimodal grounding and discrimination”
- Embodiment: The physical form, capabilities, and sensorimotor configuration of a robot or agent. “task, object, and embodiment categories may recur”
- Embedding space: A vector space in which data items are represented so that geometric relationships encode semantic or task-related similarity. “a unified action-aware retrieval space”
- Episodic memory: A memory system that stores information about particular past events or interaction episodes. “Reflexion stores verbal reflections in episodic memory”
- Flow matching: A generative-modeling method that learns a vector field for transforming a simple distribution into a target distribution. “Video and action flow-matching losses update approximately 5.00B video-expert parameters”
- Fused representation: A single representation formed by combining information from multiple modalities. “Native action is learned within the fused representation”
- Gallery: The collection of candidate items searched during information retrieval. “all 4,000 held-out queries search one shared 8,000-clip gallery”
- Hard semantic conflict: A negative training example that is especially similar at a superficial level but semantically incompatible with the query. “margins against modality swaps and hard semantic conflicts”
- In-context learning (ICL): Adapting a model’s behavior from examples provided in its input context without necessarily updating its parameters. “Embed2ICL supports robot in-context learning”
- Interaction crop: An image region selected to focus on the relevant object or interaction area. “SAM3 interaction crops provide a further, modest localization gain over full frames”
- KV cache: Stored key and value activations used to avoid recomputing attention representations during autoregressive processing. “visual KV-cache warm-up”
- LayerNorm: A normalization operation that standardizes activations across the feature dimensions of each individual example. “Pooled states pass through shared LayerNorm”
- Low-rank embedding: An embedding representation whose effective variation is concentrated in a small number of dimensions. “discourages low-variance, low-rank embeddings”
- Metric learning: Learning a representation or distance function in which semantically related items are close and unrelated items are distant. “action-aware metric learning connects perception or language with behavior”
- Modality masking: Removing one input modality during inference or evaluation to measure the model’s reliance on it. “masking query action or vision lowers clip R@1”
- Modality swap: Replacing one modality with another example to create a contrastive or discrimination training case. “modality swaps encouraging sensitivity to motion”
- Multimodal grounding: Aligning representations across modalities with shared semantic or task concepts. “Retrieval training combines predictive alignment, multimodal grounding, and representation regularization.”
- Native action: Action information represented in the original format produced by a particular data source or robot. “Each endpoint jointly encodes language, vision, and action into one normalized vector.”
- Normalized vector: A vector scaled to have unit length, often enabling cosine similarity to measure alignment. “Each endpoint jointly encodes language, vision, and action into one normalized vector.”
- Offline embedding encoder: A model that computes representations before deployment or online retrieval. “We train the policy to use this context while keeping the offline embedding encoder fixed”
- Predictive alignment: Training representations so that a current state matches the future continuation that actually follows it. “We train with predictive alignment, multimodal grounding, and representation regularization.”
- Proprioception: Sensing of a robot’s internal state, such as joint positions, velocities, or forces. “the prefilling adapter projects the tokens to Fast-WAM's context width and appends them to the native text and proprioceptive conditioning”
- Projector: A learned mapping that converts source-specific features into the shared representation space of a model. “Source-specific projectors map native actions into the backbone.”
- Recall at k (R@k): The proportion of queries for which a relevant item appears among the top retrieved results. “achieves 47.33\% clip-level and 81.77\% episode-level R@1”
- Reference reranking: Reordering initially retrieved candidates using an additional relevance model or criterion. “Reranking improves both retrievers.”
- Representation regularization: Constraints or penalties designed to produce stable, non-collapsed, and well-distributed learned representations. “Representation regularization raises R@1 to 77.0\%”
- Retrieval endpoint: One of the representations used at either the query or candidate side of a retrieval comparison. “The retrieval endpoints have different temporal roles”
- Rollout: A single generated or executed sequence of actions in an environment. “All six conditions use 100 trials per task”
- RMSNorm: A normalization method that scales activations using their root mean square without subtracting the mean. “after layer 28's generation RMSNorm”
- Sensorimotor: Relating to the interaction between sensory perception and motor action. “Robot In-Context Learning from sensorimotor or video examples”
- Shared representation: A common vector encoding used to represent information from multiple modalities or sources. “Native action is learned within the fused representation”
- Task-local memory: A memory store containing experiences associated with a particular task rather than a general corpus. “Repeated attempts grow task-local memory”
- Temporal localization: Identifying the precise time interval or continuation within a longer episode that corresponds to a query. “Episode retrieval does not resolve temporal localization”
- Token-level alignment: Matching individual token representations across sequences or modalities. “no separate head or token-level alignment or reconstruction loss”
- Trajectory: An ordered sequence of states and actions generated during an interaction. “The paired target combines the same instruction with a four-second video clip”
- Visual cache: Stored visual representations used by a policy as context for subsequent prediction. “Embed2ICL initializes the visual cache of frozen DreamZero-DROID”
- Vision–language–action (VLA) model: A model that jointly represents visual observations, language instructions, and physical actions. “the first to jointly encode all three modalities”
- Warm-up: Providing initial contextual information to a model before it begins generating or acting. “No warm-up”
- Weak labels: Imperfect or indirectly generated training labels obtained from heuristics, rules, or automatically structured data. “Structured parsing and deterministic rules supply weak labels.”






