Retrieve, Don't Retrain: Extending Vision Language Action Models to New Tasks at Test Time

Abstract: Extending a vision-language-action (VLA) policy to a new task typically requires task-specific teleoperated demonstrations and per-task fine-tuning, making adaptation costly in both data collection and compute. In this paper, we show that this target-side per-task adaptation cost can be replaced by retrieval. Our retrieval-augmented policy is trained once on paired demonstrations from the target embodiment (query) and a cheaper embodiment (pool, e.g., human-hand video), then frozen. New tasks are added at deployment by appending pool-side demonstrations to a retrieval pool. The frozen policy conditions on retrieved trajectories at every control step, so new tasks are absorbed by indexing data rather than updating parameters. Fine-tuning is needed only to take on a new, unseen embodiment, not for each new task. We show that retrieval improves policies beyond a specific backbone, including standard VLA policies, but its effect is especially pronounced in Cosmos Policy, a video-generation-based world-action model (WAM). In this setting, retrieval supplies coarse task progression, while the WAM's future-image objective provides an additional visual consistency signal that strengthens the retrieval-conditioned actions. On PushT, we study how retrieval provides a reusable high-level motion prior for cross-embodiment generalization to unseen goal angles, while on RoboTwin 2.0 our method outperforms cross-embodiment baselines on unseen tasks, and we additionally demonstrate the method on a real robot.

• The paper presents ReCAP, which uses retrieval from a demonstration pool to adapt pre-trained vision-language-action models to novel tasks at test time.
• It leverages a composite retrieval strategy combining language, visual, and proprioceptive cues to generate residual actions, significantly boosting success rates on unseen tasks.
• Experimental results on PushT and RoboTwin benchmarks show monotonic generalization gains purely by expanding the demonstration pool post-training.

Retrieval-Augmented Test-Time Task Extension for Vision-Language-Action Models

---

Introduction and Problem Setting

This work introduces ReCAP, a retrieval-augmented policy enabling vision-language-action models, specifically world-action models (WAMs), to acquire new manipulation tasks at test time without retraining. Traditional deployments of VLA policies typically demand per-task teleoperated demonstrations and costly fine-tuning per task, which both scale unfavorably with increasing behavioral coverage. ReCAP circumvents this by shifting adaptation from network optimization to retrieval over a demonstration pool, amortizing data and computational cost.

This paradigm leverages the insight that the pool embodiment (e.g., human-hand video) can supply high-level task structure, while the target embodiment (e.g., robot arm) need only perform an embodiment-specific correction. The operational protocol is to train a single residual policy—frozen thereafter—to predict target actions conditioned jointly on the current observation and a retrieved pool-side demonstration chunk. New tasks are added post-hoc by simply growing the retrieval database with additional, cheap pool-embodiment demonstrations.

Figure 1: ReCAP overview: the standard paradigm demands teleoperated demos and per-task policy fine-tuning, whereas ReCAP absorbs new tasks via retrieval over a growing human demonstration pool, eliminating retraining.

---

Model Architecture and Retrieval Mechanism

At its core, ReCAP is instantiated atop a video-generation-based WAM, here Cosmos Policy. The approach encodes the current query observation and retrieves, per control timestep, a matching pool-side state-action chunk via a composite descriptor (language goals, object pose via SAM3, proprioception, and DINOv3 features). The policy then predicts both the next action chunk and the next observation for the robotic embodiment, with the action parameterized as a residual over the retrieved pool action. This forms a coarse-to-fine behavioral pipeline: retrieval supplies the motion prior, and the model adapts it to the target embodiment.

The training objective is a joint flow-matching loss over actions and future observations. The value of the video generation (i.e., state-prediction) objective is pronounced when paired with retrieval; it enforces pixel-wise consistency between the retrieved demonstration and predicted scene evolution, generating a stronger supervision signal for high-level motion alignment.

Figure 2: Action-slot attention in ReCAP shows the model initially routes attention toward the pool-supplied motion plan, then shifts to committing to its own predicted action at the decisive layer.

---

Experimental Results

PushT Cross-Embodiment Generalization

The principal testbed is PushT, where the target (triangle pusher) and pool (disc pusher) operate with divergent dynamics across a continuous goal-angle axis.

Figure 3: PushT pool database—test-time pool comprises disc-pusher demonstrations densely covering all goal angles, enabling retrieval on out-of-distribution tasks.

Key experimental findings include:

• Monotonic generalization: As the offline demonstration pool is expanded post-training, the frozen policy exhibits strictly monotonic improvements in unseen-goal success rate (from 6.0% to 34.9%), with no weight updates.

  Figure 4: Per-angle success rate on PushT rises as the retrieval pool grows, even without retraining; shaded gap shows the incremental effect of newly added demos.

• Superiority of retrieval over baselines: Against action-replay and cross-embodiment co-training baselines, ReCAP achieves the highest averaged generalization (>34%) on unseen goals. Notably, action-only backbones benefit from retrieval (raising from 6.6% to 25.1%), but the largest gains accrue with the full WAM objective.

Figure 5: Comparative backbone analysis shows both action-only and WAM policies improve with retrieval, with WAMs benefiting most.

Ablation shows that the future-image objective is critical: removing it reduces performance to near that of an action-only backbone.

---

RoboTwin 2.0: Dual-Arm Manipulation

Within RoboTwin 2.0, a dual-arm manipulation suite, ReCAP outperforms strong cross-embodiment baselines both on seen and five held-out tasks:

• Held-out task generalization: Incorporating only cheap pool-embodiment trajectories post-training, ReCAP attains 31.5% average success on previously unseen tasks (vs. 26.0% for the strongest baseline).

  Figure 6: Qualitative transfer on held-out RoboTwin tasks—ReCAP policy conditions on retrieved UR5 demonstration and executes successful manipulation, whereas baselines consistently fail on embodiment transfer.

Test-time pool expansion produces monotonic gains, with each retrieval pool increment enabling a corresponding new task to become solvable by the unchanged policy.

---

Physical Robot Experiments

Deployments on a real robot with a human-hand video pool (wrist-pose tracked in VR) further validate the method:

• Test-time behavioral extension: Retaining the policy trained only on open-cabinet, and supplying only human trajectories for "place-bottle-in-box" and "close-cabinet" at test, the policy leaps from ineffective repetition (0–10% success) to 80% and 30% success on held-out tasks.

  Figure 7: Real-robot rollouts—without retrieval, the policy repeats an irrelevant learned motion; with retrieval, it adapts in situ to unseen tasks via human-hand trajectory conditioning.

---

Mechanism Analysis

Cross-attention probes reveal a two-stage routing: an intake layer reads the pool demonstration task region, and a commit layer shifts decisively to the policy’s own predicted next state. Failures are characterized by either attenuated commitment (universal) or inappropriate reliance on retrieval (over/under-anchoring, dependent on distance from the training set), providing actionable axes for diagnosing and improving retrieval-conditioned policies.

Figure 8: Failure modes—universal weakening of commit, and goal-angle-specific under-/over-anchoring on the retrieval prior.

---

Implications and Future Directions

ReCAP formalizes a paradigm for deploying frozen robotic policies onto novel tasks by extending a retrieval pool, substantially reducing both teleoperation and computational cost, and decoupling task adaptation from parameter updates. This architecture is most effective when (1) a shared state/action representation exists, and (2) pool-embodiment trajectories are available in a structured format.

Limitations remain: divergent action spaces (dexterous hand vs. simple gripper) and video-only pool sources require abstraction or action-translation modules. Additionally, the system’s robustness diminishes as temporal or dynamic misalignments between pool and target widen. Addressing adaptation to such mismatches and the integration of web-scale or weakly labeled demonstration sources (e.g., raw YouTube videos) constitute promising avenues for increasing transfer scalability.

---

Conclusion

ReCAP demonstrates that in retrieval-augmented VLA policy frameworks, behavioral generalization to new tasks can be achieved entirely at test time by expanding a demonstration pool, obviating the need for costly per-task retraining. This test-time retrieval mechanism, especially when paired with video prediction-based world-action objectives, supports scalable, modular robot learning across embodiment gaps, offering a compelling model for practical deployment and future foundation model scaling.