EXIMO: VLM Guided Exploration of VLA Policies

Abstract: How to efficiently finetune robot policies to learn new tasks on the fly? State of the art robotic manipulation policies are based on behaviour cloning of large vision-language-action (VLA) models with billions of parameters on huge teleoperation datasets. While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem. In particular, collecting teleoperation datasets requires hundreds of hours of expensive human labour and the alternative, reinforcement learning (RL), can be notoriously sample-inefficient especially for long-horizon tasks. In addition, RL with VLAs imposes several challenges due to the model's size and architectural design. In this work, we propose EXIMO, an efficient algorithm for finetuning of VLA policies. EXIMO operates in three stages: explore, imitate, and optimize. During the explore phase, EXIMO equips the VLA with a vision LLM (VLM) that acts as a planner. The VLM thinks and breaks down challenging long-horizon problems into shorter ones for the VLA. The VLM, together with the VLA, is used to collect an orchestrated dataset on new tasks. During the imitate phase, the VLA is finetuned with the orchestrated data. Finally, during the optimize stage, we use residual off-policy RL to further finetune the policy. In our experiments, we ablate all three stages of EXIMO and show that it outperforms existing approaches significantly in terms of sample-efficiency and final performance.

• The paper introduces a three-stage explore–imitate–optimize pipeline in which a VLM orchestrates a pretrained VLA, successful trajectories are filtered for behavior cloning, and residual off-policy RL further improves performance.
• Across 22 simulated ALOHA manipulation tasks, VLM orchestration improves exploration, supervised distillation outperforms both the base VLA and its live VLM teacher, and residual RL adds consistent success-rate gains.
• The method reduces reliance on teleoperation for compositional and reasoning-heavy tasks, but depends on ground-truth success detection and remains unvalidated on physical robots, with residual-policy distribution shift limiting some variants.

Eximo is a three-stage finetuning pipeline for vision-language-action (VLA) policies that replaces teleoperation with a vision-LLM (VLM) acting as an orchestration-based exploration policy (2608.19891). The work targets a concrete gap: pretrained VLAs such as Gemini Robotics On-Device (GROD) reliably execute atomic skills within their training distribution, but fail on long-horizon, compositional, or reasoning-heavy tasks (e.g., "put the fruit the monkey likes to eat in the bowl"), and adapting them to such tasks normally demands hundreds of hours of teleoperation or suffers from the sample inefficiency of RL applied directly to large diffusion-based policies.

The explore–imitate–optimize pipeline

The first stage uses a VLM (Gemini) as a closed-loop high-level planner over the VLA. At each step the VLM receives camera observations and the task goal $g$, and emits a single natural-language instruction $g_t \sim \pi^{VLM}(\cdot | o_{\le t}, g)$ that the GROD VLA executes directly. Because GROD accepts free-form language commands, no grounding layer is needed—a notable departure from LLM-planner pipelines that require action grounding via masked LLMs or robot APIs. Successful episodes, identified by an environment ground-truth success detector, are stored in a filtered buffer.

In the imitation stage, the VLA is finetuned by behaviour cloning on the successful orchestrated trajectories, but conditioned on the original task goal $g$ rather than the intermediate VLM instructions. This distills multi-step VLM-guided behaviour into a standalone policy, removing VLM latency and query costs at deployment. A strong claim of the paper is that this distillation actually outperforms keeping the VLM orchestrator at evaluation time (GROD + SFT vs. GROD + VLM-Orchestration), attributed to the filtered, high-quality supervision and to compiling chained behaviour into one policy.

The optimize stage applies off-policy residual RL following Ankile et al.: a residual policy $\pi^{res}$ outputs $\Delta a$ conditioned on state, goal, and the VLA's proposed action, with the executed action $a = a^{VLA} + \Delta a$, trained with MPO on the sparse binary success reward. Non-trivial post-SFT success rates make exploration tractable for this otherwise sparse-reward problem.

Empirical findings

Evaluation covers 22 simulated ALOHA manipulation tasks spanning dish placement (chained pick-and-place), semantic reasoning variants (target object described only semantically), spatial left/right caddy placement, and multi-object chaining. Each task is evaluated over 1000 episodes with mean and standard error reported.

VLM orchestration improves exploration. Orchestrated GROD achieves markedly higher success rates than the base VLA alone, with comparable time-to-success and substantially shorter episodes—gains concentrated precisely on long-horizon tasks (e.g., PlateBowlOnRack) and reasoning tasks (BananaInBowl-Reasoning). This confirms the paper's central premise: separating semantic task understanding (VLM) from motor skills (VLA) yields efficient data collection without additional teleoperation.

SFT surpasses its own teacher. After supervised finetuning, the distilled VLA outperforms both the unorchestrated base model and the live VLM orchestrator across the suite. An implication worth noting: a small 3B VLA can absorb compositional competence from a much larger foundation model through filtered self-distillation alone.

RL gains persist after accounting for data budget. For a fair comparison against residual RL finetuning of the base VLA, the authors run the baseline for more environment steps to compensate for the orchestration data used during SFT. Even so, GROD + SFT converges to higher success rates and lower time-to-success, averaged over twenty tasks and five seeds; on several tasks, SFT without any RL beats the RL-finetuned base VLA despite the latter collecting more data. The residual RL stage then consistently boosts success rate further across all tasks.

An appendix ablation shows the pipeline is robust to instruction style: restricting the VLM to pick/place-only commands performs comparably to free-form natural-language orchestration, indicating GROD tolerates coarse grounding.

Limitations and open questions

The results depend on several assumptions stated plainly by the authors. First, success detection relies on a ground-truth environment oracle, which does not exist outside simulation; replacing it with a VLM-based success detector remains an open question. Second, experiments are confined to simulation on the ALOHA platform; real-world transfer, where VLM latency matters most, is not demonstrated. Third, two appendix experiments yield negative results that constrain the method's scope: distilling the VLM orchestrator into the residual policy via advantage-weighted BC fails to help online RL due to distribution shift between offline orchestrated tuples and online rollouts, and mixing VLM-orchestrated episodes into online residual RL degrades evaluation performance relative to a no-VLM baseline—the residual policy learns corrections for orchestrated actions $a^{VLA-VLM}$ that do not match deployment-time corrections of $a^{VLA}$ alone. Finally, the framework operates off-policy only; extending it to on-policy self-distillation is left open, as is using the VLM to reset environments ("undo the task") for fully autonomous learning loops.

Conclusion

Eximo demonstrates that a frozen VLM planner can substitute for teleoperators when adapting a VLA to new long-horizon and reasoning tasks: VLM-guided exploration produces higher-quality data faster than the base policy, filtered behaviour cloning distills that competence into a standalone VLA that outperforms its orchestrator, and residual off-policy RL adds further consistent gains. The main open issues are the dependence on ground-truth success detection, validation limited to simulation, and the unresolved distribution-shift problem when orchestrating residual policies rather than the VLA itself.