Fine-tuning pre-trained vision-language-action policies with online rewards

Determine a robust and sample-efficient procedure to fine-tune pre-trained vision-language-action policy models using online reward feedback during robot interaction, so that these models can effectively learn new manipulation tasks in real-world settings without requiring task-specific demonstrations.

Background

The paper proposes ReWiND, a framework that learns a language-conditioned reward model and uses it to pre-train and then fine-tune robot policies for unseen tasks. While ReWiND demonstrates substantial gains in simulation and real-world experiments, the authors note that stronger policy architectures—such as pre-trained vision-language-action (VLA) models—could further improve performance.

However, transitioning these large pre-trained policies to online reinforcement learning settings is challenging. The authors explicitly state that the best method to fine-tune such models with online rewards is unresolved, citing practical bottlenecks like computational cost and training time during real-world interaction. This motivates an open problem focused on developing an effective methodology for online reward-based fine-tuning of VLA models.

References

However, the best way to fine-tune these models with online rewards remains an open challenge ~\citep{nakamoto2024steering, guo2025improvingvisionlanguageactionmodelonline}.

— ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations  (2505.10911 - Zhang et al., 16 May 2025) in Section 6: Limitations

While this simple approach has enabled significant advances for robotic manipulation, finetuning of VLA policies for learning new tasks still remains an open problem.

— EXIMO: VLM Guided Exploration of VLA Policies  (2608.19891 - Sukhija et al., 20 Aug 2026) in Abstract

Therefore, how to develop an efficient real-world RL fine-tuning method for contact-rich manipulation with tactile feedback is still an open-problem.

— PEARS: Physical-Prior-Guided Efficient Adaptation via Failure Reasoning and Diffusion Steering for Tactile Manipulation  (2610.08784 - Song et al., 6 Oct 2026) in Section 1, Introduction