Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation

Abstract: State-of-the-art vision-language-action (VLA) models such as $π_{0.5}$ exhibit strong semantic understanding, instruction following and task behavior. However, when deployed on new robots, even minor mismatches in hardware configuration relative to pretraining can cause severe performance drops. Finetuning the VLA on in-domain expert data from the new embodiment improves performance on the expert task but leads to a loss in its original instruction following and behavioral priors. In this paper, we propose a self-supervised method that generates online interaction rollouts from the zero-shot VLA as additional training data for finetuning. Our experiments show this finetuning scheme yields strong multi-task policies that, on the target robot, (1) inherit prior tasks distilled from the zero-shot model, (2) enable generalist instruction following, while (3) learning new skills from expert data with improved sample efficiency. We demonstrate the success of our approach across test sets probing generalization on a real ALOHA robot and a new simulation benchmark in RoboTwin. Video results are available at https://self-supervised-control.pages.dev/

• The paper introduces joint expert- and self-supervised imitation, using rollouts from a frozen VLA on the target robot to rehearse pretrained behaviors without proprietary pretraining data.
• On ALOHA, only 14 minutes of expert teleoperation and 15 filtered self-demonstrations achieved 90% cube-picking success, 55% novel-object success, and 55% pick-and-place success while restoring instruction following.
• In RoboTwin, self-supervised rehearsal reduced catastrophic forgetting, raising retained-task performance to 70.6% and averaging 56.8%, within 0.2 points of a full-data rehearsal oracle.

Motivation and problem statement

Vision-language-action (VLA) models such as $\pi_{0.5}$ exhibit strong semantic grounding, instruction following, and broad behavioral priors from large-scale pretraining, but these capabilities are fragile under embodiment transfer. When deployed on a new robot with minor mismatches in camera pose or gripper configuration relative to pretraining hardware, zero-shot performance collapses: the policy localizes and reaches for prompted objects but fails to complete grasps. The standard remedy — fine-tuning on expert teleoperation data from the target robot — restores task competence on the demonstrated tasks but induces catastrophic forgetting of instruction following ("visual-action shortcut learning") and of pretrained behaviors not covered by the expert data. This paper addresses the practical post-training setting in which the downstream user receives a pretrained VLA checkpoint without access to its proprietary pretraining data or task list, and asks how to adapt the model to a new embodiment while retaining its generalist capabilities.

Method: joint expert- and self-supervised imitation

The approach builds on $\pi_{0.5}$'s architecture: a PaliGemma VLM backbone with SigLIP vision encoding, paired with a flow-matching action expert that outputs continuous action chunks. Training uses a dual objective combining cross-entropy over FAST-discretized action tokens with conditional flow matching on continuous actions; the authors reimplement FAST conditioning (absent from OpenPI) and find that this dual loss itself mitigates forgetting relative to flow-only training.

The core contribution is a self-supervised rehearsal dataset collected by rolling out the frozen base policy on the target robot, in the deployment scenes, under prompts drawn from the base policy's pretraining task families. At each decision step, teacher action chunks are sampled from the base policy's conditional flow distribution, executed on the robot, and the resulting observation–action pairs are recorded. Because both observations and actions come from executing the base policy's own actions on the deployment hardware, the replay data carries no embodiment or domain gap — unlike classical generative replay, which samples a pretrained prior offline, or trajectory-based DGR methods that train a separate generator. This constitutes self-distillation: the frozen VLA is its own trajectory generator, and its rollouts serve as pseudo-targets even though many rollouts fail to complete the task.

The fine-tuned policy is optimized jointly as $$\mathcal{L}(\theta) = \mathcal{L}_{\text{ES}}(\theta) + \lambda\,\mathcal{L}_{\text{SS}}(\theta)$$, mixing expert-supervised (ES) demonstrations for new skills with self-supervised (SS) rehearsal data at a 1:1 ratio with uniform task sampling; the authors report that imbalanced mixtures produce erratic behavior on hardware, and that unsafe self-demos are manually filtered before distillation. Full fine-tuning of both the VLM and action expert outperforms LoRA and selective-freezing variants in their ablations, which is notable given that parameter-efficient tuning is often advocated for continual learning.

Real-robot results on ALOHA

Experiments use a stationary ALOHA-1 platform with three RealSense cameras, evaluating success rate (SR) and pre-/post-grasp instruction following (IF). Three findings stand out:

1. Zero-shot failure is an embodiment gap, not a capability gap. Zero-shot $\pi_{0.5}$ reaches prompted objects 70–80% of the time but achieves 0% success across all pick-up and pick-and-place tasks, attributed to gripper and camera-pose mismatches.
2. Expert-only fine-tuning forgets pretrained behavior. Multi-task ES training recovers grasping but destroys place behavior: 0% post-grasp IF and 0% SR on pick-and-place, since every expert demonstration ends after the pick.
3. Joint ES+SS training recovers both. With only 14 minutes of expert teleoperation plus 15 filtered self-demos, the method attains 90% SR on expert cube picking, 55% SR on novel-object picking (grasping every object it localizes), 90% post-grasp IF and 55% SR on pick-and-place — recovering place behavior purely by distillation, from 0% to 55% with no expert place demonstrations. On laundry, it reaches 40% partial SR against noisy self-demo priors.

On the contact-rich bimanual caterpillar gear-insertion benchmark — explicitly outside the base policy's pretraining distribution — adding self-supervision improves success from 30% to 90% on trained gear colors and yields 100% IF with 30% SR on unseen gear colors versus 0% for baselines, indicating improved sample efficiency beyond simple retention.

A particularly instructive comparison evaluates held-out skill families absent from both mixtures. On "Push objects," the ES+SS policy executes push motions on 70% of trials (60% correct-object SR), while the fully-expert-supervised ES+ES policy picks up objects instead 35% of the time, defaulting to the motion chain it was trained on. This suggests expert supervision on all tasks can overwrite action priors rather than adapt them, whereas self-demos leave unrelated priors intact. Against the ES+ES upper bound — which collects human teleoperation for every task — the ES+SS method matches instruction following throughout and trails modestly on success (90% vs. 95% on $T_{ES}$; 40% vs. 90% partial SR on laundry, where noisy self-demos limit distillation quality).

Simulation results on RoboTwin

Because zero-shot $\pi_{0.5}$ scores 0% in simulation, the authors first mid-train a RoboTwin base policy on 10 tasks (90.8% average), then post-train on two new stacking tasks while restricting access to mid-training data. Naive multi-task ES fine-tuning learns stacking well (93%) but forgets catastrophically, dropping $T_{SS}$ performance to 16.6%. Joint ES+SS fine-tuning with just 10 self-demo episodes per task recovers $T_{SS}$ to 70.6% while also improving new-task performance from 93% to 98%. Across all four test splits, the method averages 56.8%, within 4 points of the frame-wise rehearsal oracle that assumes full access to stored mid-training data (57.0–60.8%), and 13.5 points above multi-task ES. Ablations show the 1:1 replay ratio is critical (proportional sampling drops retention from 69.2 to 56.4 on validation), uniform task sampling beats task-balanced weighting, and halving the buffer costs 12.4 points of retention. Notably, sub-optimal self-demos match an oracle that re-collects expert demos for the same tasks, so there is little justification for paying the cost of re-collection.

Limitations and open questions

The evaluation is restricted to a single base model ($\pi_{0.5}$), although the authors argue the data-based recipe transfers to other foundation models. Prompt curation for self-supervised collection is manual, and undesired or unsafe behaviors inherited from noisy rollouts must be filtered by hand; automation of prompt selection and filtering is left open. Failure modes include repeated pick-place cycles (no terminal signal exists because neither data source shows post-placement states) and inherited artifacts such as dragging container edges during laundry. The causal mechanism remains unresolved: the paper does not disentangle whether gains arise from the state distribution traversed during rollouts or from the distilled action targets themselves. Whether rehearsing genuine pretraining data on a different embodiment would outperform self-generated rollouts is likewise untested, as is the effect of task overlap between ES and SS sets.

Conclusion

This paper demonstrates that a pretrained VLA's own generative rollouts, executed on the target robot, constitute effective rehearsal data for post-training: they close embodiment gaps when mixed with a small amount of expert data, recover forgotten behaviors without any corresponding demonstrations, preserve held-out skill families, and approach the performance of full-data rehearsal oracles without access to pretraining datasets. The result reframes VLA pretraining as useful not only for initialization but for on-demand prior generation on arbitrary downstream hardware.