$π^{*}_{0.6}$: a VLA That Learns From Experience (2511.14759v2)

Abstract: We study how vision-language-action (VLA) models can improve through real-world deployments via reinforcement learning (RL). We present a general-purpose method, RL with Experience and Corrections via Advantage-conditioned Policies (RECAP), that provides for RL training of VLAs via advantage conditioning. Our method incorporates heterogeneous data into the self-improvement process, including demonstrations, data from on-policy collection, and expert teleoperated interventions provided during autonomous execution. RECAP starts by pre-training a generalist VLA with offline RL, which we call $π^{*}_{0.6}$, that can then be specialized to attain high performance on downstream tasks through on-robot data collection. We show that the $π^{*}_{0.6}$ model trained with the full RECAP method can fold laundry in real homes, reliably assemble boxes, and make espresso drinks using a professional espresso machine. On some of the hardest tasks, RECAP more than doubles task throughput and roughly halves the task failure rate.

• The paper presents a novel RL framework utilizing an advantage-conditioned VLA approach that significantly improves robotic manipulation tasks.
• It integrates heterogeneous pretraining with iterative on-robot RL and human interventions to boost sample efficiency and success rates.
• Quantitative evaluations show throughput doubling and failure rates halved across complex tasks like laundry folding, espresso making, and box assembly.

$π^{*}_{0.6}$: A VLA That Learns From Experience

Overview

This work presents $π^{*}_{0.6}$, a vision-language-action (VLA) model architecture and training paradigm designed for scalable and robust robotic skill acquisition via reinforcement learning (RL) from real-world experience, human demonstration, and expert intervention. The paper introduces a practical advantage-conditioned policy extraction mechanism for RL-compatible VLA policy improvement and demonstrates how this approach can substantially enhance sample efficiency, throughput, and robustness in complex, long-horizon robotic manipulation tasks such as laundry folding, espresso drink preparation, and industrial box assembly.

Figure 1: Some of the tasks learned by the proposed method: espresso making, box assembly, and laundry folding, each involving significant variability and dexterity.

Methodological Contributions

Full RL Pipeline for VLAs

Unlike prior work that focuses on supervised imitation or superficial RL finetuning of VLAs, the authors establish an integrated RL pipeline encompassing pretraining on heterogeneous demonstration and web data, followed by iterative on-robot RL stages using both autonomous rollouts and optional human corrective interventions. The central methodological innovation is a simple, efficient, and scalable advantage-conditioned policy extraction objective for VLA architectures with diffusion or flow-matching continuous action heads.

Policy and Value Function Architecture

$π^{*}_{0.6}$ is constructed atop the $\pi_{0.6}$ VLA model, extending the $\pi_{0.5}$ architecture and incorporating a 4B-parameter Gemma 3 VLM backbone. The action policy uses a large (860M parameter) flow-matching "action expert," which conditions on both language input and a binarized advantage indicator. The latter is derived from a powerful distributional, multi-task value function, trained to estimate normalized returns for a wide variety of tasks, itself leveraging a smaller but VLM-based architecture.

Figure 2: Schematic showing flow of information between the VLA policy and value function modules during training and action inference.

Advantage-Conditioned Policy Extraction

Instead of attempting to stably optimize stochastic policy gradients directly on large diffusion- or flow-based policies (which have intractable exact likelihoods and challenging log-probability estimation), the method performs supervised learning with an additional optimality indicator input. This indicator is computed via a Monte Carlo return estimate from the task-conditioned value function, thresholded to produce a binary variable. The policy is thus explicitly conditioned to indicate when actions are likely to outperform the reference policy, enabling scalable offline or off-policy RL improvement without explicit on-policy rollout dependence.

Value Function Visualization and Interpretability

The work demonstrates that the trained value function can accurately localize mistakes (value drops) and track progress (value increases) over diverse and challenging task executions, even in failure cases.

Figure 3: Visualization of value function outputs on example episodes—correctly indicating episodes’ success or failure, and frame-level error localization.

Experimental Evaluation

Robotic Platform and Task Suite

Experiments employ a static bimanual platform with dual 6-DoF arms and three vision inputs. Evaluation spans: (1) long-horizon bimanual laundry folding over diverse clothing types, (2) multi-stage espresso drink preparation on a commercial system, and (3) industrial box assembly with labeling and stacking, capturing variability spanning dynamic objects, deformable materials, and liquid handling.

Figure 4: Robotic setup used in experiments—two 6-DoF arms with vision and proprioceptive sensing.

Figure 5: Example task visualizations—laundry folding (multiple item types), box assembly, and espresso drink preparation.

Quantitative Results

Experiments demonstrate that the proposed method provides strong and consistent performance improvements over pre-trained and imitation learning-only baselines:

• Throughput (Fig. 6): For the most challenging tasks (diverse laundry and espresso), throughput more than doubles, i.e., the number of successful completions per hour more than doubles after applying the RL-based improvement cycle.
• Success Rate (Fig. 7): Failure rates are typically halved (e.g., diverse laundry performance increases from ~40% to over 80%; espresso from ~45% to nearly 90%, and similar trends are seen on box assembly subtasks).
• Iterative application further improves both speed and reliability after each cycle of on-robot data collection and RL-based policy updating (Figs. 8 & 9).

Figure 6: Throughput per hour for diverse tasks; proposed method more than doubles performance under challenging settings.

Figure 7: Absolute success rates for each stage, showing significant increases post-RL improvement.

Figure 8: Throughput improvements over successive RL iterations.

Figure 9: Success rate recovery and stability across RL iterations.

Comparative and Ablative Studies

The advantage-conditioned policy extraction mechanism significantly outperforms state-of-the-art off-policy methods such as Advantage-Weighted Regression (AWR) and PPO-style approaches in this setting (Fig. 10), both in terms of absolute success rate and (especially) throughput, attributed to more efficient sample utilization and direct compatibility with flow/diffusion architectures.

Figure 10: Comparison of policy extraction methods; proposed approach consistently yields higher throughput and success than AWR/PPO baselines.

The method is also validated in its ability to selectively remove failure modes—e.g., retraining with a small adversarial dataset can eliminate incorrect garment folding behavior with little additional data (Fig. 11).

Figure 11: RL-based iterative retraining efficiently removes specific task failure modes.

Visualization across five different tasks shows robust value function interpretability (Fig. 12).

Figure 12: Multi-task value function interpretability; episodic trajectory frames highlight how the value tracks successful and failed subtask behavior.

Implications and Future Directions

Theoretical Insights

The integration of binarized advantage conditioning, distributional value estimation, and large VLM action policies yields a practical and theoretically grounded improvement operator supporting off-policy learning at scale. The findings demonstrate that end-to-end RL with large VLAs is achievable on challenging and long-horizon real-world robotic tasks without requiring on-policy policy gradient optimization.

Practical Significance

The approach bridges the performance gap between demonstration-based and reinforcement learning-based methods in robotics, enabling real transfer, rapid adaptation, and removal of failure cases with a limited number of additional training cycles or targeted data augmentation. It offers a compelling path to deploying adaptable generalist robots in unstructured environments and industrial contexts at scale, where robustness and sample efficiency are critical.

Limitations and Open Challenges

The current pipeline remains semi-automatic: episode annotation, reward labeling, and human interventions are still required, albeit at a manageable rate, to ensure reliable reward feedback and remedy catastrophic failures. Exploration is largely policy-greedy except for the benefit of stochasticity and externally provided corrective actions. Fully online, concurrent RL (as opposed to iterated offline RL) remains future work. Extensions towards hierarchical reasoning, zero-shot generalization, and more sophisticated automated feedback or exploration strategies are all natural avenues for further progress.

Conclusion

$π^{*}_{0.6}$ provides a scalable recipe for RL-driven robotic skill acquisition in high-capacity generalist VLA models, integrating policy improvement via binarized, advantage-conditioned extraction, value-based interpretability, and heterogeneous data fusion. The method sets a new bar for practical, robust, and efficient deployment of generalist robotic control systems, and offers a modular, theoretically-grounded foundation for further advances in embodied AI.