$π_\texttt{RL}$: Online RL Fine-tuning for Flow-based Vision-Language-Action Models (2510.25889v1)

Abstract: Vision-Language-Action (VLA) models enable robots to understand and perform complex tasks from multimodal input. Although recent work explores using reinforcement learning (RL) to automate the laborious data collection process in scaling supervised fine-tuning (SFT), applying large-scale RL to flow-based VLAs (e.g., $\pi_0$, $\pi_{0.5}$) remains challenging due to intractable action log-likelihoods from iterative denoising. We address this challenge with $\pi_{\text{RL}}$, an open-source framework for training flow-based VLAs in parallel simulation. $\pi_{\text{RL}}$ implements two RL algorithms: (1) {Flow-Noise} models the denoising process as a discrete-time MDP with a learnable noise network for exact log-likelihood computation. (2) {Flow-SDE} integrates denoising with agent-environment interaction, formulating a two-layer MDP that employs ODE-to-SDE conversion for efficient RL exploration. We evaluate $\pi_{\text{RL}}$ on LIBERO and ManiSkill benchmarks. On LIBERO, $\pi_{\text{RL}}$ boosts few-shot SFT models $\pi_0$ and $\pi_{0.5}$ from 57.6% to 97.6% and from 77.1% to 98.3%, respectively. In ManiSkill, we train $\pi_{\text{RL}}$ in 320 parallel environments, improving $\pi_0$ from 41.6% to 85.7% and $\pi_{0.5}$ from 40.0% to 84.8% across 4352 pick-and-place tasks, demonstrating scalable multitask RL under heterogeneous simulation. Overall, $\pi_{\text{RL}}$ achieves significant performance gains and stronger generalization over SFT-models, validating the effectiveness of online RL for flow-based VLAs.

• The paper introduces π_RL, an online RL framework that fine-tunes flow-based VLA models using Flow-Noise and Flow-SDE for precise log-likelihood estimation.
• It leverages a two-layer MDP formulation and hybrid ODE-SDE rollout to enhance high-frequency action generation and dexterous robotic control.
• Empirical results on LIBERO and ManiSkill benchmarks show significant performance gains, validating the framework's scalability and efficacy.

Online RL Fine-tuning for Flow-based Vision-Language-Action Models: The $\pi_\texttt{RL}$ Framework

Introduction

The paper presents $\pi_\texttt{RL}$, an open-source framework for online reinforcement learning (RL) fine-tuning of flow-based Vision-Language-Action (VLA) models, specifically targeting the $\pi_0$ and $\pi_{0.5}$ architectures. These models leverage flow matching for action generation, enabling high-frequency, dexterous robotic control conditioned on multimodal inputs. The central technical challenge addressed is the intractability of action log-likelihoods in flow-based models, which impedes the direct application of RL algorithms. The authors propose two solutions—Flow-Noise and Flow-SDE—that enable exact log-likelihood estimation and efficient policy optimization via PPO.

Figure 1: Two optimization methods in $\pi_{\text{RL}$: Flow-Noise (one-layer MDP with learnable noise) and Flow-SDE (two-layer MDP with ODE-to-SDE conversion).

Flow-based VLA Models and RL Integration

Flow-based VLAs generate action sequences by iteratively refining Gaussian noise through a learned vector field, conditioned on visual, linguistic, and proprioceptive inputs. The action generation process is governed by conditional flow matching (CFM), which aligns the predicted vector field with the ground-truth vector field. Unlike autoregressive models, flow-based VLAs produce continuous action chunks, facilitating dexterous manipulation.

The RL fine-tuning objective is to maximize the expected discounted return over environment interactions. However, the iterative denoising process complicates the computation of $\log \pi_\theta(a_t | s_t)$, which is essential for policy gradient methods. The deterministic nature of ODE-based sampling further limits exploration, necessitating stochasticity injection.

Flow-Noise: Learnable Stochasticity in Denoising

Flow-Noise introduces a learnable noise network into the denoising process, modeling each denoising step as a transition in a discrete-time MDP. The transition probability is parameterized as an isotropic Gaussian, with the mean given by the forward Euler update and the variance learned by the noise network. This enables exact computation of the joint log-likelihood over the denoising trajectory, which is used for policy optimization.

Figure 2: Illustration for the noise injection on the flow matching, exemplified by $\pi_{0.5}$ integrating image, language, and state information.

The noise network is trained jointly with the velocity field and discarded after fine-tuning, resulting in a deterministic policy for inference. The log-likelihood of the executed action is replaced by the joint log-likelihood of the denoised sequence, facilitating standard PPO updates.

Flow-SDE: ODE-to-SDE Conversion and Two-layer MDP

Flow-SDE converts the deterministic ODE denoising process into an equivalent SDE, introducing stochasticity while preserving the marginal action distribution. The SDE formulation incorporates a drift term (corrected velocity) and a diffusion term (noise injection), with the noise schedule analytically derived to maintain distributional equivalence.

The RL problem is reformulated as a two-layer MDP: the inner loop models the denoising process, and the outer loop models environment interaction. Rewards are only assigned at the completion of the denoising process, and transitions are Gaussian. To accelerate training, a hybrid ODE-SDE rollout strategy is employed, where a single denoising step is stochastic and the rest are deterministic.

Policy Optimization and Critic Design

Policy optimization is performed using PPO with Generalized Advantage Estimation (GAE). The chunk-based action generation of $\pi$-series models is accommodated by treating the entire action sequence as a macro-step. Critic networks are attached either after the VLM or the action expert, depending on the model variant. For $\pi_0$, the critic averages value estimates across the denoising trajectory; for $\pi_{0.5}$, it is directly attached to the VLM output.

Figure 3: Critic with the action expert, exemplified by $\pi_{0}$.

Empirical Results: LIBERO and ManiSkill Benchmarks

Experiments on LIBERO and ManiSkill demonstrate substantial performance gains over SFT baselines. On LIBERO, few-shot SFT models are boosted from 57.6% to 97.6% ($\pi_0$) and from 77.1% to 98.3% ($\pi_{0.5}$). Notably, $\pi_{\text{RL}$ achieves 94.0% on LIBERO-Long with only one SFT trajectory, surpassing the full-dataset SFT model. On ManiSkill, $\pi_{\text{RL}$ scales to 4,352 multitask combinations, improving success rates from 41.6% to 85.7% ($\pi_0$) and from 40.0% to 84.8% ($\pi_{0.5}$).

Figure 4: LIBERO Spatial task performance.

Figure 5: LIBERO evaluation curves.

Figure 6: Training curves for LIBERO.

Figure 7: Evaluation curves for LIBERO.

Figure 8: Training curves for ManiSkill.

Figure 9: Training curves for ManiSkill.

Figure 10: Evaluation curves for ManiSkill.

Figure 11: Evaluation curves for ManiSkill.

Figure 12: Evaluation curves for ManiSkill.

Figure 13: Evaluation curves for ManiSkill.

Ablation studies reveal that Flow-Noise marginally outperforms Flow-SDE in final performance but incurs higher computational cost due to trajectory recomputation. PPO consistently outperforms GRPO in convergence and stability. Critic placement after the VLM yields superior value estimation. Hyperparameter analysis highlights trade-offs between noise level, denoising steps, and action chunk size, with rollout-optimized parameters potentially destabilizing training.

Practical and Theoretical Implications

The $\pi_\texttt{RL}$ framework establishes a scalable methodology for RL fine-tuning of flow-based VLAs, overcoming the log-likelihood estimation barrier. The demonstrated gains in both in-distribution and multitask settings validate the efficacy of online RL for generalization beyond expert demonstrations. The framework's modularity enables integration with large-scale simulators and heterogeneous environments, supporting parallel training and efficient resource utilization.

Theoretically, the conversion of ODE-based flow matching to SDEs for RL compatibility is a significant advancement, enabling stochastic exploration without sacrificing distributional fidelity. The two-layer MDP formulation generalizes policy optimization for iterative generative models, and the hybrid ODE-SDE rollout strategy offers a pathway for further acceleration.

Limitations and Future Directions

Current limitations include suboptimal semantic generalization in OOD scenarios and the absence of real-world validation. The noise injection strategy, while effective, introduces train performance drops during ODE-to-SDE conversion; future work may leverage coefficients-preserving sampling or advanced mixed ODE-SDE rollouts. The impact of fine-tuning the VLM is limited in environments with low scene variability, suggesting that future research should target more diverse and challenging domains. Extending the framework to physical robots and improving semantic generalization remain open challenges.

Conclusion

$\pi_\texttt{RL}$ provides a principled and scalable solution for online RL fine-tuning of flow-based VLA models, enabling exact log-likelihood estimation and efficient policy optimization. The framework achieves strong empirical results on challenging benchmarks, supports large-scale multitask RL, and offers extensible methodologies for future research in robotic policy learning and generalization.