FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

Abstract: Post-training Vision-Language-Action (VLA) models into policies that can be reliably deployed on real robots remains a major bottleneck. SFT and DAgger exploit failure signals only indirectly, and reward-based RL is bottlenecked by the difficulty of real-world reward design and of training reliable critics. We present FlowPRO, a reward-free offline reinforced fine-tuning framework for flow-matching VLAs. Algorithmically, we propose RPRO (Robotic Flow-matching Proximalized Preference Optimization), a preference-optimization objective tailored to the flow-matching action head of VLA models. RPRO pairs a contrastive optimizer with an explicit proximal regularizer that anchors the absolute magnitude of the implicit reward, thereby eliminating the reward-hacking failure mode of plain Flow-DPO. On the data side, a teleoperated intervention-and-rollback paradigm produces naturally paired positive and negative trajectories $(τ^w, τ^l)$ on a real robot from a single operator action; a Smooth Interpolation procedure, combined with batch mixing, then converts these sparse corrections into dense per-state supervision while preserving the base policy's capabilities. On four long-horizon bimanual tasks, FlowPRO attains the highest success rate, outperforming four representative baselines, and ablations confirm the contribution of each loss component.

• The paper presents FlowPRO, a reward-free fine-tuning framework that leverages the RPRO loss for per-state contrastive learning in continuous flow-matching VLAs.
• It employs teleoperated rollouts and smooth interpolation to generate dense synthetic preference signals, enhancing robust policy correction and improvement.
• Results on bimanual tasks indicate 3–7 pp success rate gains over baselines, demonstrating safe and rapid convergence suitable for real-world deployments.

FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization

Introduction and Motivation

Post-training control policies for Vision-Language-Action (VLA) models on real robot platforms is an outstanding challenge. Existing methods including supervised fine-tuning (SFT), DAgger-style correction, and reinforcement learning from rewards are limited either by weak exploitation of failure signals, scarcity of real-world reward functions, or poor critic reliability. Preference-based RL frameworks such as GRAPE, which utilize pairwise human preferences, tend to dilute per-state learning signals due to their trajectory-level contrast and are susceptible to likelihood underdetermination, leading to reward hacking. The presented approach, FlowPRO, directly addresses these limitations in VLAs built on continuous flow-matching action models, proposing the RPRO (Robotic Flow-matching Proximalized Preference Optimization) objective.

FlowPRO Framework and RPRO Algorithm

FlowPRO operates in a two-stage, fully offline, reward-free setting. The initial stage uses SFT to train a flow-matching policy backbone on task demonstrations. The second stage performs iterative preference-based fine-tuning using offline RL, where new preference data is obtained via a teleoperated intervention-and-rollback protocol.

Figure 1: Overview of the FlowPRO framework, including SFT base model training, teleoperated data collection with rollback, preference dataset construction, Smooth Interpolation for dense per-state supervision, and the RPRO optimization loop.

For data collection, when an impending failure is detected during rollouts, the operator triggers a rollback, resets the physical state, and provides a corrected teleoperated trajectory from the previous state. This yields paired positive ($\tau^w$) and negative ($\tau^l$) example trajectories for identical initial states. A critical innovation is the Smooth Interpolation technique, which synthesizes dense per-state preference tuples by interpolating between divergent trajectories, allowing the exploitation of contrastive learning even under limited human correction bandwidth.

The RPRO loss is an instantiation of Proximalized Preference Optimization (PRO) adapted to continuous-action flow-matching VLAs (contrast to DPO and GRAPE). The loss incorporates both:

• A per-state contrastive optimizer driving the policy toward preferred actions and away from dispreferred ones.
• An explicit proximal regularizer anchoring the magnitude of the implicit reward proxy and thereby eliminating reward hacking, where both positive and negative action likelihoods are driven down indiscriminately (the main failure mode of contrastive objectives in continuous control).

The RPRO loss exhibits the gradient-vanishing property: when preference pairs are identical, the contrastive gradient cancels, so only the trust-region proximal regularizer remains, allowing unified training on mixed preferences and SFT demonstrations without degenerate optimization.

Synthetic Preference Construction and Physical Plausibility

Smooth Interpolation is implemented with geometric algorithms: cubic Bézier interpolation for position, Slerp for orientation, and linear interpolation for gripper state, taking as anchor a phase transition point along the positive trajectory. This construction ensures the per-state synthetic preferences bridge the negative trajectory onto the positive one in a physically plausible, collision-free manner. Point selection for interpolation excludes unsafe regions (e.g., near collision events), and at deployment, the policy never reaches the dangerous tail thanks to earlier corrections.

Figure 2: Visualization of the geometric region spanned by smooth interpolated synthetic positive action chunks versus negative execution paths in a real manipulation task.

Experimental Setup

Experiments are performed on a dual-arm Dobot XTrainer platform using four challenging, long-horizon, bimanual manipulation tasks: cosmetic package insertion (Pack), pen-cap assembly (Cap), USB plug insertion (USB), and deformable pencil-case packing (Case).

Figure 3: Real-robot experimental setup for the four studied bimanual tasks.

The training loop alternates between policy deployment and preference data collection—with each iteration gathering paired trajectory corrections at failure points—and batch-mixing of new and historical preferences plus SFT examples for efficient and stable optimization. All policies are evaluated over 3 seeds and 100 randomized rollouts per setting.

Results: Comparative Analysis and Ablations

FlowPRO's RPRO loss delivers superior performance over conventional baselines—DAgger, batch-buffered DAgger, advantage-conditioned (PI0.6*), and GRAPE-style trajectory-level TPO—across all tasks and both PI0 and PI0.5 base policies. Notably:

• RPRO provides 3–7 percentage points (pp) higher success rates than state-of-the-art preference-based baselines using identical data.
• Success rates climb to $99\%$ on pack/cap and $93$–$95\%$ on USB/case after three RPRO iterations, outperforming the best non-RPRO baseline by 2–7pp per task-base-policy stratum.

  Figure 4: Per-iteration success rates for all four tasks initialized from the PI0.5 base policy, showing task learning curves dominated by RPRO.

Contrastive learning at the state level (with dense per-state supervision) is shown to be significantly more effective than trajectory-level contrast (TPO/GRAPE), which is susceptible to diluted signals and learning incompleteness. RPRO’s contrastive-proximal composition is critical: ablations removing either SFT or the proximal regularizer (reducing to vanilla DPO or DPO+SFT) produce catastrophic reward hacking and failure, with success collapsing to under 15%.

Figure 5: Ablation diagnostics on implicit reward dynamics and final success rates for main loss components; RPRO regularizer prevents divergence observed in other variants.

Practical and Theoretical Implications

Practically, FlowPRO demonstrates that high-performance, deployable robotics policies can be attained with minimal human effort, exploiting scarce teleoperated interventions to drive robust corrections. The sample efficiency and reliability of the scheme make it suitable for high-cost, low-volume data collection regimes prevalent in real-world robotics. Theoretically, adapting PRO to flow-matching action models in continuous control closes a crucial gap: it provides a provably stable and effective contrastive preference alignment paradigm immune to reward hacking.

The trust-region property induced by explicit proximal regularization guarantees that the policy cannot diverge arbitrarily from the reference policy, thereby ensuring stable and safe improvement—integral for real robot deployment. The smooth interpolation and per-state preference expansion methodology represents a general pipeline applicable to other continuous-control domains with scarce feedback.

Limitations and Future Directions

The study is primarily conducted on a single bimanual platform and thus lacks evaluation on higher-DOF mobile or dexterous platforms. The dependency on manual failure detection and intervention could be mitigated in future work by learning failure detectors for autonomous rollback triggering. Further, there is scope to explore self-supervised and semi-supervised extension of preference induction to reduce annotation and demonstration effort even further.

Conclusion

FlowPRO (RPRO loss + teleoperated preference collection + smooth interpolation) represents a robust, sample-efficient framework for reward-free post-training of flow-matching VLAs. It achieves state-of-the-art success rates and rapid convergence across a diverse suite of long-horizon manipulation tasks while rigorously addressing reward hacking and degenerate optimization found in prior contrastive alignment methods. Its design principles and observed empirical advantages provide a blueprint for scalable, deployable real-world robot learning from preference signals with minimal human supervision (2606.05468).