FlowPRO: Offline Reward-Free Fine-Tuning

• FlowPRO is a reward-free offline fine-tuning framework designed to enhance vision-language-action policies for real-robot manipulation.
• It uses a teleoperated intervention-and-rollback data collection strategy combined with RPRO, a proximalized preference optimization method, to convert sparse failure signals into robust training targets.
• Empirical results show significant improvements in success rates and completion times on challenging, long-horizon bimanual tasks compared to traditional methods.

FlowPRO is a reward-free, offline reinforced fine-tuning framework for flow-matching Vision-Language-Action (VLA) policies intended for real-robot deployment. It targets a specific post-training regime in which a base VLA policy already exists, but remains unreliable on difficult real-world edge cases, especially in long-horizon, contact-rich, and precision-sensitive manipulation. The framework couples a real-robot data collection procedure based on teleoperated intervention-and-rollback with a preference-optimization objective, RPRO, that is specialized to continuous flow-matching action heads. Its central claim is that post-training VLAs for reliable deployment is bottlenecked less by initial policy construction than by how sparse failure signals are converted into stable optimization targets without hand-designed rewards or learned critics (Wu et al., 3 Jun 2026).

1. Conceptual setting and scope

FlowPRO operates in a setting where a policy conditions on visual observations $o$ and a language instruction $l$, forming a state $s=(o,l)$, and outputs a continuous action chunk $a$. The framework assumes a VLA backbone with a flow-matching action head rather than an autoregressive token policy. This matters because many preference-optimization methods were originally designed for discrete sequence models, whereas FlowPRO addresses continuous robot control.

The framework is explicitly reward-free. It does not use an environment reward, a learned reward model, a critic, or a value function. Instead, supervision comes from paired human corrective data and standard supervised fine-tuning data. The paper positions this choice against three alternatives. Vanilla supervised fine-tuning uses only positive expert actions and therefore does not directly exploit failures. DAgger-style methods correct states visited by the current policy, but still use negative behavior only indirectly. Reward-based reinforcement learning, by contrast, is limited by the difficulty of real-world reward design and by the unreliability of offline critic learning in robotics.

A recurrent misconception is that FlowPRO is simply an instance of reward-based RL without an explicit scalar reward. The paper’s formulation is different: it is a preference-driven, offline reinforced fine-tuning method whose optimization signal is constructed from comparisons between preferred and dispreferred actions collected on real hardware, then combined with an SFT anchor.

2. Flow-matching policy model

The base policy is a conditional flow-matching model over action chunks. Given $s=(o,l)$, the action head learns a velocity field

$v_\theta(a_t,t \mid s),$

where $t \in [0,1]$, $\epsilon \sim \mathcal{N}(0,I)$, and the linear interpolant between noise and action is

$a_t = (1-t)\epsilon + ta.$

The target conditional velocity is

$u(a_t \mid a) := a - \epsilon.$

The per-sample flow-matching loss is

$l$0

The paper treats this as a tractable proxy for negative log-likelihood, up to a constant. This substitution is the mathematical bridge that allows preference optimization to be imported into a continuous flow-matching control setting without requiring explicit autoregressive log-probabilities.

At inference time, the action chunk is generated by integrating the learned velocity field from Gaussian noise. In the reported implementation, inference uses 10 denoising steps per chunk, and the action chunk horizon is

$l$1

corresponding to $l$2 s of motion at $l$3 Hz.

3. RPRO: proximalized preference optimization for flow-matching VLAs

The algorithmic core of FlowPRO is RPRO, short for Robotic Flow-matching Proximalized Preference Optimization. It begins from the DPO-style observation that a reward can be written in terms of the policy-reference log-ratio, then replaces the unavailable likelihood term with the flow-matching loss proxy. The resulting implicit reward is

$l$4

where $l$5 is a frozen reference policy.

This construction gives a signed preference score: if the current policy fits an action better than the reference, then $l$6 and $l$7. FlowPRO’s critique of plain Flow-DPO is that pairwise preference margins alone do not identify the absolute scale of these implicit rewards. In continuous action space, the optimizer can increase the margin by moving away from both the preferred and the dispreferred actions, a pathology the paper describes as reward hacking or likelihood underdetermination (Wu et al., 3 Jun 2026).

To address this, the paper imports the PRO decomposition into a flow-matching setting. The loss is

$l$8

The first term is contrastive: it enforces $l$9 as preferable to $s=(o,l)$0. The second is the proximal regularizer: it is minimized at $s=(o,l)$1, anchoring each action near the reference policy in absolute scale. The paper’s claim is that this proximal term removes the reward-hacking failure mode of plain Flow-DPO.

The full RPRO objective adds an SFT component: $s=(o,l)$2

One structural result is especially important. When $s=(o,l)$3, the contrastive gradient cancels exactly, leaving only the proximal regularizer and SFT term: $s=(o,l)$4 This allows true preference tuples and SFT-style identical pairs to be optimized under a single objective.

The paper also shows that the hyper-response term used in PRO for LLMs vanishes exactly in this continuous-action setting under absolute continuity: $s=(o,l)$5 That simplification is specific to the measure-theoretic structure of continuous action space.

4. Intervention-and-rollback data collection and Smooth Interpolation

FlowPRO’s data pipeline is built around teleoperated intervention-and-rollback on a real robot. During a rollout, when the operator observes erroneous or dangerous behavior, the system rewinds the robot to an earlier state $s=(o,l)$6. The segment executed by the policy from $s=(o,l)$7 to $s=(o,l)$8 is recorded as the negative trajectory $s=(o,l)$9. The operator then restores the scene to the rollback state and teleoperates the robot correctly from the same state, producing a preferred trajectory $a$0. A single intervention therefore yields a naturally paired sample

$a$1

These pairs are not directly usable for state-wise preference optimization because after the shared initial state the trajectories diverge. FlowPRO resolves this with Smooth Interpolation. For a state $a$2, the closest point $a$3 on $a$4 is found using the weighted distance

$a$5

with

$a$6

Here $a$7 is end-effector position, $a$8 is orientation, and $a$9 is normalized gripper width.

A synthetic preferred action chunk is then built by bridging from $s=(o,l)$0 to a transition point on $s=(o,l)$1. The bridge uses a cubic Bézier curve for position, Slerp for orientation, and linear interpolation for the gripper. With

$s=(o,l)$2

the Bézier position interpolation is

$s=(o,l)$3

while orientation is

$s=(o,l)$4

and gripper interpolation is

$s=(o,l)$5

For states already on $s=(o,l)$6, the paper sets $s=(o,l)$7. For SFT data, it likewise sets $s=(o,l)$8. The combined dataset is therefore unified as preference tuples and identical-pair tuples within one training representation (Wu et al., 3 Jun 2026).

The framework supplements this with batch mixing. In round $s=(o,l)$9, mini-batches draw $v_\theta(a_t,t \mid s),$0 from current preference data and $v_\theta(a_t,t \mid s),$1 from SFT data. For $v_\theta(a_t,t \mid s),$2, the ratio is $v_\theta(a_t,t \mid s),$3 current preference data, $v_\theta(a_t,t \mid s),$4 historical preference data, and $v_\theta(a_t,t \mid s),$5 SFT data. The intended effect is to emphasize new failures while preventing catastrophic forgetting.

5. Training protocol, experimental setup, and empirical performance

The reported system uses two base VLA backbones, PI0 and PI0.5, on a Dobot XTrainer bimanual platform with two 6-DoF arms, parallel-jaw grippers, three RGB cameras, and a Meta Quest 3 interface for teleoperation and rollback. The evaluation tasks are four long-horizon bimanual tasks: Pack, Cap, USB, and Case. Results are reported over 3 independent training seeds and 100 rollouts per seed with randomized initial placements (Wu et al., 3 Jun 2026).

Stage 1 is SFT for 60,000 optimizer steps using AdamW, initial learning rate $v_\theta(a_t,t \mid s),$6, 1,000 warmup steps, cosine decay over the next 30,000 steps to $v_\theta(a_t,t \mid s),$7, and a global batch size of 64 across 4 GPUs. Stage 2 is RPRO for $v_\theta(a_t,t \mid s),$8 rounds, each with 25,000 optimizer steps, initial learning rate $v_\theta(a_t,t \mid s),$9, 1,000 warmup steps, cosine decay over the next 15,000 steps to $t \in [0,1]$0, and global batch size 20. The reported RPRO hyperparameters are

$t \in [0,1]$1

The empirical results are strongest on success rate and completion time. On PI0, RPRO reaches $t \in [0,1]$2 on Pack, $t \in [0,1]$3 on Cap, $t \in [0,1]$4 on USB, and $t \in [0,1]$5 on Case. On PI0.5, it reaches $t \in [0,1]$6, $t \in [0,1]$7, $t \in [0,1]$8, and $t \in [0,1]$9, respectively. The paper states that RPRO is the top method on all 8 task-base strata, improves over TPO by $\epsilon \sim \mathcal{N}(0,I)$0–$\epsilon \sim \mathcal{N}(0,I)$1 percentage points, over PI0.6* by $\epsilon \sim \mathcal{N}(0,I)$2–$\epsilon \sim \mathcal{N}(0,I)$3 percentage points, and over vanilla DAgger by $\epsilon \sim \mathcal{N}(0,I)$4–$\epsilon \sim \mathcal{N}(0,I)$5 percentage points, while also achieving the shortest completion time in all table entries.

The reported significance tests are Bonferroni-corrected one-sided CMH tests, with $\epsilon \sim \mathcal{N}(0,I)$6 versus DAgger, $\epsilon \sim \mathcal{N}(0,I)$7 versus DAgger-Buffered, $\epsilon \sim \mathcal{N}(0,I)$8 versus PI0.6*, and $\epsilon \sim \mathcal{N}(0,I)$9 versus TPO.

The ablations are central to the paper’s interpretation. On Pack after one round of fine-tuning, removing SFT reduces success from $a_t = (1-t)\epsilon + ta.$0 to $a_t = (1-t)\epsilon + ta.$1 on ID/OOD evaluation. Removing PRO reduces it to $a_t = (1-t)\epsilon + ta.$2. Replacing RPRO with DPO+SFT reduces it to $a_t = (1-t)\epsilon + ta.$3. Plain DPO collapses to $a_t = (1-t)\epsilon + ta.$4. These results are used to argue that contrastive preference learning, proximal regularization, and the SFT anchor are all necessary.

6. Interpretation, limitations, and relation to adjacent flow-matching VLA work

FlowPRO’s central conceptual contribution is not merely the use of preference data, but the claim that preference optimization in continuous flow-matching control is ill-posed unless the absolute magnitude of the implicit reward is explicitly regularized. In the paper’s formulation, proximalization supplies a trust-region-like anchor around the reference policy, while SFT preserves positive-action fidelity. A common misunderstanding is that the preference term alone should suffice; the ablations argue against this, and the reward-dynamics analysis shows that DPO-style objectives can drive even preferred actions into strongly negative implicit-reward regions (Wu et al., 3 Jun 2026).

The method also has clear limitations. The experiments are all on a single bimanual robot platform. Failure detection and rollback remain human-triggered, and the rollback horizon $a_t = (1-t)\epsilon + ta.$5 is operator-selected. The framework further assumes that the teleoperated correction is preferable, that scene restoration to the rollback state is feasible, and that local interpolation between failed and corrected trajectories remains meaningful.

Within the broader literature on flow-matching VLAs, FlowPRO addresses post-training and deployment reliability rather than test-time inference latency. ProbeFlow, for example, targets the action-head bottleneck by adaptively reducing the number of Euler integration steps during action decoding, whereas FlowPRO modifies the offline fine-tuning objective and data pipeline. This suggests complementary axes of optimization in flow-matching robot policies: policy improvement after failure collection versus low-latency inference at deployment time (Fang et al., 18 Mar 2026).

Overall, FlowPRO is best understood as a framework for converting sparse real-robot failures into dense, stable, reward-free optimization signals for continuous generative policies. Its importance lies in showing that preference optimization for flow-matching VLAs requires not only pairwise ranking pressure, but also explicit control of absolute implicit-reward scale, a property the paper operationalizes through RPRO and validates on long-horizon bimanual manipulation (Wu et al., 3 Jun 2026).