Experience Reflow in Reinforcement Learning

• Experience Reflow is a mechanism where on-policy experiences are geometrically rectified to improve learning stability and reward alignment in reinforcement learning.
• In ReFPO, the method implicitly and explicitly straightens on-policy action–noise pairs through advantage-weighted and uniform geometric updates.
• In CreFlow, corrective reflow aligns failed rollouts with successful prototypes using masked losses to focus on reward-relevant spatiotemporal regions.

Searching arXiv for papers on “Experience Reflow” and related Reflow/Rectified Flow concepts to ground the article. Experience Reflow denotes a family of mechanisms in which generated, on-policy, or within-condition experience is not merely consumed by an update rule but is geometrically rectified, reused, or redirected so that later optimization or sampling becomes straighter, stabler, or more reward-aligned. The phrase is not uniformly formalized across the literature. In ReFPO, it is a geometric interpretation of Flow Matching Policy Gradients, where on-policy action–noise pairs are repeatedly straightened through advantage-weighted Conditional Flow Matching updates (Wang et al., 19 Jun 2026). In CreFlow, it is an explicit corrective procedure in which failed embodied video rollouts are pulled toward within-group successful rollouts on reward-relevant spatiotemporal regions (Ni et al., 14 May 2026). Closely related Reflow work in generative modeling uses deterministic or optimized couplings, reflow distillation, and marginal-alignment objectives rather than RL experience in the narrow sense (Dai et al., 14 Jul 2025, Wang et al., 28 Jun 2026).

1. Terminological scope and core idea

The common structure across the technical uses of Experience Reflow is a closed loop between generated samples and the geometry of later updates. The “experience” being reflowed varies by domain. In online flow-based RL, it is the policy’s own on-policy action–noise pairs. In embodied video diffusion RL, it is the set of successful and failed rollouts produced under a shared task condition. In rectified-flow generative modeling, the analogous object is a coupling or trajectory family built from the current model and then reused to learn straighter paths (Wang et al., 19 Jun 2026, Ni et al., 14 May 2026, Dai et al., 14 Jul 2025).

Setting	What is reflowed	Mechanism
ReFPO	On-policy action–noise pairs	Implicit advantage-weighted rectification; explicit unweighted Reflow regularizer
CreFlow	Failed rollouts within a rollout group	Correction toward the within-group positive mean on a reward mask
Rectified-flow generation	Noise–data couplings or teacher trajectories	Trajectory straightening, data reuse, or marginal alignment

A central distinction is between implicit and explicit forms. In ReFPO, the paper states that the phrase “Experience Reflow” is not explicitly coined as a formal term, but it naturally describes the repeated, advantage-weighted straightening of on-policy experiences under the same Conditional Flow Matching geometry used for policy learning (Wang et al., 19 Jun 2026). In CreFlow, by contrast, Experience Reflow is explicit and operationalized as a corrective loss on negatives, using successful videos generated under the same condition as a low-variance positive prototype (Ni et al., 14 May 2026).

This shared vocabulary should not be confused with generic sample reuse. In both ReFPO and CreFlow, the operative object is not a replay buffer policy alone, but a structured transformation of experience geometry: straightening velocity fields in ReFPO and pulling failed trajectories toward positive prototypes in CreFlow.

2. Implicit Experience Reflow in flow-matching policy gradients

ReFPO begins from the observation that Flow Matching Policy Gradients can be rewritten in a way that reveals an implicit Reflow process. The policy is a flow policy defined by a velocity field $v_\theta(x,\tau; s)$ and a probability-flow ODE

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$

Conditional Flow Matching trains this field with

$$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$

where $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$ (Wang et al., 19 Jun 2026).

Because exact policy likelihoods are intractable for such flow policies, FPO replaces PPO’s likelihood ratio with the CFM proxy ratio

$$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$

The unclipped single-step gradient then takes the form

$$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$

This yields the geometric interpretation. When $A_t>0$, the update aligns $v_\theta$ with the straight direction $(a_t-\epsilon)$ along the interpolant, thereby straightening the path for high-advantage actions. When $A_t<0$, the direction flips and can distort or “un-straighten” paths associated with low-advantage actions (Wang et al., 19 Jun 2026).

The crucial point is that these action–noise pairs are generated on-policy and reused as the policy evolves. The method is therefore not merely fitting a static flow objective on a fixed dataset. It induces a closed-loop, advantage-weighted rectification of its own experiences. That is the sense in which Experience Reflow emerges as a property of FPO itself.

A common misconception is that this mechanism is a replay-buffer manipulation. The paper explicitly states otherwise: there is no special replay-buffer manipulation beyond storing $$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$0 pairs per action for inner optimization. The Reflow effect acts on the geometry of experience through update steps, not through an auxiliary replay stage (Wang et al., 19 Jun 2026).

3. Explicit geometric Reflow in ReFPO

ReFPO converts the implicit mechanism into an explicit regularizer by reusing the same per-sample CFM residual, but without advantage weighting:

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$1

The full objective is

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$2

with $$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$3 controlling regularization strength (Wang et al., 19 Jun 2026).

This construction gives the method two roles. First, the PPO-style surrogate still decides which trajectories are reinforced through the advantage-weighted proxy ratio. Second, the Reflow term provides a uniform geometric anchor across the fixed on-policy batch. The regularizer does not choose winners and losers; it straightens the velocity field itself.

The paper’s rationale for one-step inference follows directly from the local truncation error of coarse ODE integration:

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$4

The corresponding bound

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$5

shows that time variation and Jacobian magnitude drive one-step error. By penalizing deviations from the straight direction $$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$6 along interpolants, ReFPO empirically lowers curvature and improves one-step fidelity (Wang et al., 19 Jun 2026).

The same geometric simplification stabilizes training. For a frozen minibatch,

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$7

and a first-order expansion gives

$$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$8

Since large residuals drive large $$\dot{x}(\tau) = v_\theta(x(\tau), \tau; s), \qquad x(0)=\epsilon,\quad x(1)\approx a.$$9, the Reflow penalty smooths proxy-ratio changes and reduces clipping events in the PPO-style inner loop (Wang et al., 19 Jun 2026).

The empirical effects are substantial. On MuJoCo Playground, ReFPO* with $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$0 achieved 10-step reward $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$1, 1-step reward $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$2, Straightness $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$3, and Explosion rate $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$4, compared with FPO’s $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$5, $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$6, $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$7, and $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$8. On Humanoid Control, ReFPO*-1step improved MPJPE and success under all three conditioning regimes, including $$L^{CFM}_\theta(a;s)=\mathbb{E}_{\tau,\epsilon}\left[\left\|v_\theta(a_\tau,\tau;s)-(a-\epsilon)\right\|^2\right],$$9 versus $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$0 MPJPE for full joints and $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$1 versus $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$2 success for root+hands. In diagnostics on PointMass, FingerSpin, and BallInCup, reward drops in FPO coincide with large proxy-ratio spikes, whereas ReFPO keeps the ratio smaller and the rewards more stable (Wang et al., 19 Jun 2026).

Hyperparameter behavior is correspondingly geometric. The paper reports a sweet spot around $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$3 in MuJoCo Playground; too-small and too-large $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$4 both degrade the reward–stability trade-off. An advantage-weighted straightening variant underperforms the unweighted regularizer, which supports the interpretation that a consistent geometric anchor is more valuable than further entangling geometry with reward weights (Wang et al., 19 Jun 2026).

4. Corrective Experience Reflow in CreFlow

CreFlow uses the same phrase in a different but related sense. Under a fixed task condition $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$5, multiple generated rollouts share the same instruction and initial frame. CreFlow treats the successful rollouts in that group as i.i.d. samples from a condition-specific successful distribution and uses their empirical mean as a positive prototype:

$a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$6

For each failed rollout, the model forms a one-step prediction

$a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$7

and applies the masked corrective loss

$a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$8

Here $a_\tau=\alpha_\tau a+\sigma_\tau \epsilon$9 is a group-shared spatiotemporal mask extracted from violation witnesses returned by a compositional LTL monitor (Ni et al., 14 May 2026).

This construction matters because embodied manipulation video rewards are sparse and binary. CreFlow therefore does not rely on scalar reward alone. It first composes task requirements as finite-trace LTL clauses, builds entity-centric traces using SAM3, IDM, and a VLM, and then localizes failures to a shared mask over frames and pixels. The total objective is

$$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$0

where $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$1 is the credit-aware masked NFT loss and $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$2 is a local quadratic velocity-MSE surrogate to a frozen reference (Ni et al., 14 May 2026).

The resulting Experience Reflow is not simply “rewarding positives.” It explicitly regresses negatives toward the within-group positive mean on the violation-relevant support. That gives two advantages. First, masking confines updates to reward-relevant regions and prevents perturbations to unrelated pixels or frames. Second, the corrective branch has lower gradient variance than NFT’s reflected negative branch, because the $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$3-space formulation introduces an automatic $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$4 down-weighting and positive aggregation reduces covariance by $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$5 (Ni et al., 14 May 2026).

The reported results show both reward fidelity and downstream control gains. On a held-out 100-video set, the compositional LTL reward reached $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$6 accuracy and $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$7 F1 against human labels, and $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$8 accuracy with pairwise ranking accuracy $$\hat r_t^{FPO}(\theta)=\exp\!\left(L^{CFM}_{\theta_{old}}(a_t;s_t)-L^{CFM}_\theta(a_t;s_t)\right).$$9 against simulator labels. Across eight RoboTwin bimanual tasks, CreFlow improved execution success by $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$0 percentage points over the Vidar base, with average success $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$1 versus $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$2 for Vidar base and $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$3 for DiffusionNFT. In ablations on three tasks, credit-aware NFT alone reached $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$4 average success, corrective reflow alone $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$5, and the combined system $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$6 (Ni et al., 14 May 2026).

CreFlow therefore generalizes the notion of Experience Reflow from geometric straightening to masked correction: successful experience inside a rollout group becomes an explicit corrective direction for failed experience.

5. Relation to Reflow in generative modeling

Experience Reflow in RL inherits much of its geometry from the broader Reflow literature on rectified flows. In that literature, Reflow constructs deterministic couplings between noises and model-generated images, then reuses trajectory states across time to learn straighter dynamics. The benefit is twofold: deterministic couplings give learnable trajectories, and multi-time-scale data reuse acts like a distillation signal. The principal limitation is a distribution gap, because generated images used in the coupling deviate from the true data distribution, so repeated Reflow cycles can accumulate error (Dai et al., 14 Jul 2025).

Several later papers sharpen this picture. VRFNO replaces deterministic couplings from generated images with optimized couplings $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$7 learned by a joint encoder–velocity framework, adds a Historical Velocity Term, and reports state-of-the-art single-step and few-step image generation while avoiding the Reflow distribution gap (Dai et al., 14 Jul 2025). “Beyond Trajectory Matching” proves that trajectory matching alone underdetermines endpoint marginals, introduces a window-wise KL marginal-alignment regularizer, and derives a telescoping total-variation bound linking local marginal alignment to final-time distributional discrepancy (Wang et al., 28 Jun 2026). “Improving the Training of Rectified Flows” argues that a single Reflow iteration is sufficient under realistic settings, and with a U-shaped timestep distribution and LPIPS-Huber premetric improves CIFAR-10 1 NFE FID for prior 2-rectified flow from $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$8 to $$\frac{\partial}{\partial \theta}\big(\hat r_t^{FPO}(\theta)A_t\big) = - A_t \hat r_t^{FPO}(\theta)\nabla_\theta L^{CFM}_\theta(a_t;s_t).$$9 (Lee et al., 2024). “Simple ReFlow” revisits ReFlow training dynamics, coupling construction, and inference, achieving FIDs of $A_t>0$0 on CIFAR10, $A_t>0$1 on AFHQv2, $A_t>0$2 on FFHQ, and $A_t>0$3 on ImageNet-64 at $A_t>0$4 NFEs, with guided variants further lowering those scores (Kim et al., 2024).

A related use appears in molecular conformer generation, where reflow fine-tuning and distillation enable few-step and one-step sampling after SO(3)-Averaged Flow training. On GEOM-Drugs, one-step AvgFlow with a DiT model reaches COV-R $A_t>0$5 and AMR-R $A_t>0$6 Å, while full simulation with DiT-L reaches COV-R $A_t>0$7 and AMR-R $A_t>0$8 Å (Cao et al., 13 Jul 2025).

These works do not generally use the phrase Experience Reflow in the RL sense, but they provide the geometric substrate: straightening, deterministic or optimized recoupling, and distribution-level control. A plausible implication is that the RL formulations are best understood as importing Reflow’s path-geometry into settings where the coupling is generated online by a policy or rollout group.

6. Misconceptions, boundaries, and other uses of “Reflow”

One misconception is to equate Experience Reflow with conventional experience replay. The distinction is substantive. “Deep In-GPU Experience Replay” moves a replay buffer to the GPU, eliminating per-step CPU→GPU copies; at batch size $A_t>0$9 it reports a train-step speedup of $v_\theta$0 and an end-to-end speedup of $v_\theta$1 on $v_\theta$2M frames, but this is a systems optimization for buffer residency and data movement rather than a geometric rectification of experience (Parr, 2018).

Another misconception is that every paper containing the word “Reflow” belongs to the same conceptual family. Outside flow-based ML, the term is often simply a system name or a physical process. In mobile HCI, “Reflow” is a pixel-based runtime adaptation layer that personalizes mobile app layouts and yields, on average, $v_\theta$3 faster navigation after calibration (Wu et al., 2022). In reflow soldering process optimization, the central temperature field of a board moving through a furnace is modeled by an ODE, with best-fit $v_\theta$4 and Pearson $v_\theta$5 against measured data (Sui et al., 2022). In SMT self-alignment prediction, random forest regression predicts post-reflow component shift with average fitness $v_\theta$6, $v_\theta$7, and $v_\theta$8 for $v_\theta$9, $(a_t-\epsilon)$0, and rotation, respectively (Parviziomran et al., 2020). In cryptography, Reflow is a zero-knowledge multi-party signature scheme that aggregates unlinkable signatures and scales to thousands of participants (Roio et al., 2021).

The boundary question is therefore important. Experience Reflow, in its strict technical sense, refers to methods that transform model-generated experience into a corrective or straightening signal used by the same learning loop. ReFPO and CreFlow are the clearest instances. The former reveals that online flow-based policy gradients already induce an implicit advantage-weighted rectification of self-sampled experience; the latter shows that within-condition successful rollouts can be aggregated into an explicit corrective prototype for failed samples (Wang et al., 19 Jun 2026, Ni et al., 14 May 2026).

Open problems remain. ReFPO’s stability analysis is local and fixed-batch rather than global online theory, and extension to other diffusion or flow RL families is presented as promising but unproven. CreFlow assumes access to localized violation masks and accurate LTL specifications; its current monitor reasons primarily over image-plane states. The generative Reflow literature adds another unresolved axis: trajectory straightening alone does not guarantee marginal alignment, which suggests that future Experience Reflow methods may need both geometric and distributional control (Wang et al., 19 Jun 2026, Ni et al., 14 May 2026, Wang et al., 28 Jun 2026).