Reward-Guided Conditional Flow Matching

• Reward-Guided CFM is a family of methods that biases the standard Conditional Flow Matching process by incorporating external reward signals during training, sampling, or candidate selection.
• Variants include reward-weighted objectives, test-time velocity modifications, learned candidate scorers, and RL-style post-training to align outputs with desired criteria.
• Empirical studies show enhanced efficiency and behavioral alignment in applications such as robot navigation and trajectory prediction, while cautioning against potential loss of diversity.

Searching arXiv for papers on reward-guided conditional flow matching and closely related guided CFM frameworks. Reward-guided Conditional Flow Matching (CFM) denotes a family of methods that preserve the regression-based structure of Conditional Flow Matching while biasing training, sampling, or downstream choice toward outputs that score highly under an external reward, score, or preference signal. In current usage, the label does not refer to a single canonical algorithm. Instead, recent work uses several distinct mechanisms: reward-weighted CFM objectives that tilt the effective data distribution, test-time modification of the learned velocity field by reward gradients, learned scoring functions that rank CFM-generated candidates, and RL-style post-training of pretrained conditional flow predictors. These variants all build on the same flow-matching premise: a time-dependent vector field transports samples from a simple source distribution to a target distribution without likelihood-based ODE training (Lipman et al., 2024, Fan et al., 9 Feb 2025).

1. Foundational formulation and the meaning of guidance

In flow matching, a deterministic flow is defined by the ODE

$$\frac{d}{dt}\psi_t(x)=u_t(\psi_t(x)),\qquad \psi_0(x)=x,$$

where $u_t$ is a velocity field and $X_t=\psi_t(X_0)$ transports a source distribution to a target distribution. The central computational idea is to avoid training by differentiating through ODE solves. Instead, one chooses a probability path $p_t$ and regresses a neural vector field toward an analytically tractable conditional target. In the standard CFM formulation,

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$

and for the simplest Gaussian or OT-style path one obtains

$$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$

A key theoretical property is that the conditional objective has the same gradients as the marginal FM objective in expectation, which is the basic justification for CFM training (Lipman et al., 2024).

Within this framework, “guidance” is broader than RL-style reward maximization. The general FM reference discusses conditional generation, classifier guidance, and classifier-free guidance. For a conditioning signal $Y$, the guided velocity can be written through a conditional path $p_{t|Y}(x|y)$, and for Gaussian paths the guide describes classifier guidance as

$$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$

together with classifier-free guidance

$$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$

The same source is explicit that reward-guided CFM is not introduced there as a standalone RL objective. Rather, the guide provides the general conditional and guided machinery that can be adapted when a reward is represented as a conditioning variable, a label, or a guidance signal (Lipman et al., 2024).

2. Boundary of the term: conditional CFM is not necessarily reward-guided

A recurring source of confusion is the equation of any conditional or goal-conditioned flow model with reward-guided CFM. Recent application papers show that this equivalence is false.

In FlowNav, CFM is used for goal-conditioned robot navigation, but the paper explicitly does not use reward guidance directly. There is no reward, cost, value function, preference model, or RL objective. The model is trained in a supervised way on robot trajectories and conditions on current observation, past observations with $u_t$0, a goal image $u_t$1, and an auxiliary temporal distance prediction head, while predicting a horizon of $u_t$2 normalized waypoint-like actions. Its navigation loss is

$u_t$3

which is conditioning plus auxiliary supervision rather than reward guidance. The paper’s practical motivation for replacing diffusion with CFM is efficiency: diffusion needs about $u_t$4 steps, whereas CFM achieves comparable performance in about $u_t$5 Euler steps, with roughly an $u_t$6 speedup overall (Gode et al., 2024).

FlowCast makes the same conceptual distinction in another domain. It applies Independent CFM to precipitation nowcasting in latent space, conditioning on past radar observations rather than on reward. The latent interpolation is

$u_t$7

with $u_t$8, and the model regresses

$u_t$9

The paper explicitly states that it does not implement reward-guided CFM, although it identifies conditioning or sampling-time guidance as natural extensions. Its comparison with diffusion again explains why reward-guided variants are attractive: CFM peaks around $X_t=\psi_t(X_0)$0–$X_t=\psi_t(X_0)$1 steps, while diffusion needs $X_t=\psi_t(X_0)$2–$X_t=\psi_t(X_0)$3 steps for good performance and degrades sharply below $X_t=\psi_t(X_0)$4 steps (Ribeiro et al., 12 Nov 2025).

This boundary matters conceptually. Reward-guided CFM is not defined by the presence of conditioning alone, but by the introduction of a signal that changes which samples are preferred, either during optimization, during ODE rollout, or during candidate selection.

3. Training-time reward weighting and distribution tilting

The most direct formulation of reward-guided CFM appears in Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization. Starting from the standard CFM objective,

$X_t=\psi_t(X_0)$5

the paper introduces an offline reward-weighted version

$X_t=\psi_t(X_0)$6

Its central result is distributional rather than merely algorithmic: idealized optimization induces

$X_t=\psi_t(X_0)$7

The online version, ORW-CFM, replaces the fixed data distribution by the current model distribution,

$X_t=\psi_t(X_0)$8

so repeated updates compound the reward tilt:

$X_t=\psi_t(X_0)$9

The paper proves a collapse result: if $p_t$0 reaches its maximum at $p_t$1, then as $p_t$2 the distribution converges to $p_t$3. This is the formal statement that unregularized online reward-weighted CFM can overoptimize and lose diversity (Fan et al., 9 Feb 2025).

To counter that effect, the same work introduces Wasserstein-2 regularization. Because direct $p_t$4 computation between flow models is intractable, the paper derives an upper bound in terms of vector fields:

$p_t$5

The resulting ORW-CFM-W2 objective is

$p_t$6

where the first term promotes reward maximization and the second keeps the fine-tuned model close to a reference. The paper explicitly interprets this as a flow-matching analogue of KL-regularized RL or trust-region policy improvement (Fan et al., 9 Feb 2025).

A mathematically adjacent, but not explicitly reward-conditioned, line is Weighted Conditional Flow Matching. W-CFM reweights source–target pairs with a Gibbs kernel,

$p_t$7

leading to

$p_t$8

The paper shows that this recovers the entropic OT coupling up to a multiplicative tilt of the marginals and establishes an equivalence to minibatch OT-CFM in the large-batch limit. It is explicit that W-CFM is not itself reward-conditioned, but it is also explicit that replacing transport cost by a negative reward would yield a reward-weighted CFM objective with the same change-of-measure structure. This makes W-CFM a theoretical template for “guidance through pairwise reweighting,” together with a warning: the weighting may distort the true marginals unless the Schrödinger potentials are effectively constant (Calvo-Ordonez et al., 29 Jul 2025).

4. Test-time reward guidance as velocity-field modification

A second major interpretation of reward-guided CFM keeps supervised CFM training largely intact and injects reward during ODE rollout. “Unified Generation-Refinement Planning: Bridging Flow Matching and Sampling-Based MPC” formulates this explicitly for robot planning. The base model learns a conditional flow over control sequences $p_t$9 with

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$0

and standard linear interpolation

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$1

Training uses

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$2

augmented by a terminal goal term

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$3

with $$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$4 (Mizuta et al., 2 Aug 2025).

The reward-guided step appears at inference:

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$5

The reward combines safety and terminal goal terms,

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$6

where

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$7

and

$$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$8

The experimental settings use $$\mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,Z,X_t\sim p_{t|Z}(\cdot|Z)}\,D\!\big(u_t(X_t|Z),u_t^\theta(X_t)\big),$$9, $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$0, markup factor $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$1, $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$2, and collision radius $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$3 m, with $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$4 trajectories sampled for each $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$5, giving $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$6 total trajectories. The paper emphasizes that reward is evaluated on estimated noise-free trajectories, unlike guided diffusion methods that may evaluate reward on noisy intermediate samples (Mizuta et al., 2 Aug 2025).

This reward-guided flow model is not used in isolation. It is coupled bidirectionally with MPPI: CFM provides multimodal candidate control trajectories, MPPI refines them under constraints, and the optimal MPPI trajectory warm-starts the next CFM rollout at $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$7. The inference schedule is fixed as $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$8. Quantitatively, the full CFM-MPPI system achieves the best overall performance on UCY, SDD, and a simulated crowd, including $$X_t=(1-t)X_0+tX_1,\qquad \mathcal{L}_{CFM}(\theta)=\mathbb{E}_{t,X_0,X_1}\big\|u_t^\theta(X_t)-(X_1-X_0)\big\|^2.$$9 collision rate on UCY and SDD. The same paper also notes important limits: reward guidance is not a hard safety guarantee, the method assumes exact obstacle positions and velocities, and it does not present a clean ablation isolating reward-guided CFM from unguided CFM (Mizuta et al., 2 Aug 2025).

5. Reward-guided selection among CFM-generated candidates

A third pattern leaves the flow generator untouched and places reward on candidate selection. Crowd-FM is exemplary. Its CFM component learns a distribution of collision-free trajectory primitives in the space of Bernstein polynomial control points,

$Y$0

with order $Y$1. The flow ODE is

$Y$2

and independent linear interpolation yields

$Y$3

A context-conditioned network $Y$4 is trained with the standard regression loss over LiDAR, dynamic obstacle states, and goal heading (Singha et al., 6 Feb 2026).

The reward-guided element is a separate learned score function. Given a candidate $Y$5 and context $Y$6, the scorer outputs

$Y$7

and test-time selection is

$Y$8

Training labels are defined by Euclidean proximity to an expert trajectory, and the scorer is optimized as a $Y$9-class classification problem with cross-entropy plus optimizer-cost regularization. This is reward-guided selection in the literal sense: the learned score does not shape the CFM loss directly, but it decides which of the safe CFM-generated candidates is retained (Singha et al., 6 Feb 2026).

Crowd-FM also introduces a distinct collision-cost guidance term during flow integration,

$p_{t|Y}(x|y)$0

followed by PRIEST refinement for kinodynamic feasibility. The paper is explicit that this geometric collision guidance is conceptually different from the learned score: the first is a direct safety term, the second a learned human-likeness preference derived from demonstrations. Empirically, the paper reports that vanilla CFM already beats DRL-VO, collision guidance can raise success from $p_{t|Y}(x|y)$1 to $p_{t|Y}(x|y)$2 in one ablation, closed-loop inference stays under $p_{t|Y}(x|y)$3 ms with CFM itself around $p_{t|Y}(x|y)$4 ms, and the learned scorer consistently reduces Human-Likeness Points relative to hand-tuned cost selection (Singha et al., 6 Feb 2026).

This suggests a useful taxonomy. In one branch, reward alters the vector field itself; in another, reward ranks or filters a multimodal set already generated by CFM. Both fit the broader idea of reward-guided CFM, but they act at different points in the pipeline and induce different failure modes. Selection-based methods are limited by candidate quality, while direct guidance risks distorting the flow trajectories themselves.

6. Reward-driven post-training and behavioral alignment

A fourth interpretation treats a pretrained CFM generator as a policy prior and aligns it by RL-style post-training. TIGFlow-GRPO is the clearest example. Its first stage builds a context token $p_{t|Y}(x|y)$5 from a Trajectory-Interaction-Graph with view-aware neighbor selection and edge-aware gated message passing, then trains a conditional flow model

$p_{t|Y}(x|y)$6

to predict future trajectories from Gaussian noise under context $p_{t|Y}(x|y)$7. This stage is supervised and captures multimodal coverage, but the paper states that it remains primarily data-fitting rather than reward-aligned (Jing et al., 26 Mar 2026).

The second stage converts deterministic flow rollout into a stochastic policy by reformulating ODE rollout as ODE-to-SDE sampling. The score is recovered from the pretrained velocity field,

$p_{t|Y}(x|y)$8

with diffusion coefficient

$p_{t|Y}(x|y)$9

and rollout step

$$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$0

$$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$1

This “policyization” permits exploration and permits GRPO updates over trajectory groups. For each scene, the paper samples $$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$2 rollouts and computes a group-relative advantage

$$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$3

leading to a clipped GRPO objective with a reference-regularization term that keeps the updated policy close to the frozen flow prior (Jing et al., 26 Mar 2026).

The reward is composite:

$$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$4

The paper emphasizes two components. The view-aware social reward makes interaction penalties direction-sensitive by partitioning neighbors into strong-view, weak-view, and rear-view sets based on heading-relative angle and distance. The map-aware reward uses a signed distance field and penalizes only the extra obstacle risk introduced by the prediction relative to the observed history. The stated effect is not collapse to a single mode, but alignment: high-reward modes gain probability mass, low-reward modes are suppressed, and stochastic rollout preserves multimodality (Jing et al., 26 Mar 2026).

The empirical evidence is correspondingly framed as behavioral alignment rather than pure imitation. On ETH/UCY, TIGFlow-GRPO reports average $$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$5 and average $$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$6; over longer horizons on ETH, $$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$7 at $$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$8 drops from $$\tilde{u}_t^{\theta,\phi}(x|y)=u_t^\theta(x)+b_t\nabla\log p_{Y|t}^\phi(y|x),$$9 for MoFlow to $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$0; and average collision rate drops from $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$1 to $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$2 across dataset-horizon pairs. On SDD, it reports $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$3 and $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$4, improving over MoFlow’s $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$5 and $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$6. The ablation study states that removing GRPO increases $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$7 to $$\tilde{u}_t^\theta(x|y)=(1-w)u_t^\theta(x|\emptyset)+w\,u_t^\theta(x|y).$$8, which the paper presents as evidence that supervised flow matching alone is insufficient to suppress low-quality samples (Jing et al., 26 Mar 2026).

Taken together, these lines of work define reward-guided CFM less as a single objective than as a design space. The reward may reweight training samples, regularize online fine-tuning, alter the velocity field during rollout, select among generated candidates, or align a pretrained flow by RL-style policy updates. The common element is that CFM supplies the transport model, while reward supplies a criterion of preference that is external to maximum-likelihood-style data fitting.