---
title: Flow Map Reward Guidance (FMRG)
url: https://www.emergentmind.com/topics/flow-map-reward-guidance-fmrg
type: topic
---

# Flow Map Reward Guidance (FMRG)

Flow Map Reward Guidance (FMRG) denotes a line of methods that use reward information to steer flow-based generative or planning systems. In the most specific usage, introduced in "How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance" [2604.27147], FMRG is a training-free, single-trajectory guidance framework in which the flow map is used both to integrate a deterministic probability flow ODE and to compute the reward-aligned control signal. Recent usage also suggests a broader sense: reward information can be mapped onto a flow trajectory through a flow map, a value function, stochastic posterior sampling, or an explicitly reward-conditioned decoder, as in multimodal driving planning with FlowR2A [2606.24231]. The unifying theme is that reward is not treated merely as a terminal score, but as structure that modifies the transport itself.

## 1. Formal problem statements and theoretical scope

A standard formalization of reward guidance is the reward-tilted terminal law
\[
\tilde{\rho}_1(x)\propto e^{\lambda r(x)}\rho_1(x),
\]
or equivalently the search for trajectories that increase a terminal reward \(r(x)\) while remaining close to the pretrained model distribution. In the deterministic-flow setting, FMRG reframes this as an optimal-control problem:
\[
\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u)
\quad \text{s.t.} \quad
\dot{x}_t^u=b_t(x_t^u)+u_t.
\]
Here \(u_t\) is an additive control, \(b_t\) is the base flow field, and \(\lambda\) trades control effort against terminal reward [2604.27147].

A complementary stochastic formulation appears in work on reward guidance for flow and diffusion models. There, the exact target is again the reward-tilted measure, but the mathematically exact mechanism is the Doob \(h\)-transform with
\[
h_t(x)=\mathbb{E}\!\left[e^{\lambda r(X_1)}\mid X_t=x\right].
\]
If the true \(h_t\) is available, the guided probability-flow ODE has terminal marginal exactly equal to the reward-tilted distribution. This establishes that exact reward guidance is not merely heuristic ascent on a reward gradient, but a conditional-expectation problem over the model’s own posterior futures [2606.02884].

These two viewpoints are compatible rather than contradictory. The deterministic-control perspective emphasizes single-trajectory steering in few-step flows, whereas the Doob-transform perspective emphasizes exact probabilistic tilting. Much of the modern FMRG literature can be read as exploring approximations, relaxations, or algorithmic surrogates between these two poles.

## 2. Flow maps, look-ahead reward evaluation, and the canonical FMRG algorithm

The central object in FMRG is the flow map \(X_{s,t}(x_s)=x_t\), the solution operator associated with the ODE \(\dot{x}_t=b_t(x_t)\). In "How to Guide Your Flow" [2604.27147], the flow map is not only an acceleration device for few-step generation; it is also the mechanism by which perturbations at time \(t\) are related to terminal reward at time \(1\). The small-\(\lambda\) analysis yields the greedy guidance rule
\[
u_t^J(x)=\lambda \nabla X_{t,1}(x)^T \nabla r(X_{t,1}(x)),
\]
which replaces the unknown controlled flow map with the pretrained uncontrolled flow map and is second-order accurate in the sense that
\[
\|u_t^*(x)-u_t^J(x)\|=O(\lambda^2).
\]
This produces a deterministic, training-free, single-trajectory method.

Two practical variants are emphasized. **FMRG-J** uses the Jacobian-projected gradient \(\nabla X_{t,1}(x)^T \nabla r(X_{t,1}(x))\), while **FMRG-E** uses the Euclidean gradient \(\nabla r(X_{t,1}(x))\). The Jacobian is not a cosmetic addition: the paper shows that if the data lie on a manifold \(M\), then the Jacobian removes reward-gradient components orthogonal to that manifold. This is why FMRG-J is described as more manifold-preserving for complex neural rewards [2604.27147].

Algorithmically, FMRG uses operator splitting. A trajectory is first advanced by the base flow map,
\[
\tilde{x}_{t_{k+1}}=X_{t_k,t_{k+1}}(x_{t_k}),
\]
and then refined by one or more reward-gradient steps. The same learned flow map is therefore used for both propagation and guidance. This dual use is the defining structural feature of canonical FMRG [2604.27147].

A closely related formulation appears in "Test-time scaling of diffusions with flow maps" [2511.22688], where Flow Map Trajectory Tilting (FMTT) uses the look-ahead reward
\[
r_t(x)=t\,r(X_{t,1}(x)).
\]
In the flow-map case, the importance-weight dynamics simplify to
\[
A_t=\int_0^t r(X_{s,1}(\tilde{x}_s))\,ds,
\]
which supports either exact tilted sampling by importance weighting or principled search for high-reward local maximizers. This paper reports that FMTT achieves the best overall mean GenEval score among the tested flow-map methods, with mean score \(0.79\), and highlights strong performance for geometric constraints and VLM-based rewards [2511.22688].

## 3. Approximation error, reward hacking, and known failure modes

A major theme in the literature is that practical reward guidance is generally not equal to exact reward tilting. "Are we really tilting? The mechanics of reward guidance in flow and diffusion models" [2606.02884] shows that reward hacking arises from a specific approximation used in most practical guided samplers: finite-particle plug-in estimation of the Doob \(h\)-function. After the outer step size tends to zero, this does not merely introduce Monte Carlo noise; it changes the limiting dynamics themselves.

The paper isolates two failure modes. The first is **within-mode reward hacking**: in a single Gaussian target with quadratic reward, finite-particle guidance over-centers samples and contracts covariance too aggressively. The second is **mode-selection failure**: in Gaussian mixtures, the plug-in estimator cannot move mass toward distant high-reward modes because sampled endpoints remain concentrated in the local mode of the current state. In the symmetric two-mode example with step reward, the exact tilted distribution places probability
\[
\tilde p=(1+e^{-\lambda R})^{-1}
\]
on the correct mode, whereas any finite-\(k\) plug-in guidance leaves the correct-mode probability at
\[
\tilde p_1^{(k)}=\tfrac12.
\]

Within this analysis, FMRG itself appears as a deterministic endpoint-based surrogate:
\[
u_t^{\mathrm{FMRG}}(x)=\lambda\,X_{t,1}(x)^T r(X_{t,1}(x)),
\]
and for quadratic reward,
\[
u_t^{\mathrm{FMRG}}(x)=-2\lambda\,X_{t,1}(x)^T(X_{t,1}(x)-a).
\]
The paper argues that this structure is similar to the \(k=1\) plug-in method: both are single-endpoint approximations rather than exact \(h\)-transforms. In the Gaussian case, FMRG avoids the plug-in mean overshoot but still contracts covariance aggressively; in Gaussian mixtures, it inherits the same mode-selection failure because the deterministic endpoint map continues to favor the local mode [2606.02884].

Canonical FMRG also has an internal mitigation strategy. In its Gaussian toy analysis, greedy guidance collapses variance more aggressively than exact reward tilting, and the paper proposes **early stopping**—guiding only until \(t_{\mathrm{stop}}<1\), then finishing with the unguided flow—to preserve diversity while retaining reward gains [2604.27147]. A related remedy in the plug-in-guidance literature is a closed-form **reward damping schedule**, which corrects within-mode bias without extra inner samples, while **best-of-\(n\)** compensates for the mode-selection failure by exploiting multiple random initializations rather than repairing the dynamics themselves [2606.02884].

These analyses correct a common misconception: FMRG-style look-ahead guidance is principled, but it is not automatically exact reward tilting.

## 4. Value functions, stochastic posterior sampling, and scalable alignment

A second major branch of the literature replaces single-endpoint look-ahead with explicit estimation of the value function
\[
V_t^r(x_t)=\log \mathbb{E}_{z\sim p_{1|t}(\cdot\mid x_t)}[\exp(r(z))].
\]
The guided vector field then becomes
\[
u_t^r(x)=u_t(x)+b_t\nabla_{x_t}V_t^r(x_t),
\]
so the core algorithmic problem is posterior estimation rather than reward differentiation alone [2602.05993].

"Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps" [2602.05993] argues that deterministic flow maps are insufficient because they collapse uncertainty into a single future and therefore cannot estimate \(V_t^r\) faithfully. The paper introduces **Posterior Diamond Maps**, which sample directly from the posterior \(p_{1|t}(\cdot\mid x_t)\), and **Weighted Diamond Maps**, which create stochastic futures by renoising a deterministic flow map and then correcting the resulting estimator with local reward, score correction, and bridge/path terms. The resulting posterior Monte Carlo estimator,
\[
V_t^r(x_t)\approx \log\frac{1}{N}\sum_{k=1}^N \exp(r(z^k)),
\]
supports guided sampling, sequential Monte Carlo, and search. The paper reports that Posterior Diamond Maps produce a better Pareto frontier across reward scale and compute, and that Weighted Diamond Maps outperform Prompt Optimization, Best-of-\(N\), and ReNO on high-resolution text-to-image alignment [2602.05993].

"Meta Flow Maps enable scalable reward alignment" [2601.14430] pushes this logic further by learning a stochastic one-step posterior sampler
\[
X_{0,1}(\epsilon;t,x)\sim p_{1|t}(\cdot\mid x),
\]
with differentiable reparameterization. This yields direct Monte Carlo estimators of \(\nabla V_t(x)\), including
\[
\widehat{\nabla V_t(x)}=
\nabla_x \log\!\left( \frac{1}{N}\sum_{i=1}^{N} \exp(r(X_{0,1}(\epsilon^{(i)};t,x))) \right).
\]
Because posterior draws become amortized one-step operations rather than nested ODE or SDE rollouts, the paper reports that a single-particle steered-MFM sampler outperforms a Best-of-1000 baseline on ImageNet across multiple rewards at a fraction of the compute, and that even MFM-GF with \(N=1\) beats Best-of-1000 while using about \(100\times\) fewer NFEs [2601.14430].

This branch of work suggests a conceptual bifurcation inside FMRG. One path treats the flow map as a deterministic look-ahead operator. The other treats reward guidance as value-function estimation under the posterior over future endpoints.

## 5. Reinforcement learning, post-training, and transport-specific adaptations

FMRG has also become a template for reward-aligned training or post-training of flow models. These methods do not merely perturb trajectories at inference; they alter the model or its induced policy so that future trajectories become more reward-favorable.

"Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization" [2502.06061] turns conditional flow matching into a reward-weighted policy optimization problem. The offline loss
\[
\mathcal{L}_{\mathrm{RW-CFM}}
=
\mathbb{E}\!\left[w(x_1)\|v_t(x;\theta)-u_t(x\mid x_1)\|^2\right]
\]
induces a new data distribution
\[
p^{\mathrm{new}}(x_1)=\frac{w(x_1)q(x_1)}{Z}.
\]
In the online setting, repeated reward weighting yields
\[
q_\theta^N(x_1)\propto \exp(\tau N r(x_1))q(x_1),
\]
and the paper proves convergence to a Dirac delta at the reward maximizer in the unregularized limit. To prevent this collapse, ORW-CFM-W2 adds a Wasserstein-2 regularizer computed from vector-field differences. The paper reports, for one SD3 alignment benchmark, CLIP \(35.46\) and Diversity \(4.21\) for ORW-CFM-W2, compared with CLIP \(33.63\) and Diversity \(3.47\) for ORW-CFM without W2 [2502.06061].

"TempFlow-GRPO: When Timing Matters for GRPO in Flow Models" [2508.04324] addresses the mismatch between sparse terminal reward and the temporal structure of flow generation. Its **trajectory branching** mechanism concentrates stochasticity at a chosen timestep \(k\), creating an ODE-SDE-ODE hybrid path so that a terminal reward can be attributed to a specific perturbation. Its **noise-aware weighting** uses normalized \(\sigma_t\sqrt{\Delta t}\) to emphasize earlier, high-impact steps. The paper reports Geneval \(0.97\) at 4,400 steps, versus \(0.90\) for Flow-GRPO, and states that TempFlow-GRPO reaches about \(0.95\) around 2,000 steps while Flow-GRPO needs roughly 5,600 steps [2508.04324].

"Value Gradient Guidance for Flow Matching Alignment" [2512.05116] derives finetuning from optimal control. The central principle is
\[
\tilde v^\star(x,t)=-\frac{1}{\lambda}\nabla V(x,t),
\]
so the residual velocity of the finetuned model should match a value-gradient field. VGG-Flow implements this through a gradient-matching loss, a value-consistency loss, and a terminal boundary loss, with heuristic initialization from reward gradients of a one-step prediction. On Stable Diffusion 3, the paper reports improved reward with better diversity and prior preservation than ReFL and DRaFT, for example PickScore \(23.21\) versus base \(21.81\) [2512.05116].

"Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition" [2607.00535] adapts RL post-training to deterministic few-step flow-map generators. Its key device, **Anchored Stochastic Flow Map Composition (ASFMC)**, converts a deterministic map into a stochastic transition by transporting to an anchor time and sampling back from the corresponding conditional distribution while preserving the original marginal probability path. This makes GRPO-style likelihood-ratio optimization possible for models such as MeanFlow and sCM without retraining them as native stochastic samplers [2607.00535].

A more transport-specific inference-time adaptation appears in "PG-MAP: Joint MAP Optimization for Inference-Time Alignment of Diffusion and Flow-Matching Models" [2606.22958]. For diffusion, PG-MAP jointly optimizes conditioning \(c\) and latent state \(z_t\) under a forward-consistency coupling and optional reward tilt. For flow matching, however, the paper states that the effective specialization collapses to a latent-only variant, UG-FM, operating on the data side of the trajectory. This result is a useful negative boundary for FMRG: reward-guided control in flow matching does not always benefit from joint conditioning optimization [2606.22958].

## 6. Reward-conditioned action distributions in multimodal driving planning

A distinct but related use of FMRG appears in multimodal driving planning. "FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning" [2606.24231] addresses the tension between scoring-based methods, which receive dense reward supervision but rank only a fixed action vocabulary, and anchor-based methods, which generate proposals dynamically but are trained from a single ground-truth trajectory per scene. Its core move is to reframe simulation-based rewards from discriminative targets into generative conditions and to learn
\[
p(a\mid r),
\]
where \(a\) is a trajectory and \(r\) is a reward vector describing simulated outcome.

The paper constructs a dense action vocabulary \(\mathcal V_a\) of **8192 four-second trajectories** clustered from nuPlan trajectories. Each trajectory is rolled out in the NAVSIM simulator, producing dense \((a,r)\) pairs. A flow-matching decoder then learns the reward-conditioned action distribution using the interpolation
\[
a_t=t\,a+(1-t)\epsilon,\qquad \epsilon\sim\mathcal N(0,I),
\]
and the loss
\[
\mathcal L_{\mathrm{dec}}
=
\mathbb E_{t,\epsilon}\left[\left\|\hat a_\theta(a_t,t,c)-a\right\|^2\right].
\]
The reward encoder embeds each reward signal separately and injects the resulting condition vector into the decoder via AdaLN [2606.24231].

The reward set spans **safety**, **progress**, **comfort**, and **rule compliance**. It includes no at-fault collisions (NC), time-to-collision (TTC), ego progress (EP), history comfort (HC), drivable area compliance (DAC), driving direction compliance (DDC), traffic light compliance (TLC), and lane keeping (LK). The paper emphasizes that a single scalar score such as PDMS or EPDMS is too coarse, so it introduces per-timestep structure through a **TTC-time array** and an **ego-area array**, thereby exposing where a safety or rule-compliance violation occurs along the trajectory [2606.24231].

To balance hard and soft objectives, FlowR2A augments continuous rewards with Gaussian noise,
\[
\tilde r_k\sim \mathcal N(r_k,\sigma^2),
\]
using \(\sigma=0.05\), while sharpening safety and drivable-area signals through fine-grained temporal arrays. At inference, the learned \(p(a\mid r)\) becomes a controllable sampler through classifier-free reward guidance,
\[
g_\theta(a_t,t)=v_\theta(a_t,t,r_\emptyset)+w_g\left(v_\theta(a_t,t,r_{\mathrm{high}})-v_\theta(a_t,t,r_\emptyset)\right),
\]
with \(w_g=5\) by default, and through **anchored sampling**, which initializes denoising from a noisy version of any anchor trajectory rather than from pure noise [2606.24231].

Empirically, the paper reports substantially improved proposal quality and state-of-the-art benchmark performance. With **1 proposal**, FlowR2A achieves about **89.4 PDMS**; with **4 proposals**, **91.8 PDMS**; and over **all proposals**, **92.9 PDMS**, with mean PDMS over all proposals **89.4 ± 7.4**, compared with **77.9 ± 22.4** for iPad and **60.3 ± 33.1** for DiffusionDrive. On NAVSIM v1 navtest it reports **PDMS = 92.8**, with NC **98.8**, DAC **98.0**, TTC **96.0**, and EP **90.1**; on NAVSIM v2 navtest it reports **EPDMS = 88.9**, with NC **98.9**, DAC **98.1**, TTC **98.5**, and EP **91.5** [2606.24231].

This usage broadens the meaning of FMRG. Instead of steering a pretrained sampler toward a reward-tilted data distribution, FlowR2A learns a reward-conditioned action distribution whose multimodality is intrinsic to \(p(a\mid r)\). A plausible implication is that FMRG is becoming a domain-general design pattern: reward can be injected into a flow either as a control signal during sampling or as a conditioning variable in the learned transport itself.

Source: https://www.emergentmind.com/topics/flow-map-reward-guidance-fmrg