Flow Map Reward Guidance (FMRG)

• Flow Map Reward Guidance (FMRG) is a framework that integrates reward information into flow maps, steering generative and planning processes toward high-reward outcomes.
• It combines deterministic control through look-ahead guidance with stochastic approaches such as value function estimation and posterior sampling.
• Practical variants like FMRG-J and FMRG-E implement strategies to balance reward gains with diversity while mitigating issues like covariance collapse and mode-selection failure.

Flow Map Reward Guidance (FMRG) denotes a line of methods that use reward information to steer flow-based generative or planning systems. In the most specific usage, introduced in "How to Guide Your Flow: Few-Step Alignment via Flow Map Reward Guidance" (Huang et al., 29 Apr 2026), FMRG is a training-free, single-trajectory guidance framework in which the flow map is used both to integrate a deterministic probability flow ODE and to compute the reward-aligned control signal. Recent usage also suggests a broader sense: reward information can be mapped onto a flow trajectory through a flow map, a value function, stochastic posterior sampling, or an explicitly reward-conditioned decoder, as in multimodal driving planning with FlowR2A (Li et al., 23 Jun 2026). The unifying theme is that reward is not treated merely as a terminal score, but as structure that modifies the transport itself.

1. Formal problem statements and theoretical scope

A standard formalization of reward guidance is the reward-tilted terminal law

$$\tilde{\rho}_1(x)\propto e^{\lambda r(x)}\rho_1(x),$$

or equivalently the search for trajectories that increase a terminal reward $r(x)$ while remaining close to the pretrained model distribution. In the deterministic-flow setting, FMRG reframes this as an optimal-control problem: $$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$ Here $u_t$ is an additive control, $b_t$ is the base flow field, and $\lambda$ trades control effort against terminal reward (Huang et al., 29 Apr 2026).

A complementary stochastic formulation appears in work on reward guidance for flow and diffusion models. There, the exact target is again the reward-tilted measure, but the mathematically exact mechanism is the Doob $h$-transform with

$$h_t(x)=\mathbb{E}\!\left[e^{\lambda r(X_1)}\mid X_t=x\right].$$

If the true $h_t$ is available, the guided probability-flow ODE has terminal marginal exactly equal to the reward-tilted distribution. This establishes that exact reward guidance is not merely heuristic ascent on a reward gradient, but a conditional-expectation problem over the model’s own posterior futures (Dandapanthula et al., 1 Jun 2026).

These two viewpoints are compatible rather than contradictory. The deterministic-control perspective emphasizes single-trajectory steering in few-step flows, whereas the Doob-transform perspective emphasizes exact probabilistic tilting. Much of the modern FMRG literature can be read as exploring approximations, relaxations, or algorithmic surrogates between these two poles.

2. Flow maps, look-ahead reward evaluation, and the canonical FMRG algorithm

The central object in FMRG is the flow map $X_{s,t}(x_s)=x_t$, the solution operator associated with the ODE $r(x)$0. In "How to Guide Your Flow" (Huang et al., 29 Apr 2026), the flow map is not only an acceleration device for few-step generation; it is also the mechanism by which perturbations at time $r(x)$1 are related to terminal reward at time $r(x)$2. The small-$r(x)$3 analysis yields the greedy guidance rule

$r(x)$4

which replaces the unknown controlled flow map with the pretrained uncontrolled flow map and is second-order accurate in the sense that

$r(x)$5

This produces a deterministic, training-free, single-trajectory method.

Two practical variants are emphasized. FMRG-J uses the Jacobian-projected gradient $r(x)$6, while FMRG-E uses the Euclidean gradient $r(x)$7. The Jacobian is not a cosmetic addition: the paper shows that if the data lie on a manifold $r(x)$8, then the Jacobian removes reward-gradient components orthogonal to that manifold. This is why FMRG-J is described as more manifold-preserving for complex neural rewards (Huang et al., 29 Apr 2026).

Algorithmically, FMRG uses operator splitting. A trajectory is first advanced by the base flow map,

$r(x)$9

and then refined by one or more reward-gradient steps. The same learned flow map is therefore used for both propagation and guidance. This dual use is the defining structural feature of canonical FMRG (Huang et al., 29 Apr 2026).

A closely related formulation appears in "Test-time scaling of diffusions with flow maps" (Sabour et al., 27 Nov 2025), where Flow Map Trajectory Tilting (FMTT) uses the look-ahead reward

$$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$0

In the flow-map case, the importance-weight dynamics simplify to

$$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$1

which supports either exact tilted sampling by importance weighting or principled search for high-reward local maximizers. This paper reports that FMTT achieves the best overall mean GenEval score among the tested flow-map methods, with mean score $$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$2, and highlights strong performance for geometric constraints and VLM-based rewards (Sabour et al., 27 Nov 2025).

3. Approximation error, reward hacking, and known failure modes

A major theme in the literature is that practical reward guidance is generally not equal to exact reward tilting. "Are we really tilting? The mechanics of reward guidance in flow and diffusion models" (Dandapanthula et al., 1 Jun 2026) shows that reward hacking arises from a specific approximation used in most practical guided samplers: finite-particle plug-in estimation of the Doob $$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$3-function. After the outer step size tends to zero, this does not merely introduce Monte Carlo noise; it changes the limiting dynamics themselves.

The paper isolates two failure modes. The first is within-mode reward hacking: in a single Gaussian target with quadratic reward, finite-particle guidance over-centers samples and contracts covariance too aggressively. The second is mode-selection failure: in Gaussian mixtures, the plug-in estimator cannot move mass toward distant high-reward modes because sampled endpoints remain concentrated in the local mode of the current state. In the symmetric two-mode example with step reward, the exact tilted distribution places probability

$$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$4

on the correct mode, whereas any finite-$$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$5 plug-in guidance leaves the correct-mode probability at

$$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$6

Within this analysis, FMRG itself appears as a deterministic endpoint-based surrogate: $$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$7 and for quadratic reward,

$$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$8

The paper argues that this structure is similar to the $$\min_u \int_0^1 \frac{\|u_t\|^2}{2\lambda}\,dt - r(x_1^u) \quad \text{s.t.} \quad \dot{x}_t^u=b_t(x_t^u)+u_t.$$9 plug-in method: both are single-endpoint approximations rather than exact $u_t$0-transforms. In the Gaussian case, FMRG avoids the plug-in mean overshoot but still contracts covariance aggressively; in Gaussian mixtures, it inherits the same mode-selection failure because the deterministic endpoint map continues to favor the local mode (Dandapanthula et al., 1 Jun 2026).

Canonical FMRG also has an internal mitigation strategy. In its Gaussian toy analysis, greedy guidance collapses variance more aggressively than exact reward tilting, and the paper proposes early stopping—guiding only until $u_t$1, then finishing with the unguided flow—to preserve diversity while retaining reward gains (Huang et al., 29 Apr 2026). A related remedy in the plug-in-guidance literature is a closed-form reward damping schedule, which corrects within-mode bias without extra inner samples, while best-of-$u_t$2 compensates for the mode-selection failure by exploiting multiple random initializations rather than repairing the dynamics themselves (Dandapanthula et al., 1 Jun 2026).

These analyses correct a common misconception: FMRG-style look-ahead guidance is principled, but it is not automatically exact reward tilting.

4. Value functions, stochastic posterior sampling, and scalable alignment

A second major branch of the literature replaces single-endpoint look-ahead with explicit estimation of the value function

$u_t$3

The guided vector field then becomes

$u_t$4

so the core algorithmic problem is posterior estimation rather than reward differentiation alone (Holderrieth et al., 5 Feb 2026).

"Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps" (Holderrieth et al., 5 Feb 2026) argues that deterministic flow maps are insufficient because they collapse uncertainty into a single future and therefore cannot estimate $u_t$5 faithfully. The paper introduces Posterior Diamond Maps, which sample directly from the posterior $u_t$6, and Weighted Diamond Maps, which create stochastic futures by renoising a deterministic flow map and then correcting the resulting estimator with local reward, score correction, and bridge/path terms. The resulting posterior Monte Carlo estimator,

$u_t$7

supports guided sampling, sequential Monte Carlo, and search. The paper reports that Posterior Diamond Maps produce a better Pareto frontier across reward scale and compute, and that Weighted Diamond Maps outperform Prompt Optimization, Best-of-$u_t$8, and ReNO on high-resolution text-to-image alignment (Holderrieth et al., 5 Feb 2026).

"Meta Flow Maps enable scalable reward alignment" (Potaptchik et al., 20 Jan 2026) pushes this logic further by learning a stochastic one-step posterior sampler

$u_t$9

with differentiable reparameterization. This yields direct Monte Carlo estimators of $b_t$0, including

$b_t$1

Because posterior draws become amortized one-step operations rather than nested ODE or SDE rollouts, the paper reports that a single-particle steered-MFM sampler outperforms a Best-of-1000 baseline on ImageNet across multiple rewards at a fraction of the compute, and that even MFM-GF with $b_t$2 beats Best-of-1000 while using about $b_t$3 fewer NFEs (Potaptchik et al., 20 Jan 2026).

This branch of work suggests a conceptual bifurcation inside FMRG. One path treats the flow map as a deterministic look-ahead operator. The other treats reward guidance as value-function estimation under the posterior over future endpoints.

5. Reinforcement learning, post-training, and transport-specific adaptations

FMRG has also become a template for reward-aligned training or post-training of flow models. These methods do not merely perturb trajectories at inference; they alter the model or its induced policy so that future trajectories become more reward-favorable.

"Online Reward-Weighted Fine-Tuning of Flow Matching with Wasserstein Regularization" (Fan et al., 9 Feb 2025) turns conditional flow matching into a reward-weighted policy optimization problem. The offline loss

$b_t$4

induces a new data distribution

$b_t$5

In the online setting, repeated reward weighting yields

$b_t$6

and the paper proves convergence to a Dirac delta at the reward maximizer in the unregularized limit. To prevent this collapse, ORW-CFM-W2 adds a Wasserstein-2 regularizer computed from vector-field differences. The paper reports, for one SD3 alignment benchmark, CLIP $b_t$7 and Diversity $b_t$8 for ORW-CFM-W2, compared with CLIP $b_t$9 and Diversity $\lambda$0 for ORW-CFM without W2 (Fan et al., 9 Feb 2025).

"TempFlow-GRPO: When Timing Matters for GRPO in Flow Models" (He et al., 6 Aug 2025) addresses the mismatch between sparse terminal reward and the temporal structure of flow generation. Its trajectory branching mechanism concentrates stochasticity at a chosen timestep $\lambda$1, creating an ODE-SDE-ODE hybrid path so that a terminal reward can be attributed to a specific perturbation. Its noise-aware weighting uses normalized $\lambda$2 to emphasize earlier, high-impact steps. The paper reports Geneval $\lambda$3 at 4,400 steps, versus $\lambda$4 for Flow-GRPO, and states that TempFlow-GRPO reaches about $\lambda$5 around 2,000 steps while Flow-GRPO needs roughly 5,600 steps (He et al., 6 Aug 2025).

"Value Gradient Guidance for Flow Matching Alignment" (Liu et al., 4 Dec 2025) derives finetuning from optimal control. The central principle is

$\lambda$6

so the residual velocity of the finetuned model should match a value-gradient field. VGG-Flow implements this through a gradient-matching loss, a value-consistency loss, and a terminal boundary loss, with heuristic initialization from reward gradients of a one-step prediction. On Stable Diffusion 3, the paper reports improved reward with better diversity and prior preservation than ReFL and DRaFT, for example PickScore $\lambda$7 versus base $\lambda$8 (Liu et al., 4 Dec 2025).

"Flow-Map GRPO: Reinforcement Learning for Few-Step Flow-Map Generators via Anchored Stochastic Composition" (Li et al., 1 Jul 2026) adapts RL post-training to deterministic few-step flow-map generators. Its key device, Anchored Stochastic Flow Map Composition (ASFMC), converts a deterministic map into a stochastic transition by transporting to an anchor time and sampling back from the corresponding conditional distribution while preserving the original marginal probability path. This makes GRPO-style likelihood-ratio optimization possible for models such as MeanFlow and sCM without retraining them as native stochastic samplers (Li et al., 1 Jul 2026).

A more transport-specific inference-time adaptation appears in "PG-MAP: Joint MAP Optimization for Inference-Time Alignment of Diffusion and Flow-Matching Models" (Sun et al., 22 Jun 2026). For diffusion, PG-MAP jointly optimizes conditioning $\lambda$9 and latent state $h$0 under a forward-consistency coupling and optional reward tilt. For flow matching, however, the paper states that the effective specialization collapses to a latent-only variant, UG-FM, operating on the data side of the trajectory. This result is a useful negative boundary for FMRG: reward-guided control in flow matching does not always benefit from joint conditioning optimization (Sun et al., 22 Jun 2026).

6. Reward-conditioned action distributions in multimodal driving planning

A distinct but related use of FMRG appears in multimodal driving planning. "FlowR2A: Learning Reward-to-Action Distribution for Multimodal Driving Planning" (Li et al., 23 Jun 2026) addresses the tension between scoring-based methods, which receive dense reward supervision but rank only a fixed action vocabulary, and anchor-based methods, which generate proposals dynamically but are trained from a single ground-truth trajectory per scene. Its core move is to reframe simulation-based rewards from discriminative targets into generative conditions and to learn

$h$1

where $h$2 is a trajectory and $h$3 is a reward vector describing simulated outcome.

The paper constructs a dense action vocabulary $h$4 of 8192 four-second trajectories clustered from nuPlan trajectories. Each trajectory is rolled out in the NAVSIM simulator, producing dense $h$5 pairs. A flow-matching decoder then learns the reward-conditioned action distribution using the interpolation

$h$6

and the loss

$h$7

The reward encoder embeds each reward signal separately and injects the resulting condition vector into the decoder via AdaLN (Li et al., 23 Jun 2026).

The reward set spans safety, progress, comfort, and rule compliance. It includes no at-fault collisions (NC), time-to-collision (TTC), ego progress (EP), history comfort (HC), drivable area compliance (DAC), driving direction compliance (DDC), traffic light compliance (TLC), and lane keeping (LK). The paper emphasizes that a single scalar score such as PDMS or EPDMS is too coarse, so it introduces per-timestep structure through a TTC-time array and an ego-area array, thereby exposing where a safety or rule-compliance violation occurs along the trajectory (Li et al., 23 Jun 2026).

To balance hard and soft objectives, FlowR2A augments continuous rewards with Gaussian noise,

$h$8

using $h$9, while sharpening safety and drivable-area signals through fine-grained temporal arrays. At inference, the learned $$h_t(x)=\mathbb{E}\!\left[e^{\lambda r(X_1)}\mid X_t=x\right].$$0 becomes a controllable sampler through classifier-free reward guidance,

$$h_t(x)=\mathbb{E}\!\left[e^{\lambda r(X_1)}\mid X_t=x\right].$$1

with $$h_t(x)=\mathbb{E}\!\left[e^{\lambda r(X_1)}\mid X_t=x\right].$$2 by default, and through anchored sampling, which initializes denoising from a noisy version of any anchor trajectory rather than from pure noise (Li et al., 23 Jun 2026).

Empirically, the paper reports substantially improved proposal quality and state-of-the-art benchmark performance. With 1 proposal, FlowR2A achieves about 89.4 PDMS; with 4 proposals, 91.8 PDMS; and over all proposals, 92.9 PDMS, with mean PDMS over all proposals 89.4 ± 7.4, compared with 77.9 ± 22.4 for iPad and 60.3 ± 33.1 for DiffusionDrive. On NAVSIM v1 navtest it reports PDMS = 92.8, with NC 98.8, DAC 98.0, TTC 96.0, and EP 90.1; on NAVSIM v2 navtest it reports EPDMS = 88.9, with NC 98.9, DAC 98.1, TTC 98.5, and EP 91.5 (Li et al., 23 Jun 2026).

This usage broadens the meaning of FMRG. Instead of steering a pretrained sampler toward a reward-tilted data distribution, FlowR2A learns a reward-conditioned action distribution whose multimodality is intrinsic to $$h_t(x)=\mathbb{E}\!\left[e^{\lambda r(X_1)}\mid X_t=x\right].$$3. A plausible implication is that FMRG is becoming a domain-general design pattern: reward can be injected into a flow either as a control signal during sampling or as a conditioning variable in the learned transport itself.