Q-Guided Flow: Integrating Flow and Q-Value Guidance

• Q-Guided Flow is a method coupling flow-based generative policies with learned Q-values to guide action trajectories.
• It leverages critic gradients at test time or as local velocity targets during training to stabilize policy improvements without full backpropagation through denoising chains.
• Applications span offline RL, continuous control, and vision-language-action tasks, demonstrating notable performance gains and computational efficiency.

Q-Guided Flow (QGF) denotes a class of methods in which a learned action-value function or critic steers the dynamics of a flow-based generative policy. In recent work, the term is used most prominently in offline reinforcement learning, continuous-control policy optimization, and flow-matching vision-language-action (VLA) systems. The common objective is to preserve the expressivity of multi-step flow or flow-matching policies while using $Q$-information to bias sampling, denoising, or policy updates toward higher-value actions. Depending on the formulation, this guidance is applied entirely at test time, converted into local velocity-field targets during training, or embedded into trust-region or reverse-time sampling rules (Zhou et al., 9 Jun 2026).

1. Definition and scope

Within the recent literature, Q-guided flow is not a single standardized algorithm but a family of closely related mechanisms for coupling flow-based generation with action values. The unifying pattern is that a policy is represented by a flow, ODE, or flow map, and a critic contributes either gradients, value targets, or selection rules that alter how the flow evolves. This family includes test-time gradient guidance of a frozen behavioral-cloning flow policy (Zhou et al., 9 Jun 2026), intermediate-value-guided flow optimization without backpropagation through time (Doo et al., 13 May 2026), critic-aligned flow map adaptation under a trust-region constraint (Ziakas et al., 12 May 2026), and Q-guided inference or fine-tuning for flow-matching VLA policies (Yang et al., 2 Jul 2026).

The motivation is consistent across these works. Flow-based policies and flow-matching policies are attractive because they can represent highly expressive and multimodal action distributions. The difficulty is that optimizing such policies directly with conventional actor-critic machinery is often unstable or computationally expensive, especially when it requires differentiating through a multi-step ODE or denoising chain. Q-guided flow methods therefore aim to separate expressive action modeling from value-driven improvement. In some formulations the policy is trained only with supervised flow matching and improved non-parametrically at inference; in others, the value information is converted into local regression targets so that policy learning remains solver-free or avoids full BPTT (Doo et al., 13 May 2026).

The term should also be distinguished from adjacent notions. "Guided Flows" in classifier-free guidance mix unconditional and conditional velocity fields for conditional generation, rather than using a learned action-value function (Zheng et al., 2023). QGFN, despite the similar acronym, operates in GFlowNets on DAG-structured compositional generation and combines a GFlowNet forward policy with an action-value estimate through a controllable greediness parameter $p$, rather than guiding a continuous denoising flow in action space (Lau et al., 2024).

2. Canonical test-time formulation

A canonical QGF formulation pre-trains a reference flow policy $\hat{\pi}$ by standard flow matching or behavioral cloning and trains a value critic separately, then performs all policy improvement at inference time by modifying generated actions rather than policy parameters. In this setup, the actor is trained only via the supervised flow-matching objective

$$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$

with $a_t=(1-t)a_0+t a_1$, while the critic is trained separately with offline RL, for example with IQL-style $Q$- and $V$-losses (Zhou et al., 9 Jun 2026).

At test time, the policy starts from noise $a_0 \sim \mathcal{N}(0,I)$ and iteratively denoises. At each step it first forms a first-order estimate of the clean action,

$\hat a_1 = a_t + (1-t)\, v_\theta(s,a_t,t),$

then evaluates the value gradient with respect to that denoised action,

$g=\nabla_{\hat a_1} Q(s,\hat a_1),$

and updates the action by

$p$0

The stated purpose of this construction is to avoid two alternatives that proved problematic: querying the critic at noisy intermediate actions, which is out-of-distribution, and backpropagating through the full denoising chain, which is expensive and high-variance (Zhou et al., 9 Jun 2026).

The theoretical rationale is framed through a KL-regularized RL objective in which an improved policy remains close to the reference policy. In score form, the improved policy decomposes as

$p$1

This makes the guidance term an additive value-gradient correction to the reference flow dynamics. A common misconception is that more exact gradient estimators must be preferable. The reported finding is the opposite: using the Jacobian or full ODE integration hurts performance or increases variance, whereas the simple gradient estimator based on $p$2 is described as clean, cheap, low variance, and empirically best (Zhou et al., 9 Jun 2026).

3. Value-gradient matching and trust-region variants

A second line of work moves the $p$3-guidance from pure test-time correction to local training targets. In "Q-Flow," the policy is a continuous normalizing flow with deterministic inner dynamics

$p$4

The central construction is a flow-consistent intermediate value function,

$p$5

which assigns to each intermediate latent state the value of the terminal action it deterministically flows to. Policy optimization is then performed by matching the model velocity to a Q-guided target velocity

$p$6

through the regression loss

$p$7

The stated advantages are that no BPTT is required and the full multi-step flow expressiveness is retained rather than collapsed into a one-step map (Doo et al., 13 May 2026).

A related but distinct formulation appears in flow map policies. There, the policy learns arbitrary-size jumps along the generative dynamics through a flow map

$p$8

Online adaptation is posed as a trust-region optimization problem: maximize the critic while remaining close to the offline reference. The reported closed-form solution perturbs the average velocity in the normalized direction of the action-gradient,

$p$9

and learning uses the corresponding FMQ regression target with stop-gradient. An adaptive trust region further scales $\hat{\pi}$0 by critic disagreement, and inference can be refined with Q-guided beam search, which combines renoising with beam search and iterative Q-guided updates (Ziakas et al., 12 May 2026).

These approaches differ in implementation but share the same structural move: instead of differentiating the critic through a long flow trajectory, they convert value information into local guidance fields or closed-form targets that can be learned or applied stepwise.

4. Q-guided flow in vision-language-action policies

In VLA policies, Q-guided flow has become a mechanism for improving frozen or few-shot-initialized flow-matching action experts without requiring end-to-end RL over the full denoising chain. "Guided Action Flow" keeps a pretrained SmolVLA policy frozen and trains an action-chunk critic from real success and failure rollouts. The critic can condition on task-description features taken from the frozen SmolVLA language pathway and is used only through action gradients during reverse-time sampling. Because SmolVLA uses a reverse-time convention, the clean action estimate is

$\hat{\pi}$1

and the guided velocity becomes

$\hat{\pi}$2

with $\hat{\pi}$3 and an uncertainty gate

$\hat{\pi}$4

The ensemble-disagreement gate is introduced to attenuate guidance when critic uncertainty is high (Yang et al., 2 Jul 2026).

"Q-VGM" addresses the harder problem of fine-tuning a flow-matching VLA policy itself. It treats the critic gradient as a denoising-time velocity correction rather than an end-to-end objective. For a noisy denoising state $\hat{\pi}$5 at time $\hat{\pi}$6, it first projects forward to a base clean-action estimate

$\hat{\pi}$7

then performs iterative Q-gradient ascent in clean action space with keep-best selection, and finally forms the effective correction

$\hat{\pi}$8

The policy is updated through residual velocity matching,

$\hat{\pi}$9

The critic is described as an action-sensitive Cal-QL ensemble over compact RLT features with per-layer action injection, and the method is explicitly designed to require no action likelihoods and no backpropagation through the denoising chain (Wang et al., 6 Jun 2026).

A plausible implication is that VLA-oriented Q-guided flow methods are converging on a common design principle: keep the base flow prior or base velocity as an anchor, query the critic on approximately clean action chunks rather than noisy states, and introduce uncertainty control or keep-best mechanisms to reduce harmful off-manifold guidance.

5. Empirical record

The empirical literature reports improvements in several settings, but the metrics and protocols differ substantially across papers. The common pattern is that Q-guidance is most effective when it exploits critic gradients without destabilizing the underlying generative policy.

Method	Setting	Reported result
Q-Flow	Offline learning on OGBench	Outperforms state-of-the-art baselines by an average of 10.6 percentage points
FMQ / QGBS	Offline-to-online RL on OGBench and RoboMimic	IQM success rate $$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$0 with CI $$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$1; 21.3% higher than MVP; QGBS raises IQM to $$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$2
Guided Action Flow	LIBERO manipulation	68.0% to 82.0%; 82.0% to 86.0%; 46.0% to 56.0%; 65.0% to 67.5%
Q-VGM	LIBERO, RoboTwin 2.0, real-robot tabletop tasks	75.0% to 92.5%; 76.4% to 87.2%; 40.0% to 67.5%

In offline RL with expressive flow policies, Q-Flow reports that policy optimization gradients remain stable for any guidance strength in 2D synthetic experiments and that the method is consistently the top-performing flow-based method across OGBench environments, including antmaze, humanoidmaze, puzzle, and cube (Doo et al., 13 May 2026). In flow map policies, FMQ is reported to converge to the highest reward or success-rate levels 2.77$$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$3 faster on average than MVP during the online phase and to be nearly $$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$4 computationally more efficient than MVP, while QGBS further improves the compute-accuracy tradeoff (Ziakas et al., 12 May 2026).

For frozen VLA policies, the strongest results currently reported are still accompanied by caveats about generalization. Guided Action Flow shows substantial single-task gains on LIBERO and a positive but modest locked held-out test gain in the multi-family setting, while emphasizing that critic generalization and uncertainty-aware guidance remain the central bottlenecks (Yang et al., 2 Jul 2026). Q-VGM, by contrast, reports gains not only on LIBERO and RoboTwin 2.0 but also on two real-robot tabletop tasks, outperforming same-backbone, same-critic baselines across all three settings (Wang et al., 6 Jun 2026).

The test-time-only QGF formulation is reported to outperform prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks with high-dimensional action spaces, to be competitive with state-of-the-art training-time algorithms, and to scale favorably with model size by avoiding actor-critic instability (Zhou et al., 9 Jun 2026).

6. Limitations, misconceptions, and open problems

A recurring misconception is that Q-guided flow necessarily means reinforcement learning through actor updates. Several of the most prominent formulations do not update the policy at RL time at all. Test-time QGF and Guided Action Flow leave the base flow policy frozen and improve actions only by modifying the sampling trajectory with critic gradients (Zhou et al., 9 Jun 2026). Conversely, methods such as Q-Flow, FMQ, and Q-VGM use the critic to define local targets that can be amortized into policy parameters, but they do so specifically to avoid unstable end-to-end backpropagation through denoising (Doo et al., 13 May 2026).

Another misconception is that any stronger or more exact use of the critic must be better. The literature repeatedly reports failure modes from naively applying gradients at the wrong points. In test-time QGF, gradients with respect to noisy actions are described as high variance and out-of-distribution, while more exact Jacobian- or ODE-based estimators hurt performance or increase variance (Zhou et al., 9 Jun 2026). In Guided Action Flow, the reverse-time sign convention, gradient clipping, and ensemble-disagreement gating are all treated as essential implementation details; without clipping and uncertainty gating, critic gradients can create regressions (Yang et al., 2 Jul 2026).

The central unresolved problem is critic quality. Multiple papers identify critic generalization, calibration, and out-of-distribution behavior as the main bottlenecks. Guided Action Flow explicitly states that critic generalization and uncertainty-aware guidance remain the central bottlenecks, with stronger gains near the critic’s training distribution than on held-out tasks (Yang et al., 2 Jul 2026). Test-time QGF likewise notes dependence on the quality and coverage of both the reference policy and the critic, sensitivity to guidance weight $$\mathcal{L}_{\mathrm{FM}}(\theta)= \mathbb{E}_{t \sim \mathcal{U}[0,1],\, a_0 \sim p_0,\, a_1 \sim p_1} \left[ \left\| v_\theta(a_t,t)-(a_1-a_0)\right\|_2^2 \right],$$5, and the requirement that the denoising process be accessible so that gradients can be injected at each step (Zhou et al., 9 Jun 2026).

A second open issue concerns how much of the value guidance should remain at inference and how much should be amortized into the policy. Test-time methods are modular and avoid actor-critic coupling, but they retain per-action inference overhead. Training-time or locally amortized methods reduce that overhead but require stable constructions for translating clean-action value information into denoising-time or flow-time correction fields. This suggests that future work will likely continue to focus on critic uncertainty, in-distribution anchors, and local value-gradient representations rather than on direct BPTT through ever larger flow policies.