One-Step Flow Q-Learning (OFQL) Overview

• One-Step Flow Q-Learning (OFQL) is a reinforcement learning paradigm that replaces multi-step denoising with a one-step flow-based mapping from noise to action in offline settings.
• The framework leverages flow matching techniques to generate actions directly during training and inference, thereby reducing computational and optimization bottlenecks.
• Variants like BFQ and MeanFlowQL demonstrate OFQL’s practical efficiency and performance gains on benchmarks such as D4RL by integrating Q-learning with streamlined flow policies.

One-Step Flow Q-Learning (OFQL) is an offline reinforcement learning paradigm in which a policy generates actions by a single flow-based mapping from noise to action, rather than by multi-step denoising or numerical integration. The framework was introduced explicitly in “Revisiting Diffusion Q-Learning: From Iterative Denoising to One-Step Action Generation,” which proposes OFQL as a reformulation of Diffusion Q-Learning (DQL) within Flow Matching, with one-step action generation during both training and inference and without auxiliary models, distillation, or multi-phase training (Nguyen et al., 19 Aug 2025). In broader usage, OFQL denotes a class of methods that replace multi-step generative policies with a one-step flow-based mapping $a=f_\theta(s,z)$, motivated by the computational and optimization bottlenecks of diffusion-style policy learning in offline RL (Nguyen et al., 9 Jun 2026).

1. Historical emergence and terminological scope

The immediate antecedent of OFQL is Flow Q-Learning (FQL), which trained an expressive flow-matching policy for behavioral cloning and a separate one-step policy for RL, thereby avoiding unstable recursive backpropagation through the iterative generator and eliminating costly iterative action generation at test time (Park et al., 4 Feb 2025). The 2025 OFQL paper then revisited DQL directly, identified its reliance on multi-step denoising during training and inference as the central practical limitation, and proposed a one-step flow-based alternative that learns an average velocity field for direct action generation (Nguyen et al., 19 Aug 2025).

Subsequent literature does not use the term in a fully standardized way. Some works treat OFQL as a general paradigm; for example, Bootstrapped Flow Q-Learning states that “OFQL is a paradigm for offline RL that replaces multi-step generative policies with a single-step flow-based mapping $a=f_\theta(s,z)$” (Nguyen et al., 9 Jun 2026). Other papers use the label only descriptively: “One-Step Generative Policies with Q-Learning” identifies its method as MeanFlowQL and notes that “OFQL” is a descriptive alias rather than the paper’s formal name (Wang et al., 17 Nov 2025); Flow-Anchored Noise-conditioned Q-Learning similarly states that OFQL is not a formal name used in that paper and maps the idea to FAN (Lee et al., 3 May 2026). In latent-policy and goal-conditioned settings, the term is also used as a conceptual umbrella for one-step flow priors guided by Q-learning rather than as a single canonical algorithm (Im et al., 5 Mar 2026, Dong et al., 10 Apr 2026).

This suggests that OFQL is best understood as a methodological family centered on one-step flow-based policy extraction under actor–critic or value-guided training, rather than as a single invariant implementation.

2. Core formulation

In the formulation given by BFQ, offline RL uses a fixed dataset $D=\{(s,a,r,s')\}$ to learn a policy $\pi_\theta(a\mid s)$ that maximizes expected return without further environment interaction, and a one-step flow policy is written as

$a=f_\theta(s,z), \qquad z\sim p_0,$

with $p_0$ typically a simple base distribution such as $\mathcal{N}(0,I)$ (Nguyen et al., 9 Jun 2026). A conditional Flow Matching path connects clean action $a$ and noise $z$ by linear interpolation,

$x_t=(1-t)a+t z,\qquad t\in[0,1],$

and the conditional velocity used for supervision is

$a=f_\theta(s,z)$0

The associated conditional Flow Matching loss is

$a=f_\theta(s,z)$1

In BFQ, this supervision is used “on the fly,” without separate teachers or multi-stage training (Nguyen et al., 9 Jun 2026).

The original OFQL paper places this same general objective in direct relation to DQL. It reformulates DQL within the sample-efficient Flow Matching framework and argues that conventional Flow Matching induces curved generative trajectories that impede one-step generation; OFQL instead learns an average velocity field that facilitates direct, accurate action generation (Nguyen et al., 19 Aug 2025). That distinction is central: one-step generation is not merely a matter of taking a single Euler step, because naive one-step denoising in DQL leads to a drastic performance drop (Nguyen et al., 19 Aug 2025).

A closely related formalization appears in Flow Map Q-Guidance, where a flow map policy learns a two-time jump operator

$a=f_\theta(s,z)$2

with $a=f_\theta(s,z)$3 interpreted as the average velocity between inner times $a=f_\theta(s,z)$4 and $a=f_\theta(s,z)$5. Evaluating $a=f_\theta(s,z)$6 yields one-step action generation from Gaussian noise in a single jump across the generative dynamics (Ziakas et al., 12 May 2026).

3. Principal algorithmic realizations

Different OFQL realizations differ mainly in how they obtain a one-step map and how Q-information enters training.

Realization	One-step mechanism	Characteristic training signal
FQL	Distilled student policy $a=f_\theta(s,z)$7	Distillation plus $a=f_\theta(s,z)$8-maximization
BFQ	Displacement operator $a=f_\theta(s,z)$9	Boundary anchoring plus compositional consistency
MeanFlowQL	Residual map $D=\{(s,a,r,s')\}$0	MeanFlow Identity regression plus $D=\{(s,a,r,s')\}$1-learning
FMQ	Flow map $D=\{(s,a,r,s')\}$2	Closed-form $D=\{(s,a,r,s')\}$3-guided trust-region target

In FQL, the iterative flow policy is trained only with behavioral cloning, while a separate one-step policy $D=\{(s,a,r,s')\}$4 is trained by direct distillation,

$D=\{(s,a,r,s')\}$5

and by a reparameterized actor loss

$D=\{(s,a,r,s')\}$6

thereby bypassing backpropagation through the flow ODE entirely (Park et al., 4 Feb 2025).

BFQ removes the teacher–student split and learns a displacement operator directly. For a point $D=\{(s,a,r,s')\}$7 on the flow path, the endpoint policy operator is

$D=\{(s,a,r,s')\}$8

where $D=\{(s,a,r,s')\}$9 is the integrated displacement from time $\pi_\theta(a\mid s)$0 to $\pi_\theta(a\mid s)$1. BFQ enforces a boundary loss that aligns the learned local velocity to the conditional Flow Matching target and a compositional consistency loss,

$\pi_\theta(a\mid s)$2

The resulting one-step action generator is

$\pi_\theta(a\mid s)$3

with no multi-step denoising or numerical ODE integration (Nguyen et al., 9 Jun 2026).

MeanFlowQL reformulates MeanFlow as a residual one-step policy,

$\pi_\theta(a\mid s)$4

and performs inference by

$\pi_\theta(a\mid s)$5

Its training signal is the MeanFlow Identity regression

$\pi_\theta(a\mid s)$6

combined with a $\pi_\theta(a\mid s)$7-maximization term, yielding a single-stage one-step generative policy without distillation (Wang et al., 17 Nov 2025).

The design pattern also extends beyond flat action-space policies. Latent Policy Steering uses a differentiable one-step MeanFlow base policy as a behavior prior and backpropagates original-action-space $\pi_\theta(a\mid s)$8-gradients through that one-step generator to a latent actor; Efficient Hierarchical Implicit Flow Q-learning uses high-level and low-level goal-conditioned mean flow policies to generate subgoals and primitive actions by one-step sampling; Safe Flow Q-Learning distills a flow policy into a one-step actor and then augments training with safety critics and conformal calibration (Im et al., 5 Mar 2026, Dong et al., 10 Apr 2026, Tayal et al., 16 Mar 2026).

4. Q-learning coupling and regularization geometry

A central OFQL question is how to combine one-step generation with offline RL’s need for behavioral regularization. In FQL, the one-step actor’s regularizer is distillation from a behavior-cloned flow teacher, and the paper shows that the distillation loss upper-bounds the squared $\pi_\theta(a\mid s)$9-Wasserstein distance between the induced one-step policy and the flow policy, making the “BC” term a metric-aware behavioral constraint in action space (Park et al., 4 Feb 2025).

BFQ retains the actor–critic template but replaces distillation with a behavior cloning objective that mixes local and global structural constraints: $a=f_\theta(s,z), \qquad z\sim p_0,$0 and the actor loss becomes

$a=f_\theta(s,z), \qquad z\sim p_0,$1

Its critic is standard double $a=f_\theta(s,z), \qquad z\sim p_0,$2-learning with target networks updated by EMA (Nguyen et al., 9 Jun 2026).

FMQ makes explicit that OFQL need not imply a single regularization geometry. It contrasts a KL-constrained view of one-step policy improvement,

$a=f_\theta(s,z), \qquad z\sim p_0,$3

with its own $a=f_\theta(s,z), \qquad z\sim p_0,$4 trust region in action or average-velocity space. Under a first-order Taylor expansion of the critic and an $a=f_\theta(s,z), \qquad z\sim p_0,$5 trust region, FMQ derives the closed-form target

$a=f_\theta(s,z), \qquad z\sim p_0,$6

which is then regressed by the one-step flow-map actor (Ziakas et al., 12 May 2026).

In maximum-entropy RL, FLAME gives a soft-policy analogue of OFQL. It trains a one-step flow actor with Q-Reweighted Flow Matching or Q-Reweighted MeanFlow, using weights proportional to $a=f_\theta(s,z), \qquad z\sim p_0,$7, and employs a decoupled entropy estimator to correct the discretization bias that arises when one-step flows are used inside MaxEnt critic targets (Li et al., 2 Feb 2026). FAN, by contrast, combines a one-step actor with Flow Anchoring,

$a=f_\theta(s,z), \qquad z\sim p_0,$8

and a noise-conditioned distributional critic whose upper expectile approximates the essential supremum of the return distribution using only a single Gaussian noise sample per update (Lee et al., 3 May 2026).

5. Empirical profile

The original OFQL paper reports that OFQL outperforms DQL and other diffusion-based baselines on the D4RL benchmark while substantially reducing both training and inference time compared to DQL; its abstract further attributes the gain to the removal of multi-step sampling and recursive gradient updates (Nguyen et al., 19 Aug 2025).

BFQ provides the most explicit benchmark-level evidence for an OFQL instance. On D4RL locomotion, it reports an average normalized score of $a=f_\theta(s,z), \qquad z\sim p_0,$9, improving over DQL at $p_0$0, SRPO at $p_0$1, SORL at $p_0$2, and FQL at $p_0$3. On D4RL AntMaze, it reports an average of $p_0$4, competitive with OFQL at $p_0$5 and above DQL at $p_0$6, SRPO at $p_0$7, SORL at $p_0$8, and FQL at $p_0$9. Its compute profile is equally explicit: on MuJoCo with $\mathcal{N}(0,I)$0M steps on an A100 GPU, BFQ trains in $\mathcal{N}(0,I)$1 h with decision frequency $\mathcal{N}(0,I)$2 Hz, whereas DQL ranges from $\mathcal{N}(0,I)$3 h at $\mathcal{N}(0,I)$4 denoising steps and $\mathcal{N}(0,I)$5 Hz to $\mathcal{N}(0,I)$6 h at $\mathcal{N}(0,I)$7 steps and $\mathcal{N}(0,I)$8 Hz (Nguyen et al., 9 Jun 2026).

MeanFlowQL shows that single-stage one-step policies can remain competitive without distillation. Across $\mathcal{N}(0,I)$9 tasks on OGBench and D4RL, it reports strong offline results such as $a$0 average on D4RL antmaze and substantial offline-to-online gains, including humanoidmaze-medium-navigate from $a$1 offline to $a$2 after online fine-tuning and cube-double-play from $a$3 to $a$4 (Wang et al., 17 Nov 2025).

Other one-step flow variants exhibit similar efficiency emphases. FAN reports approximately $a$5–$a$6 reductions in training runtime compared to distributional baselines and the best inference FLOPs and wall-clock time among its baselines, while achieving state-of-the-art or near-best performance in $a$7 of $a$8 environment families (Lee et al., 3 May 2026). In offline-to-online settings, FMQ reports state-of-the-art performance across $a$9 tasks with IQM success $z$0, outperforming MVP at $z$1 by a $z$2 relative margin; adding Q-Guided Beam Search raises this to $z$3 (Ziakas et al., 12 May 2026).

6. Theory, limitations, and open directions

The strongest formal convergence result specific to a one-step flow policy appears in the linear-quadratic analysis of Flow-Policy Gradient Learning. There, the one-step policy loss—an average expected cost plus behavioral cloning regularization—is shown to be $z$4-smooth and gradient dominant, and under a stabilizing initializer $z$5, step size $z$6, and $z$7, gradient descent yields stabilizing iterates and linear convergence to an optimal gain $z$8 in the offline LQR setting (Yaghmaie et al., 14 Nov 2025). This analysis does not cover general nonlinear OFQL, but it establishes that BC-regularized one-step flow policies can admit rigorous convergence guarantees in a structured regime.

Several limitations recur across the broader OFQL family. BFQ states that single-step sufficiency depends on reliable marginal velocities and short-range displacements; if marginal velocity estimation is biased or the probability path is highly curved, one-step mapping may degrade, and extreme distribution shift or out-of-distribution states can still cause failure (Nguyen et al., 9 Jun 2026). MeanFlowQL identifies sensitivity to the behavior-cloning coefficient $z$9, Jacobian-vector-product overhead, and instability from continuous $x_t=(1-t)a+t z,\qquad t\in[0,1],$0 sampling in low-dimensional action spaces (Wang et al., 17 Nov 2025). OTQL, which targets few-step rather than strict one-step generation, reports that $x_t=(1-t)a+t z,\qquad t\in[0,1],$1–$x_t=(1-t)a+t z,\qquad t\in[0,1],$2 NFEs are reliable without distillation and notes that robust $x_t=(1-t)a+t z,\qquad t\in[0,1],$3-step behavior may require stricter straightness constraints than its advantage-weighted conditional OT flow matching alone (Sochopoulos et al., 7 Jul 2026).

A distinct line of critique concerns the stability–expressivity trade-off. Q-Flow argues that OFQL-style one-step distillation stabilizes optimization but can collapse modes and reduce the expressive capacity of the original continuous-time normalizing flow; it addresses this by keeping the full CNF policy and replacing solver unrolling with intermediate value-gradient matching, reporting an average improvement of $x_t=(1-t)a+t z,\qquad t\in[0,1],$4 percentage points over FQL on OGBench (Doo et al., 13 May 2026). This does not refute OFQL, but it clarifies an important open question: whether the most effective one-step flow policy is obtained by direct one-step parameterization, by distillation from a richer multi-step teacher, or by reformulating the flow dynamics so that one-step generation remains faithful to the original expressive model.

The literature therefore converges on a common objective—eliminating multi-step denoising and backpropagation through time—while remaining unsettled on the best geometric object to learn: a distilled terminal map, an average velocity field, a compositional displacement operator, a trust-region flow map, or a latent or hierarchical one-step prior.