Diffusion Q-Learning in Offline RL

• Diffusion Q-Learning is an offline RL method that uses conditional diffusion models to capture multimodal behavior and improve policy fidelity.
• It combines a denoising objective with Q-maximization to focus on high-return actions while constraining improvements to behavior-supported distributions.
• The approach shows strong performance on benchmarks like D4RL but involves higher computational costs during iterative denoising at inference.

to=arxiv_search 天天买彩票 񎔥json-schema {"query":"Diffusion Q-Learning offline reinforcement learning diffusion policy DQL arXiv", "max_results": 10} to=search_arxiv 彩票天天乐json {"query":"Diffusion Q-Learning offline reinforcement learning", "max_results": 10} Diffusion Q-Learning usually refers to Diffusion-QL, an offline reinforcement learning method that represents the policy as a conditional diffusion model over actions given states and trains that policy with a joint objective combining diffusion denoising / behavior cloning and Q-maximization (Wang et al., 2022). In the canonical formulation, the aim is not unconstrained maximization of a learned critic, but the search for high-return actions that remain near the behavior policy, thereby addressing the offline RL failure mode in which function approximation error on out-of-distribution actions is exploited during policy improvement (Wang et al., 2022).

1. Concept and offline RL setting

Diffusion-QL is defined in the standard offline RL setting, where training uses only a fixed dataset

$\mathcal D \triangleq \{(s,a,r,s')\},$

collected by a behavior policy $\pi_b$, with no further environment interaction during learning (Wang et al., 2022). The underlying environment is an MDP

$M=\{S, A, P, R, \gamma, d_0\},$

and the central difficulty is distribution shift: the critic is learned only on dataset actions, but policy improvement tends to search over actions that may be absent from the dataset, causing extrapolation error and unstable improvement (Wang et al., 2022).

The original Diffusion-QL paper positions this problem against earlier policy-regularized offline RL methods that use limited policy classes such as diagonal Gaussian or tanh-Gaussian actors. Its central claim is that offline RL regularization is only as good as the behavior-support model to which the policy is regularized, and that multimodal or otherwise complex action distributions are poorly represented by simple parametric actors (Wang et al., 2022). Diffusion models are introduced precisely to address this bottleneck: they provide a highly expressive implicit conditional distribution over actions, which can better capture multimodality and thereby make behavior-constrained policy improvement more faithful to dataset support (Wang et al., 2022).

2. Canonical Diffusion-QL formulation

In Diffusion-QL, the policy $\pi_\theta(a\mid s)$ is implemented as the endpoint of a reverse diffusion chain conditioned on the state: $$\pi_{\theta}(a \mid s) = p_{\theta}(a^{0:N} \mid s) = \mathcal N(a^N; \mathbf{0}, I) \prod_{i=1}^N p_{\theta}(a^{i-1} \mid a^{i}, s).$$ Thus action generation is a generative process: sample $a^N$ from Gaussian noise, then iteratively denoise to obtain $a^0$, which is used as the policy action (Wang et al., 2022).

The behavior-cloning component is the simplified DDPM denoising objective

$$L_d(\theta) = E_{i \sim U(1,N),\, \epsilon \sim \mathcal N(\mathbf{0}, I),\, (s,a) \sim \mathcal D} \left[ \left\| \epsilon - \epsilon_\theta\!\left( \sqrt{\bar{\alpha}_i} a + \sqrt{1-\bar{\alpha}_i}\epsilon, s, i \right) \right\|^2 \right].$$

This term makes the conditional diffusion policy match the dataset action distribution given the state (Wang et al., 2022).

Policy improvement is introduced directly into the diffusion actor loss through

$$L_q(\theta) = -\alpha \cdot E_{s\sim \mathcal D,\; a^0\sim \pi_\theta(\cdot\mid s)} \left[Q_\phi(s,a^0)\right].$$

The defining objective of Diffusion-QL is therefore

$$L(\theta)=L_d(\theta)+L_q(\theta) = L_d(\theta) - \alpha \cdot E_{s \sim \mathcal D,\; a^0 \sim \pi_\theta} \left[ Q_{\phi}(s, a^0) \right].$$

The coefficient is normalized as

$\pi_b$0

following a TD3+BC-style scale normalization in which the denominator is not differentiated through (Wang et al., 2022).

Critic learning uses two Q-networks, target networks, and standard Bellman regression with double Q: $\pi_b$1 The novelty is therefore not a new critic target, but the coupling of a diffusion actor with direct Q-guided policy improvement under a denoising regularizer (Wang et al., 2022).

3. Expressiveness, behavior support, and multimodality

The main theoretical and empirical rationale for Diffusion-QL is that policy expressiveness matters in offline RL. If the conditional action distribution in the dataset is multimodal, then a unimodal Gaussian behavior model may place probability mass in regions between modes where the dataset has low density. In that case, “regularization toward the behavior policy” can regularize toward the wrong distribution (Wang et al., 2022).

The original paper demonstrates this effect in a simple 2D bandit with a 4-mode behavior distribution. In that example, BC-Diffusion recovers the four modes well, whereas Gaussian MLE and CVAE exhibit mode-covering behavior; when Q-learning is added, Diffusion-QL finds the high-reward mode, while prior regularization methods remain constrained by poorer support models (Wang et al., 2022). The paper therefore argues that two ingredients are jointly responsible for the method’s performance: the expressiveness of the diffusion model-based policy and the coupling of behavior cloning with policy improvement under the same diffusion policy (Wang et al., 2022).

This construction yields a specific interpretation of policy improvement in Diffusion-QL. The diffusion loss alone would imitate the behavior policy, while the Q term alone would aggressively chase high predicted values and risk unsupported actions. Their combination seeks optimal actions that are near the behavior policy, which is a form of policy improvement inside a behavior-supported action manifold rather than unconstrained actor optimization (Wang et al., 2022).

4. Benchmark profile and practical limitations

On D4RL, the original Diffusion-QL paper reports strong average performance across all four major benchmark groups and presents the method as state of the art on the majority of tasks (Wang et al., 2022).

D4RL domain	Diffusion-QL average	Comparison reported in the same study
Gym-MuJoCo	88.0	higher than CQL 77.6, IQL 77.0, TD3+BC 75.3
AntMaze	69.6	higher than IQL 63.0, CQL 50.6, TD3+BC 27.3
Adroit	65.1	higher than BCQ 56.5, IQL 54.4, CQL 31.2
Kitchen	69.0	higher than IQL 53.3, CQL 48.2, BC 38.4

The reported strengths are especially pronounced on medium and medium-replay MuJoCo tasks, and the method is also competitive on sparse-reward AntMaze, Adroit, and Kitchen benchmarks (Wang et al., 2022). At the same time, the original study does not present Diffusion-QL as uniformly dominant on every row: for example, on antmaze-umaze-diverse-v0, Diffusion-QL reports 66.2, below CQL 84.0, and on walker2d-medium-expert-v2, Diffusion-QL reports 110.1, tied with TD3+BC and below Onestep RL 113.0 (Wang et al., 2022).

The principal practical limitation is computational. The denoising loss is efficient because only one diffusion step is sampled during training, but actual action generation requires evaluating the denoiser $\pi_b$2 times, and the policy-improvement term backpropagates through the reverse diffusion chain (Wang et al., 2022). The original paper studies $\pi_b$3, finds that larger $\pi_b$4 improves stability and expressiveness, and uses $\pi_b$5 in most D4RL experiments; with $\pi_b$6, training time is reported as similar to CQL, but inference remains roughly 4–5× slower than one-step feedforward policies (Wang et al., 2022). Subsequent work treats this iterative denoising requirement as the central bottleneck of DQL-style methods (Chen et al., 2024).

5. One-step, trust-region, alignment, and flow-based successors

A substantial later literature keeps the offline RL intuition of Diffusion-QL but attempts to remove or bypass iterative denoising. These methods differ in whether they treat DQL as something to distill, constrain, align, or reformulate.

Method	Core idea	Reported property
DTQL	dual policy: diffusion policy for pure behavior cloning plus a one-step actor connected by a diffusion trust region loss	eliminates iterative denoising during both training and inference (Chen et al., 2024)
EDA	two-stage optimization: reward-free diffusion behavior pretraining, then Q-guided alignment via density ratios	overall D4RL average 83.7 vs Diffusion-QL 82.1 in its table (Chen et al., 2024)
OFQL	reformulates DQL within sample-efficient Flow Matching and learns an average velocity field	abstract states one-step action generation without auxiliary models, distillation, or multi-phase training (Nguyen et al., 19 Aug 2025)
BFQ	bootstrapped single-step flow-map learning from short-range displacement supervision	MuJoCo average 92.8 vs DQL 89.0; AntMaze average 83.9 vs DQL 81.3 (Nguyen et al., 9 Jun 2026)

Diffusion Trusted Q-Learning (DTQL) keeps a diffusion policy only as an expressive behavior model and delegates control to a one-step Gaussian actor $\pi_b$7, trained with a diffusion trust region loss rather than KL-based distribution matching. Its critic follows IQL rather than the original DQL critic update, and on D4RL it reports Gym average 88.7 versus 88.0 for DQL, AntMaze average 73.6 versus 69.6, Adroit average 72.7 versus 65.1, and Kitchen average 71.8 versus 69.0; on antmaze-umaze-v0, it reports training time per 1k steps of 21.83 s for DTQL versus 24.13 s for DQL, total training time of 3.33 h versus 6.70 h, and inference time per trajectory of 0.35 s versus about 3.03–3.04 s for DQL and IDQL (Chen et al., 2024).

Efficient Diffusion Alignment (EDA) reframes offline RL as a two-stage procedure: first pretrain a diffusion behavior model on reward-free behavior data, then fine-tune it by aligning policy-vs-behavior density ratios to Q-induced action preferences. A technical novelty is the representation of the diffusion policy as the derivative of a scalar neural network with respect to action inputs, which the paper uses to enable direct density calculation up to normalization. In its D4RL table, EDA reports overall average 83.7 versus 82.1 for Diffusion-QL, while remaining weaker than Diffusion-QL on several expert-ish rows such as halfcheetah-medium-expert and walker2d-medium-replay (Chen et al., 2024).

One-Step Flow Q-Learning (OFQL) explicitly diagnoses the reliance of DQL on multi-step denoising during both training and inference, argues that naive one-step denoising causes a drastic performance drop, and proposes a flow-matching reformulation that learns an average velocity field for direct one-step action generation. Its abstract states that OFQL enables one-step action generation during both training and inference without requiring auxiliary models, distillation, or multi-phase training, and that D4RL experiments show it outperforms DQL and other diffusion-based baselines while substantially reducing training and inference time (Nguyen et al., 19 Aug 2025).

Bootstrapped Flow Q-Learning (BFQ) takes the strongest anti-denoising position. It replaces multi-step denoising with a single-step flow-map learner trained by short-range displacement supervision and compositional bootstrapping. On D4RL, it reports MuJoCo average 92.8 versus 89.0 for DQL and AntMaze average 83.9 versus 81.3; it also reports average wall-clock training time over MuJoCo tasks of 7.8 hours for BFQ versus 11.7, 16.1, 24.8, and 49.5 hours for DQL with 5, 10, 20, and 50 denoising steps, respectively, and inference throughput of 851.2 Hz versus 238.1, 150.1, 75.2, and 35.5 Hz (Nguyen et al., 9 Jun 2026). This suggests that, in later flow-based reformulations, the original generative-policy advantage of DQL is increasingly treated as separable from its multi-step sampler.

6. Terminological ambiguity and neighboring research

The acronym DQL is historically ambiguous. In older and mainstream RL literature, DQL commonly denotes Deep Q-Learning, not diffusion-model-based offline RL. That older usage is exemplified by work on instability under the deadly triad—bootstrapping, off-policy learning, and function approximation—which is conceptually relevant to critic stability in offline RL but is not about diffusion policies (Achiam et al., 2019).

Several neighboring research lines also use overlapping terminology while remaining distinct from canonical Diffusion-QL. Q-Score Matching trains a diffusion-model policy from rewards by aligning the actor’s score field with the critic action-gradient $\pi_b$8, proposing the actor loss

$\pi_b$9

and arguing that previous diffusion-policy RL methods often fail to exploit the score structure directly (Psenka et al., 2023). This is highly relevant to DQL, but it is presented as a more structured off-policy diffusion actor-critic variant rather than as the canonical offline RL method.

By contrast, “Reward-Directed Score-Based Diffusion Models via q-Learning” formulates training of score-based diffusion models as a continuous-time entropy-regularized control problem, where the learned object called $M=\{S, A, P, R, \gamma, d_0\},$0 is not the discrete-time Bellman $M=\{S, A, P, R, \gamma, d_0\},$1 of offline RL; it is therefore adjacent to diffusion-and-RL research but not the same algorithmic family as Diffusion-QL (Gao et al., 2024). Likewise, “Reinforcement Learning for Discounted and Ergodic Control of Diffusion Processes” studies quantized Q-learning for controlled diffusion processes in the stochastic-control sense, where “diffusion” means an SDE on $M=\{S, A, P, R, \gamma, d_0\},$2, not a denoising diffusion model (Bayraktar et al., 13 Mar 2026).

This terminological spread suggests that Diffusion Q-Learning now names a research area more than a single immutable method. In the narrow sense, it denotes the original Diffusion-QL formulation of offline RL with a conditional diffusion policy and a joint denoising-plus-Q objective (Wang et al., 2022). In the broader contemporary sense, it also motivates a lineage of trust-region, alignment-based, score-based, and flow-based methods that preserve the core idea of expressive generative policies guided by value functions, while differing sharply on whether multi-step denoising is essential, dispensable, or something to be replaced altogether (Chen et al., 2024).