Quantile-Projected Bellman Equation in RL

• The topic establishes a fixed-point formulation by projecting the Bellman update onto a quantile family for finite-dimensional distributional policy evaluation.
• It demonstrates how quantile regression aligns with Wasserstein metrics to ensure contraction properties and statistical efficiency in RL.
• Key applications span QR-DQN, IQN, FQF, and FlowIQN, with insights on convergence, flow-based transport, and approximation limitations.

Searching arXiv for recent and foundational papers on quantile-projected distributional Bellman equations and related distributional RL. The quantile-projected distributional Bellman equation is the fixed-point relation obtained by composing a distributional Bellman operator with a projection onto a quantile-parameterized family of return distributions. In distributional reinforcement learning, this construction formalizes the step “apply the Bellman update to a return law, then project the resulting target law back into a finite or implicit quantile representation,” thereby preserving compatibility with Wasserstein geometry and quantile regression training objectives. In recent work, the same template has been extended from classical quantile critics to flow-based critics, most explicitly in FlowIQN, where sorted quantile coupling is used to operationalize a Wasserstein-aligned approximate projection in one-dimensional return space (Groom et al., 8 May 2026). The formulation also admits statistical analysis as a finite-dimensional fixed point, including non-asymptotic error bounds and semiparametric efficiency results for policy evaluation under a generative model (Cheng et al., 9 Jul 2026).

1. Definition and operator-theoretic formulation

In discounted reinforcement learning, the distributional Bellman operator for a fixed policy $\pi$ maps a candidate return distribution to the next-step target law induced by immediate reward and discounted continuation. In the scalar-return setting used in distributional RL, this is written as

$$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$

This law-valued recursion underlies both classical return-distributional RL and later quantile parameterizations (Dabney et al., 2017, Groom et al., 8 May 2026).

The quantile-projected distributional Bellman equation augments this recursion with a projection $\Pi$ onto a restricted family of quantile functions or quantile-supported atomic laws. In its general form,

$Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$

When the critic is represented by a parametric quantile function $Q_\theta(\tau|s,a)$, the projection is defined as a Wasserstein-aligned projection onto the quantile family: $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$ with the metric computed through monotone coupling of quantiles in one dimension (Groom et al., 8 May 2026).

A finite-dimensional version uses an $m$-point quantile grid and defines the projected law as

$$\Pi_m\nu=\frac{1}{m}\sum_{i=1}^m\delta_{F^{-1}_\nu(\tau_i)},$$

where $\tau_i=(2i-1)/(2m)$ are midpoint quantile levels. The corresponding fixed point $\eta_m$ solves

$$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$0

and is referred to as the quantile-projected return distribution (Cheng et al., 9 Jul 2026).

This operator viewpoint is broader than standard expected-return Bellman equations. In a lifted dynamic-programming formulation, the Bellman recursion can act on distributions over state–reward pairs or on more general law-valued states, and quantile projection can then be appended as a representational restriction when one wishes to learn a finite quantile approximation rather than the full law (Bäuerle et al., 27 May 2025).

2. Quantile projection and Wasserstein geometry

The central mathematical fact enabling the quantile-projected equation is that in one dimension the Wasserstein distance is characterized by quantile functions. For probability measures $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$1 with finite $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$2-th moments,

$$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$3

Accordingly, one-dimensional optimal transport is realized by monotone rearrangement, i.e., matching equal quantile levels to one another (Groom et al., 8 May 2026, Dabney et al., 2017).

This has two direct consequences. First, a quantile representation is not merely a convenient parameterization; it is aligned with the natural geometry in which the distributional Bellman operator is contractive. Second, projection onto a finite quantile family can be characterized by selecting the target quantiles at prescribed probability levels. In QR-DQN, the quantile fractions are fixed at midpoints $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$4, and the projected quantile atoms $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$5 approximate $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$6 (Dabney et al., 2017). In the more general projection language used in recent statistical work, the quantile-projected fixed point is the unique solution of the contraction mapping $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$7 under $$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$8 (Cheng et al., 9 Jul 2026).

A common misconception is that any sample-matching loss over return samples constitutes a Wasserstein-compatible projection. The recent flow-based literature explicitly identifies this as false in general: if source and target samples are coupled arbitrarily, the resulting training loss need not respect monotone optimal transport, and therefore need not align with the Wasserstein metric in which the Bellman operator contracts (Groom et al., 8 May 2026). This distinction is central to the phrase “quantile-projected” rather than merely “distributionally approximated.”

3. Classical quantile-based realizations: QR-DQN, IQN, and FQF

The canonical realization of the quantile-projected distributional Bellman equation is QR-DQN. There, the return law at each state–action pair is approximated by a uniform discrete distribution supported on learnable atom locations,

$$T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').$$9

The Bellman-updated target distribution is then projected back onto this family by quantile regression at the fixed midpoint fractions. The paper shows that this realizes the $\Pi$0 projection, and proves that $\Pi$1 is a $\Pi$2-contraction in $\Pi$3 (Dabney et al., 2017).

The training objective is built from the quantile regression loss

$\Pi$4

or its Huber-smoothed version

$\Pi$5

with

$\Pi$6

This quantile loss has unbiased sample gradients for the projection objective, in contrast to direct empirical Wasserstein minimization, whose gradients are biased in general (Dabney et al., 2017).

IQN generalizes the same projected Bellman update by conditioning a network on sampled $\Pi$7, thereby representing the full quantile function rather than a fixed finite set of atoms. The Bellman target is still projected via the same tilted quantile loss, so IQN remains an instance of $\Pi$8 with an implicit quantile-function class (Groom et al., 8 May 2026).

FQF pushes the construction further by learning both the quantile fractions and the quantile values. Its projected Bellman update remains quantile-based, but now the quantile grid itself is adapted through a Wasserstein-driven gradient. For fixed fractions, the optimal atoms satisfy

$\Pi$9

and the partial derivative of the induced $Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$0 objective with respect to an interior quantile fraction is

$Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$1

This makes explicit that the quantile-projected Bellman equation can be instantiated not only with fixed grids but also with learned discretizations of the inverse CDF (Yang et al., 2019).

4. The projection guarantee in FlowIQN

A recent development is the use of conditional flow matching critics for return distributions. Standard CFM critics transport samples from a source distribution to Bellman target samples using a learned velocity field, but they typically employ independent source–target couplings. In one-dimensional return space, that practice induces a metric mismatch: arbitrary pairings can cross quantiles and therefore fail to approximate the monotone coupling that defines Wasserstein distance (Groom et al., 8 May 2026).

FlowIQN resolves this by sorting source quantile fractions and Bellman target samples within each mini-batch, then pairing order statistics. If $Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$2 and $Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$3 denote the sorted source fractions and sorted target samples for a transition, the method constructs linear transport paths

$Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$4

and minimizes

$Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$5

This sorted quantile coupling approximates the monotone transport map and yields a conditional upper bound: $Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$6 where $Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$7 is the critic-induced law and $Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$8 is the Bellman target law (Groom et al., 8 May 2026).

That statement is presented as a Wasserstein-aligned approximate projection guarantee. In effect, FlowIQN supplies a flow-based realization of the quantile-projected Bellman update in one dimension, replacing arbitrary sample pairings with quantile-aligned flow paths. The associated recursion

$Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).$9

shows how Bellman contraction and projection error combine additively when the projection is only approximate (Groom et al., 8 May 2026).

This suggests a conceptual unification: QR-DQN, IQN, FQF, and FlowIQN all implement variants of “Bellman update plus quantile projection,” but differ in their parameterization of the quantile function and in how the projection surrogate is realized computationally.

5. Fixed points, convergence, and statistical efficiency

The fixed-point theory of the quantile-projected Bellman equation has several layers. At the operator level, for fixed policy evaluation, the distributional Bellman operator is contractive in Wasserstein metrics, while the quantile projection is non-expansive in the relevant metric. In the finite-grid setting used in recent statistical analysis, $Q_\theta(\tau|s,a)$0 is a $Q_\theta(\tau|s,a)$1-contraction on $Q_\theta(\tau|s,a)$2, hence admits a unique fixed point $Q_\theta(\tau|s,a)$3 (Cheng et al., 9 Jul 2026).

At the stochastic-approximation level, quantile temporal-difference learning is subtler. Although the projected dynamic-programming operator is contractive in $Q_\theta(\tau|s,a)$4, the QTD update itself is highly non-linear and may have multiple fixed points when quantiles are non-unique. The analysis in “An Analysis of Quantile Temporal-Difference Learning” shows almost sure convergence of tabular QTD to the set of fixed points of a family of projected operators $Q_\theta(\tau|s,a)$5, where $Q_\theta(\tau|s,a)$6 resolves left- versus right-quantile selections on flat CDF plateaus (Rowland et al., 2023). The corresponding family of fixed-point equations is

$Q_\theta(\tau|s,a)$7

Recent statistical work studies the same fixed point from an estimation perspective. Assuming access to a generative model, the empirical MDP induces an empirical operator $Q_\theta(\tau|s,a)$8, and the estimator $Q_\theta(\tau|s,a)$9 is defined by

$$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$0

For fixed $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$1, the non-asymptotic bound

$$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$2

holds with probability at least $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$3, implying $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$4 estimation error and the optimal parametric $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$5 rate for fixed-dimensional quantile policy evaluation (Cheng et al., 9 Jul 2026). The same work derives

$$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$6

and shows that this covariance attains the semiparametric efficiency bound (Cheng et al., 9 Jul 2026).

A plausible implication is that the quantile-projected Bellman equation should now be understood not only as an algorithmic device but also as a statistically well-defined finite-dimensional target. This is an important shift from earlier work, where the emphasis was primarily on approximation quality and contraction properties.

6. Generalizations, applications, and limitations

The quantile-projected Bellman construction is not limited to the classical aleatoric return distribution. In value-distributional model-based RL, the same projection principle is applied to an epistemic value-distribution operator, whose fixed point is the posterior distribution over values induced by model uncertainty. There the update takes the form

$$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$7

with $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$8 again defined as the $$\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),$$9-Wasserstein quantile projection onto a $m$0-quantile representation (Luis et al., 2023). Under independent-transition and acyclicity assumptions, the operator is a $m$1-contraction and repeated application converges to the unique posterior value-distribution fixed point (Luis et al., 2023).

The same operator language also extends to lifted MDPs where the objective is a functional of the joint distribution of terminal state and accumulated reward. In that setting, the Bellman recursion acts on distributions over $m$2 pairs, and quantile MDPs appear as a special case. Appending a standard quantile projection then recovers the quantile-projected distributional Bellman equation as a fixed-policy approximation scheme within a more general distributional dynamic-programming framework (Bäuerle et al., 27 May 2025).

Several limitations remain explicit in the literature.

Limitation	Description
Finite quantile resolution	Projection error depends on $m$3 or $m$4, and discretization vanishes only as the number of quantiles grows (Cheng et al., 9 Jul 2026).
Non-uniqueness on flat CDFs	Multiple projected fixed points can arise, handled via the $m$5 family (Rowland et al., 2023).
One-dimensional transport dependence	FlowIQN’s explicit Wasserstein-aligned guarantee relies on scalar returns and monotone coupling (Groom et al., 8 May 2026).
Approximate rather than exact projection	Flow-based surrogates yield upper bounds or exact equivalence only in special cases such as the single-step shortcut limit (Groom et al., 8 May 2026).
Function-approximation gaps	Convergence and efficiency results are strongest in tabular, generative-model, or fixed-policy settings rather than general deep control (Rowland et al., 2023, Cheng et al., 9 Jul 2026).

A further misconception is to treat downstream control performance as a direct proxy for correctness of the projected return law. The FlowIQN study explicitly distinguishes fixed-policy Wasserstein accuracy from offline control performance, noting that policy extraction, regularization, target-network dynamics, and action support jointly affect end-task outcomes (Groom et al., 8 May 2026).

Taken together, the quantile-projected distributional Bellman equation is best regarded as a unifying template for finite-dimensional distributional policy evaluation and control: the Bellman operator specifies the law-valued target, quantile projection defines the admissible approximation class, and the learning algorithm implements the projection through quantile regression, quantile-conditioned function approximation, or quantile-coupled transport. Across these variants, the defining mathematical structure is the same fixed-point equation $m$6, with one-dimensional Wasserstein geometry supplying both its theoretical justification and its most effective algorithmic realizations (Dabney et al., 2017, Groom et al., 8 May 2026, Cheng et al., 9 Jul 2026).