---
title: Quantile-Projected Bellman Equation in RL
url: https://www.emergentmind.com/topics/quantile-projected-distributional-bellman-equation
type: topic
---

# Quantile-Projected Bellman Equation in RL

Searching arXiv for recent and foundational papers on quantile-projected distributional Bellman equations and related distributional RL.
The quantile-projected distributional Bellman equation is the fixed-point relation obtained by composing a distributional Bellman operator with a projection onto a quantile-parameterized family of return distributions. In distributional reinforcement learning, this construction formalizes the step “apply the Bellman update to a return law, then project the resulting target law back into a finite or implicit quantile representation,” thereby preserving compatibility with Wasserstein geometry and quantile regression training objectives. In recent work, the same template has been extended from classical quantile critics to flow-based critics, most explicitly in FlowIQN, where sorted quantile coupling is used to operationalize a Wasserstein-aligned approximate projection in one-dimensional return space [2605.08515]. The formulation also admits statistical analysis as a finite-dimensional fixed point, including non-asymptotic error bounds and semiparametric efficiency results for policy evaluation under a generative model [2607.08444].

## 1. Definition and operator-theoretic formulation

In discounted reinforcement learning, the distributional Bellman operator for a fixed policy $\pi$ maps a candidate return distribution to the next-step target law induced by immediate reward and discounted continuation. In the scalar-return setting used in distributional RL, this is written as
\[
T^\pi Z(s,a) \stackrel{D}{=} R(s,a) + \gamma Z(s',a'),\qquad s' \sim P(\cdot|s,a),\; a' \sim \pi(\cdot|s').
\]
This law-valued recursion underlies both classical return-distributional RL and later quantile parameterizations [1710.10044, 2605.08515].

The quantile-projected distributional Bellman equation augments this recursion with a projection $\Pi$ onto a restricted family of quantile functions or quantile-supported atomic laws. In its general form,
\[
Z_\theta = \Pi\!\big(T^\pi Z_\theta\big).
\]
When the critic is represented by a parametric quantile function $Q_\theta(\tau|s,a)$, the projection is defined as a Wasserstein-aligned projection onto the quantile family:
\[
\Pi(Z)(s,a) = \arg\min_\theta W_p\!\big(Z(s,a),Q_\theta(\cdot|s,a)\big),
\]
with the metric computed through monotone coupling of quantiles in one dimension [2605.08515].

A finite-dimensional version uses an $m$-point quantile grid and defines the projected law as
\[
\Pi_m\nu=\frac{1}{m}\sum_{i=1}^m\delta_{F^{-1}_\nu(\tau_i)},
\]
where $\tau_i=(2i-1)/(2m)$ are midpoint quantile levels. The corresponding fixed point $\eta_m$ solves
\[
\bm\eta=\Pi_m \mathcal{T}^\pi \bm\eta,
\]
and is referred to as the quantile-projected return distribution [2607.08444].

This operator viewpoint is broader than standard expected-return Bellman equations. In a lifted dynamic-programming formulation, the Bellman recursion can act on distributions over state–reward pairs or on more general law-valued states, and quantile projection can then be appended as a representational restriction when one wishes to learn a finite quantile approximation rather than the full law [2505.21098].

## 2. Quantile projection and Wasserstein geometry

The central mathematical fact enabling the quantile-projected equation is that in one dimension the Wasserstein distance is characterized by quantile functions. For probability measures $\mu,\nu$ with finite $p$-th moments,
\[
W_p^p(\mu,\nu)=\int_0^1 |F_\mu^{-1}(\tau)-F_\nu^{-1}(\tau)|^p\,d\tau.
\]
Accordingly, one-dimensional optimal transport is realized by monotone rearrangement, i.e., matching equal quantile levels to one another [2605.08515, 1710.10044].

This has two direct consequences. First, a quantile representation is not merely a convenient parameterization; it is aligned with the natural geometry in which the distributional Bellman operator is contractive. Second, projection onto a finite quantile family can be characterized by selecting the target quantiles at prescribed probability levels. In QR-DQN, the quantile fractions are fixed at midpoints $\tau_i=(i-1/2)/N$, and the projected quantile atoms $\theta_i$ approximate $F_Z^{-1}(\tau_i)$ [1710.10044]. In the more general projection language used in recent statistical work, the quantile-projected fixed point is the unique solution of the contraction mapping $\Pi_m\mathcal{T}^\pi$ under $\bar W_\infty$ [2607.08444].

A common misconception is that any sample-matching loss over return samples constitutes a Wasserstein-compatible projection. The recent flow-based literature explicitly identifies this as false in general: if source and target samples are coupled arbitrarily, the resulting training loss need not respect monotone optimal transport, and therefore need not align with the Wasserstein metric in which the Bellman operator contracts [2605.08515]. This distinction is central to the phrase “quantile-projected” rather than merely “distributionally approximated.”

## 3. Classical quantile-based realizations: QR-DQN, IQN, and FQF

The canonical realization of the quantile-projected distributional Bellman equation is QR-DQN. There, the return law at each state–action pair is approximated by a uniform discrete distribution supported on learnable atom locations,
\[
Z_\theta(s,a)=\frac{1}{N}\sum_{i=1}^N \delta_{\theta_i(s,a)}.
\]
The Bellman-updated target distribution is then projected back onto this family by quantile regression at the fixed midpoint fractions. The paper shows that this realizes the $W_1$ projection, and proves that $\Pi_Q T^\pi$ is a $\gamma$-contraction in $\overline d_\infty$ [1710.10044].

The training objective is built from the quantile regression loss
\[
\rho_\tau(u)=u\big(\tau-\mathbf{1}\{u<0\}\big),
\]
or its Huber-smoothed version
\[
\rho^\kappa_\tau(u)=\big|\tau-\mathbf{1}\{u<0\}\big|\,L_\kappa(u),
\]
with
\[
L_\kappa(u)=
\begin{cases}
\frac12 u^2,& |u|\le \kappa,\\
\kappa\big(|u|-\frac12\kappa\big),& |u|>\kappa.
\end{cases}
\]
This quantile loss has unbiased sample gradients for the projection objective, in contrast to direct empirical Wasserstein minimization, whose gradients are biased in general [1710.10044].

IQN generalizes the same projected Bellman update by conditioning a network on sampled $\tau\sim \mathrm{Unif}[0,1]$, thereby representing the full quantile function rather than a fixed finite set of atoms. The Bellman target is still projected via the same tilted quantile loss, so IQN remains an instance of $\Pi(T^\pi\cdot)$ with an implicit quantile-function class [2605.08515].

FQF pushes the construction further by learning both the quantile fractions and the quantile values. Its projected Bellman update remains quantile-based, but now the quantile grid itself is adapted through a Wasserstein-driven gradient. For fixed fractions, the optimal atoms satisfy
\[
\theta_i^*=F_Z^{-1}(\hat\tau_i),
\]
and the partial derivative of the induced $W_1$ objective with respect to an interior quantile fraction is
\[
\frac{\partial W_1}{\partial \tau_i}=2F_Z^{-1}(\tau_i)-F_Z^{-1}(\hat\tau_i)-F_Z^{-1}(\hat\tau_{i-1}).
\]
This makes explicit that the quantile-projected Bellman equation can be instantiated not only with fixed grids but also with learned discretizations of the inverse CDF [1911.02140].

## 4. The projection guarantee in FlowIQN

A recent development is the use of conditional flow matching critics for return distributions. Standard CFM critics transport samples from a source distribution to Bellman target samples using a learned velocity field, but they typically employ independent source–target couplings. In one-dimensional return space, that practice induces a metric mismatch: arbitrary pairings can cross quantiles and therefore fail to approximate the monotone coupling that defines Wasserstein distance [2605.08515].

FlowIQN resolves this by sorting source quantile fractions and Bellman target samples within each mini-batch, then pairing order statistics. If $\tilde\tau_{i,1}\le\cdots\le \tilde\tau_{i,K}$ and $\tilde y_{i,1}\le\cdots\le \tilde y_{i,K}$ denote the sorted source fractions and sorted target samples for a transition, the method constructs linear transport paths
\[
z_{i,k,t}=(1-t)\tilde z_{0,i,k}+t\tilde y_{i,k},
\qquad
u_{i,k}=\tilde y_{i,k}-\tilde z_{0,i,k},
\]
and minimizes
\[
L_{\mathrm{FlowIQN}}(\theta)=\mathbb{E}_{i,k,t}\Big[\big|v_\theta(t,z_{i,k,t},s_i,a_i,\tilde\tau_{i,k})-(\tilde y_{i,k}-\tilde z_{0,i,k})\big|^2\Big].
\]
This sorted quantile coupling approximates the monotone transport map and yields a conditional upper bound:
\[
W_2^2\big(\eta_\theta(\cdot|s,a),\nu(\cdot|s,a)\big)\le L_{\mathrm{FlowIQN}}(\theta;s,a),
\]
where $\eta_\theta(\cdot|s,a)$ is the critic-induced law and $\nu(\cdot|s,a)$ is the Bellman target law [2605.08515].

That statement is presented as a Wasserstein-aligned approximate projection guarantee. In effect, FlowIQN supplies a flow-based realization of the quantile-projected Bellman update in one dimension, replacing arbitrary sample pairings with quantile-aligned flow paths. The associated recursion
\[
\bar d_p(Z_{k+1},Z^\pi)\le \gamma \bar d_p(Z_k,Z^\pi)+\epsilon_k
\]
shows how Bellman contraction and projection error combine additively when the projection is only approximate [2605.08515].

This suggests a conceptual unification: QR-DQN, IQN, FQF, and FlowIQN all implement variants of “Bellman update plus quantile projection,” but differ in their parameterization of the quantile function and in how the projection surrogate is realized computationally.

## 5. Fixed points, convergence, and statistical efficiency

The fixed-point theory of the quantile-projected Bellman equation has several layers. At the operator level, for fixed policy evaluation, the distributional Bellman operator is contractive in Wasserstein metrics, while the quantile projection is non-expansive in the relevant metric. In the finite-grid setting used in recent statistical analysis, $\Pi_m\mathcal{T}^\pi$ is a $\gamma$-contraction on $(\mathcal{P},\bar W_\infty)$, hence admits a unique fixed point $\bm\eta_m$ [2607.08444].

At the stochastic-approximation level, quantile temporal-difference learning is subtler. Although the projected dynamic-programming operator is contractive in $\bar w_\infty$, the QTD update itself is highly non-linear and may have multiple fixed points when quantiles are non-unique. The analysis in “An Analysis of Quantile Temporal-Difference Learning” shows almost sure convergence of tabular QTD to the set of fixed points of a family of projected operators $\{\Pi^\lambda T^\pi\}$, where $\lambda$ resolves left- versus right-quantile selections on flat CDF plateaus [2301.04462]. The corresponding family of fixed-point equations is
\[
Z^*=\Pi^\lambda T^\pi Z^*.
\]

Recent statistical work studies the same fixed point from an estimation perspective. Assuming access to a generative model, the empirical MDP induces an empirical operator $\mathcal{T}_n^\pi$, and the estimator $\bm\eta_m^{(n)}$ is defined by
\[
\bm\eta_m^{(n)}=\Pi_m\mathcal{T}_n^\pi \bm\eta_m^{(n)}.
\]
For fixed $m$, the non-asymptotic bound
\[
\Bar W_\infty(\bm\eta_m^{(n)},\bm\eta_m)\le C_1(\mathcal M)\sqrt{\frac{m\,\log(6|\mathcal S||\mathcal A|m/\delta)}{n}}
\]
holds with probability at least $1-\delta$, implying $\widetilde O(\sqrt{m/n})$ estimation error and the optimal parametric $\sqrt n$ rate for fixed-dimensional quantile policy evaluation [2607.08444]. The same work derives
\[
\sqrt n(\bm\theta_m^{(n)}-\bm\theta_m)\Rightarrow \mathcal N\big(\mathbf 0_m,\;K_m^{-1}\Lambda_m K_m^{-\top}\big),
\]
and shows that this covariance attains the semiparametric efficiency bound [2607.08444].

A plausible implication is that the quantile-projected Bellman equation should now be understood not only as an algorithmic device but also as a statistically well-defined finite-dimensional target. This is an important shift from earlier work, where the emphasis was primarily on approximation quality and contraction properties.

## 6. Generalizations, applications, and limitations

The quantile-projected Bellman construction is not limited to the classical aleatoric return distribution. In value-distributional model-based RL, the same projection principle is applied to an epistemic value-distribution operator, whose fixed point is the posterior distribution over values induced by model uncertainty. There the update takes the form
\[
q_i^{\text{new}}(s)=\big(\Pi_Q\circ \mathcal T^\pi Z\big)_{\tau_i}(s),
\]
with $\Pi_Q$ again defined as the $1$-Wasserstein quantile projection onto a $K$-quantile representation [2308.06590]. Under independent-transition and acyclicity assumptions, the operator is a $\gamma$-contraction and repeated application converges to the unique posterior value-distribution fixed point [2308.06590].

The same operator language also extends to lifted MDPs where the objective is a functional of the joint distribution of terminal state and accumulated reward. In that setting, the Bellman recursion acts on distributions over $(x,s)$ pairs, and quantile MDPs appear as a special case. Appending a standard quantile projection then recovers the quantile-projected distributional Bellman equation as a fixed-policy approximation scheme within a more general distributional dynamic-programming framework [2505.21098].

Several limitations remain explicit in the literature.

| Limitation | Description |
|---|---|
| Finite quantile resolution | Projection error depends on $m$ or $K$, and discretization vanishes only as the number of quantiles grows [2607.08444]. |
| Non-uniqueness on flat CDFs | Multiple projected fixed points can arise, handled via the $\Pi^\lambda$ family [2301.04462]. |
| One-dimensional transport dependence | FlowIQN’s explicit Wasserstein-aligned guarantee relies on scalar returns and monotone coupling [2605.08515]. |
| Approximate rather than exact projection | Flow-based surrogates yield upper bounds or exact equivalence only in special cases such as the single-step shortcut limit [2605.08515]. |
| Function-approximation gaps | Convergence and efficiency results are strongest in tabular, generative-model, or fixed-policy settings rather than general deep control [2301.04462, 2607.08444]. |

A further misconception is to treat downstream control performance as a direct proxy for correctness of the projected return law. The FlowIQN study explicitly distinguishes fixed-policy Wasserstein accuracy from offline control performance, noting that policy extraction, regularization, target-network dynamics, and action support jointly affect end-task outcomes [2605.08515].

Taken together, the quantile-projected distributional Bellman equation is best regarded as a unifying template for finite-dimensional distributional policy evaluation and control: the Bellman operator specifies the law-valued target, quantile projection defines the admissible approximation class, and the learning algorithm implements the projection through quantile regression, quantile-conditioned function approximation, or quantile-coupled transport. Across these variants, the defining mathematical structure is the same fixed-point equation $Z=\Pi(T^\pi Z)$, with one-dimensional Wasserstein geometry supplying both its theoretical justification and its most effective algorithmic realizations [1710.10044, 2605.08515, 2607.08444].

Source: https://www.emergentmind.com/topics/quantile-projected-distributional-bellman-equation