---
title: Fitted-Q-Evaluation Estimator
url: https://www.emergentmind.com/topics/fitted-q-evaluation-style-estimator
type: topic
---

# Fitted-Q-Evaluation Estimator

Searching arXiv for recent and foundational papers on fitted Q-evaluation and related estimators.
A fitted-Q-evaluation-style estimator is a backward, Bellman-residual-minimizing, supervised-learning implementation of policy evaluation under off-policy data. In its canonical form, it estimates the value of a target policy by recursively estimating target-policy \(Q\)-functions from samples generated by a different behavior policy, initializing the terminal stage, fitting each preceding stage to a Bellman backup of the next-stage estimate, and then plugging the first-stage estimate into the target initial state-action distribution [2206.02887][2406.10438]. Closely related procedures appear in control, robust evaluation, average-reward offline RL, and distributional off-policy evaluation; in those settings, the common structure is still a regression-based Bellman backup computed from logged transitions rather than a symbolic dynamic-programming solution [1703.11000][2302.00662][2506.20048].

## 1. Canonical formulation

In the finite-horizon setting, the problem is posed on a time-inhomogeneous MDP
\[
(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),
\]
with unknown transition kernels and reward models. A target policy \(\pi=\{\pi_h\}_{h=1}^H\) is to be evaluated from data generated under an unknown behavior policy \(\pi_0\), and its value is
\[
v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].
\]
The stagewise action-value functions satisfy
\[
Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],
\]
together with the Bellman equation
\[
Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.
\]
The core off-policy difficulty is distribution shift: the target-policy visitation distribution \(q_h^\pi\) generally differs from the behavior distribution \(q_h^{\pi_0}\), so evaluation must proceed from samples collected under the wrong policy [2206.02887].

The standard fitted recursion works backward. With \(\widehat Q_{H+1}^\pi=0\), each stage solves a supervised regression problem whose target is the Bellman backup of the next-stage estimate:
\[
\widehat Q_h^\pi
=
\arg\min_{f\in\mathcal F}
\sum_{k=1}^{K}
\left(
f(s_{h,k},a_{h,k})
-
r_{h,k}
-
\int_{\mathcal A}\widehat Q_{h+1}^\pi(s'_{h,k},a)\,\pi_h(a\mid s'_{h,k})\,da
\right)^2.
\]
The value estimator is then
\[
\widehat v^\pi
=
\int_{\mathcal X}\widehat Q_1^\pi(s,a)\,\xi(s)\pi(a\mid s)\,ds\,da.
\]
An equivalent finite-horizon episodic description uses an offline dataset
\[
\mathcal D_n=\left\{\{(S_{i,t},A_{i,t},R_{i,t})\}_{t=1}^T\right\}_{i=1}^n
\]
generated by a behavior policy \(\pi^b\), stagewise function classes \(\mathcal Q^{(t)}\), and a backward least-squares recursion for \(\hat Q_t^\pi\) followed by a plug-in value estimate \(\hat\nu(\pi)=\mathbb E_{(S_1,A_1)\sim d_1^\pi}[\hat Q_1^\pi(S_1,A_1)]\) [2406.10438].

## 2. Relation to fitted Q-iteration

Fitted Q-evaluation is the policy-evaluation analogue of fitted Q-iteration, and several papers use the phrase “FQE-style” for estimators that preserve the Bellman-regression template even when the overall algorithm performs policy improvement. In the control setting, the fitted-Q step regresses onto a target of the form
\[
r+\gamma \max_{a'} Q(s',a')
\]
or
\[
c+\gamma \min_u Q(s',u),
\]
rather than onto a fixed-policy continuation value, but the estimator remains a sample-based Bellman backup fitted by supervised regression [1801.06077][1703.11000].

This distinction is explicit in the QLBS option-pricing framework. There, the fitted-Q component is described as a sample-based estimator of the \(Q\)-function rather than a direct dynamic-programming solver: it takes a dataset of transitions, forms the standard Bellman target, solves a least-squares problem for the \(Q\)-approximation, and then improves the hedge greedily. The paper emphasizes that the method is off-policy because the observed actions need not be optimal [1801.06077].

A related formulation appears in visual servoing. The learned controller uses a linear-in-parameters \(Q_{\theta,b}(s,u)\), Bellman targets of the form
\[
c_t + \gamma \min_u Q(s_{t+1},u),
\]
and a regularized fitted Q-iteration loop with explicit scale-and-bias fitting followed by a constrained update of \(\theta\ge 0\). The paper describes this as a fitted Q-iteration method with a parametric Q-evaluation subproblem, because each inner step is still a regression onto sampled Bellman targets [1703.11000].

Average-reward offline RL extends the same template further. “Anchored Fitted Q-Iteration” defines the regression step
\[
\hat{T}f_k
=
\arg\min_{f\in F_{k+1}}
\sum_{i=1}^n
\Big(f(s_i,a_i)-r_i-\max_{a\in A}f_k(s'_i,a)\Big)^2,
\]
and then adds an anchor
\[
f_{k+1}=(1-\lambda_{k+1})f_0+\lambda_{k+1}\hat{T}f_k.
\]
This is not policy evaluation in the discounted finite-horizon sense, but it is explicitly described as a fitted-Q-evaluation-style regression scheme embedded in an iteration [2510.17391].

## 3. Error bounds, horizon dependence, and distribution shift

A central theoretical question is how the estimation error of FQE depends on sample size, horizon, and off-policy mismatch. A fine-grained analysis of finite-horizon FQE shows that, under a completeness assumption on the \(Q\)-functions, parametric FQE satisfies a high-probability bound with leading term
\[
T^{3/2}\sqrt{\kappa/n},
\]
written more explicitly as
\[
|\hat\nu(\pi)-\nu(\pi)|
=
\left(
\sqrt{\frac{T^3\kappa}{n}+T^3\frac{\log n\,\log T}{n}}
\right)
\quad\text{w.h.p.}
\]
When the marginal probability ratio functions are also realizable in the same linear or sieve space, the leading horizon dependence improves from \(T^{1.5}/\sqrt n\) to \(T/\sqrt n\), matching the sharpest known tabular MIS rate [2406.10438].

The same analysis establishes a nonparametric picture. With sieve approximation error \(K^{-\beta_Q}\), nonparametric FQE obeys
\[
|\hat\nu(\pi)-\nu(\pi)|
=
\left(
T^2K^{-\beta_Q}
+
\sqrt{\frac{T^3\kappa}{n}+\frac{T^3K\log n\log T}{n}}
\right)
\quad\text{w.h.p.}
\]
For fixed horizon, the estimator achieves the optimal \(n^{-1/2}\) rate when \(\beta_Q>1\). The same work also shows that the FQE estimator admits a marginalized importance sampling representation,
\[
\hat\nu(\pi)=\sum_{t=1}^T \frac1n\sum_{i=1}^n \hat w_{i,t}R_{i,t},
\]
which explains why ratio-function realizability sharpens the horizon dependence [2406.10438].

A different nonparametric line studies deep fitted Q-evaluation on low-dimensional manifolds. Under low-dimensional manifold and Besov smoothness/completeness assumptions, and with an appropriately chosen CNN architecture, the paper proves
\[
\mathbb E\bigl|v^\pi-\widehat v^\pi\bigr|
\le
C\,H^2\,\kappa\,K^{-\frac{\alpha}{2\alpha+d}}
\]
up to polylogarithmic factors, with the displayed theorem written as
\[
\mathbb E\left|v^\pi-\widehat v^\pi\right|
\le
C\,H^2\,\kappa\,K^{-\frac{\alpha}{2\alpha+d}\log^{5/2}K}.
\]
Here \(d\) is the intrinsic dimension, not the ambient dimension \(D\), and \(\kappa\) is a restricted \(\chi^2\)-divergence defined on a function space tied to the estimator. The paper stresses that this mismatch term can remain small even when the tabular density ratio is huge or unbounded, because it measures behavior-target mismatch only through the function class relevant to the estimator [2206.02887].

This body of theory suggests that the statistical difficulty of FQE-style estimation is not determined solely by raw state-action coverage. A plausible implication is that function-space geometry, ratio-function realizability, and the norm under which Bellman error is minimized are all first-order objects in the analysis.

## 4. Function approximation, geometry, and norm alignment

The choice of function class is central. One framework studies FQE with general differentiable function approximators
\[
\mathcal F=\{f_\theta(\phi(\cdot)):\theta\in\Theta\},
\]
with \(f_\theta\) at least third-order continuously differentiable and with policy completeness
\[
r\in\mathcal F,\qquad \mathcal P f\in\mathcal F \ \text{for all } f\in\mathcal F.
\]
In that setting, FQE is analyzed as a Z-estimator. The asymptotic variance is determined jointly by the tangent space of the function class at the ground truth, the reward structure, and the distribution shift due to off-policy learning, and the paper also derives a function-class-dependent restricted \(\chi^2\)-divergence in the tangent space \(\mathcal G_h\) [2202.04970].

The low-dimensional-manifold analysis provides a complementary approximation result for convolutional networks. For any \(f_0\) in the relevant Besov class on a compact \(d\)-dimensional Riemannian manifold embedded in \(\mathbb R^D\), there exists a CNN \(\widehat f\) such that
\[
\|\widehat f-f_0\|_\infty \le (\widetilde M\widetilde J)^{-\alpha/d},
\]
with architecture complexity depending on \(d\) rather than fundamentally on \(D\). The proof proceeds by decomposing the function on a chart atlas, approximating local pieces by cardinal B-splines, implementing splines and chart indicators by convolutional blocks, and assembling the result into one CNN [2206.02887].

A further refinement addresses Bellman completeness directly. “Stationary-weighted Fitted Q-Evaluation” argues that standard FQE regresses under the behavior distribution even though the Bellman operator is naturally contractive under the stationary distribution \(\mu\) of the target policy. The proposed estimator replaces unweighted least squares with
\[
\widehat{Q}^{(k+1)} \in \arg\min_{f \in \mathcal F}\; \frac{1}{n} \sum_{i=1}^{n} \widehat{w}_\mu(S_i,A_i) \Bigl\{ R_i
  + \gamma\,(\pi \widehat{Q}^{(k)})(S_i') -\, f(S_i,A_i) \Bigr\}^2 ,
\]
where \(\widehat w_\mu\) estimates the stationary density ratio. The paper’s main message is that this norm alignment removes the need for Bellman completeness, prevents geometric error blow-up, and keeps the computational structure of regression-based evaluation [2512.23805].

## 5. Inference, bootstrap, and estimator selection

FQE-style estimators have also been developed as inferential objects. One bootstrap line studies episode-wise resampling for off-policy inference. Because transitions within an episode are dependent, the bootstrap resamples entire episodes rather than individual tuples, reruns FQE on each resampled dataset, and uses the resulting law of \(\hat v_\pi^*-\hat v_\pi\) for confidence intervals, variance estimation, and correlation estimation. Under policy completeness and linear approximation, the paper proves asymptotic normality, asymptotic efficiency, and bootstrap distributional consistency; it also introduces a subsampled bootstrap with order-of-magnitude runtime improvement [2102.03607].

The differentiable-function-approximation analysis reaches a similar inferential conclusion by a different route. It derives asymptotic normality for \(\widehat v_\pi\), valid vanilla and multiplier bootstrap schemes performed by episodes, and a Cramér–Rao lower bound matched by the FQE asymptotic variance. The resulting confidence intervals are asymptotically valid, and the variance formula depends explicitly on the stagewise Jacobians, residual covariance structure, and target-policy sensitivity vectors \(\nu_h\) [2202.04970].

Hyperparameter selection is itself treated as an estimation problem. The approximate hyperparameter selection framework defines a criterion \(\varepsilon(X)\) satisfying
\[
\Delta J(Q^X)\le \varepsilon(X),\qquad \varepsilon(B_\pi)=0,
\]
and then seeks an operator \(X\) whose induced \(Q^X\) is \((\mathcal X,\varepsilon)\)-optimal. The paper derives four methods: Regret Minimization, Kernel Loss Minimization, and their fixed-point analogues RM-FP and KLM-FP. Its conclusion is that RM and RM-FP are the first completely hyperparameter-free FQE selection methods with formal error guarantees, while KLM and KLM-FP trade extra kernel tuning for potentially tighter modeling of the Bellman residual [2201.02300].

A recurrent practical misconception is that any resampling or tuning procedure can be attached to FQE without regard to its dependence structure or Bellman geometry. The episode-wise bootstrap and AHS results indicate otherwise.

## 6. Robust, distributional, and application-specific extensions

Robust variants modify the Bellman target itself. Under sequentially exogenous unobserved confounders and a marginal sensitivity model, robust fitted-Q-evaluation replaces the nominal target with a closed-form robust Bellman expression involving a truncated conditional expectation or CVaR-like functional. The computational pipeline becomes behavior-policy estimation, quantile regression, and pseudo-outcome regression, iterated backward in time. An orthogonalized pseudo-outcome cancels the first-order effect of quantile-estimation error, so the impact of estimating the quantile enters quadratically rather than linearly [2302.00662].

Distributional off-policy evaluation extends the fitted template from scalar \(Q\)-functions to return laws. “Fitted distributional evaluation” defines
\[
\Upsilon_t = \arg\min_{\Upsilon\in M} \frac{1}{|D|}\sum_{(s,a,r,s')\in D} d\!\left(\Upsilon(s,a), \Psi_\pi(r,s',\Upsilon_{t-1})\right),
\]
where \(\Upsilon(s,a)\) is a return distribution and \(d\) is a discrepancy between distributions. The paper’s guiding principles require a contractive metric, a correct population minimizer, and loss-to-metric control, and it identifies functional Bregman divergences as the key family ensuring that the population minimizer is exactly the Bellman backup [2506.20048].

Application papers show how widely the fitted-Q template travels. In image-based offline policy evaluation, deep FQE with CNNs on CartPole exhibits performance similar across high- and low-resolution observations, consistent with dependence on intrinsic state structure rather than pixel dimension [2206.02887]. In visual servoing, a regularized fitted Q-iteration algorithm learns effective control with just 20 training trajectory samples for reinforcement learning and reports an improvement in sample-efficiency of more than two orders of magnitude over standard model-free deep reinforcement learning algorithms [1703.11000]. In option hedging and pricing, the QLBS fitted-Q method matches the dynamic-programming result in the on-policy case, with the RL/FQI price reported as \(4.90 \pm 0.12\), identical to the DP result within MC error, and remains stable under suboptimal actions [1801.06077].

Across these variants, the defining feature is unchanged: a fitted-Q-evaluation-style estimator replaces exact Bellman evaluation with backward regression on Bellman targets computed from logged or simulated transitions. What varies is the target—nominal, robust, anchored, or distributional—the function class, and the geometry used to control approximation and distribution shift.

Source: https://www.emergentmind.com/topics/fitted-q-evaluation-style-estimator