Fitted-Q-Evaluation Estimator

• Fitted-Q-Evaluation-Style Estimator is a policy evaluation method that uses supervised regression on backward Bellman backups to estimate Q-functions from off-policy data.
• The approach recursively fits Q-functions from the terminal stage backward, addressing distribution shift by aligning behavior and target policy distributions.
• Recent extensions incorporate robust, distributional, and CNN-based methods to improve sample efficiency and tighten error bounds in offline reinforcement learning.

Searching arXiv for recent and foundational papers on fitted Q-evaluation and related estimators. A fitted-Q-evaluation-style estimator is a backward, Bellman-residual-minimizing, supervised-learning implementation of policy evaluation under off-policy data. In its canonical form, it estimates the value of a target policy by recursively estimating target-policy $Q$-functions from samples generated by a different behavior policy, initializing the terminal stage, fitting each preceding stage to a Bellman backup of the next-stage estimate, and then plugging the first-stage estimate into the target initial state-action distribution (Ji et al., 2022, Wang et al., 2024). Closely related procedures appear in control, robust evaluation, average-reward offline RL, and distributional off-policy evaluation; in those settings, the common structure is still a regression-based Bellman backup computed from logged transitions rather than a symbolic dynamic-programming solution (Lee et al., 2017, Bruns-Smith et al., 2023, Hong et al., 24 Jun 2025).

1. Canonical formulation

In the finite-horizon setting, the problem is posed on a time-inhomogeneous MDP

$$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$

with unknown transition kernels and reward models. A target policy $\pi=\{\pi_h\}_{h=1}^H$ is to be evaluated from data generated under an unknown behavior policy $\pi_0$, and its value is

$$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$

The stagewise action-value functions satisfy

$$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$

together with the Bellman equation

$Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$

The core off-policy difficulty is distribution shift: the target-policy visitation distribution $q_h^\pi$ generally differs from the behavior distribution $q_h^{\pi_0}$, so evaluation must proceed from samples collected under the wrong policy (Ji et al., 2022).

The standard fitted recursion works backward. With $\widehat Q_{H+1}^\pi=0$, each stage solves a supervised regression problem whose target is the Bellman backup of the next-stage estimate: $$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$0 The value estimator is then

$$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$1

An equivalent finite-horizon episodic description uses an offline dataset

$$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$2

generated by a behavior policy $$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$3, stagewise function classes $$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$4, and a backward least-squares recursion for $$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$5 followed by a plug-in value estimate $$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$6 (Wang et al., 2024).

2. Relation to fitted Q-iteration

Fitted Q-evaluation is the policy-evaluation analogue of fitted Q-iteration, and several papers use the phrase “FQE-style” for estimators that preserve the Bellman-regression template even when the overall algorithm performs policy improvement. In the control setting, the fitted-Q step regresses onto a target of the form

$$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$7

or

$$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$8

rather than onto a fixed-policy continuation value, but the estimator remains a sample-based Bellman backup fitted by supervised regression (Halperin, 2018, Lee et al., 2017).

This distinction is explicit in the QLBS option-pricing framework. There, the fitted-Q component is described as a sample-based estimator of the $$(\mathcal S,\mathcal A,\{P_h\}_{h=1}^H,\{R_h\}_{h=1}^H,H,\xi),$$9-function rather than a direct dynamic-programming solver: it takes a dataset of transitions, forms the standard Bellman target, solves a least-squares problem for the $\pi=\{\pi_h\}_{h=1}^H$0-approximation, and then improves the hedge greedily. The paper emphasizes that the method is off-policy because the observed actions need not be optimal (Halperin, 2018).

A related formulation appears in visual servoing. The learned controller uses a linear-in-parameters $\pi=\{\pi_h\}_{h=1}^H$1, Bellman targets of the form

$\pi=\{\pi_h\}_{h=1}^H$2

and a regularized fitted Q-iteration loop with explicit scale-and-bias fitting followed by a constrained update of $\pi=\{\pi_h\}_{h=1}^H$3. The paper describes this as a fitted Q-iteration method with a parametric Q-evaluation subproblem, because each inner step is still a regression onto sampled Bellman targets (Lee et al., 2017).

Average-reward offline RL extends the same template further. “Anchored Fitted Q-Iteration” defines the regression step

$\pi=\{\pi_h\}_{h=1}^H$4

and then adds an anchor

$\pi=\{\pi_h\}_{h=1}^H$5

This is not policy evaluation in the discounted finite-horizon sense, but it is explicitly described as a fitted-Q-evaluation-style regression scheme embedded in an iteration (Lee et al., 20 Oct 2025).

3. Error bounds, horizon dependence, and distribution shift

A central theoretical question is how the estimation error of FQE depends on sample size, horizon, and off-policy mismatch. A fine-grained analysis of finite-horizon FQE shows that, under a completeness assumption on the $\pi=\{\pi_h\}_{h=1}^H$6-functions, parametric FQE satisfies a high-probability bound with leading term

$\pi=\{\pi_h\}_{h=1}^H$7

written more explicitly as

$\pi=\{\pi_h\}_{h=1}^H$8

When the marginal probability ratio functions are also realizable in the same linear or sieve space, the leading horizon dependence improves from $\pi=\{\pi_h\}_{h=1}^H$9 to $\pi_0$0, matching the sharpest known tabular MIS rate (Wang et al., 2024).

The same analysis establishes a nonparametric picture. With sieve approximation error $\pi_0$1, nonparametric FQE obeys

$\pi_0$2

For fixed horizon, the estimator achieves the optimal $\pi_0$3 rate when $\pi_0$4. The same work also shows that the FQE estimator admits a marginalized importance sampling representation,

$\pi_0$5

which explains why ratio-function realizability sharpens the horizon dependence (Wang et al., 2024).

A different nonparametric line studies deep fitted Q-evaluation on low-dimensional manifolds. Under low-dimensional manifold and Besov smoothness/completeness assumptions, and with an appropriately chosen CNN architecture, the paper proves

$\pi_0$6

up to polylogarithmic factors, with the displayed theorem written as

$\pi_0$7

Here $\pi_0$8 is the intrinsic dimension, not the ambient dimension $\pi_0$9, and $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$0 is a restricted $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$1-divergence defined on a function space tied to the estimator. The paper stresses that this mismatch term can remain small even when the tabular density ratio is huge or unbounded, because it measures behavior-target mismatch only through the function class relevant to the estimator (Ji et al., 2022).

This body of theory suggests that the statistical difficulty of FQE-style estimation is not determined solely by raw state-action coverage. A plausible implication is that function-space geometry, ratio-function realizability, and the norm under which Bellman error is minimized are all first-order objects in the analysis.

4. Function approximation, geometry, and norm alignment

The choice of function class is central. One framework studies FQE with general differentiable function approximators

$$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$2

with $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$3 at least third-order continuously differentiable and with policy completeness

$$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$4

In that setting, FQE is analyzed as a Z-estimator. The asymptotic variance is determined jointly by the tangent space of the function class at the ground truth, the reward structure, and the distribution shift due to off-policy learning, and the paper also derives a function-class-dependent restricted $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$5-divergence in the tangent space $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$6 (Zhang et al., 2022).

The low-dimensional-manifold analysis provides a complementary approximation result for convolutional networks. For any $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$7 in the relevant Besov class on a compact $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$8-dimensional Riemannian manifold embedded in $$v^\pi := \mathbb E^\pi\!\left[\sum_{h=1}^H r_h(s_h,a_h)\,\middle|\,s_1\sim \xi\right].$$9, there exists a CNN $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$0 such that

$$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$1

with architecture complexity depending on $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$2 rather than fundamentally on $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$3. The proof proceeds by decomposing the function on a chart atlas, approximating local pieces by cardinal B-splines, implementing splines and chart indicators by convolutional blocks, and assembling the result into one CNN (Ji et al., 2022).

A further refinement addresses Bellman completeness directly. “Stationary-weighted Fitted Q-Evaluation” argues that standard FQE regresses under the behavior distribution even though the Bellman operator is naturally contractive under the stationary distribution $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$4 of the target policy. The proposed estimator replaces unweighted least squares with

$$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$5

where $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$6 estimates the stationary density ratio. The paper’s main message is that this norm alignment removes the need for Bellman completeness, prevents geometric error blow-up, and keeps the computational structure of regression-based evaluation (Laan et al., 29 Dec 2025).

5. Inference, bootstrap, and estimator selection

FQE-style estimators have also been developed as inferential objects. One bootstrap line studies episode-wise resampling for off-policy inference. Because transitions within an episode are dependent, the bootstrap resamples entire episodes rather than individual tuples, reruns FQE on each resampled dataset, and uses the resulting law of $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$7 for confidence intervals, variance estimation, and correlation estimation. Under policy completeness and linear approximation, the paper proves asymptotic normality, asymptotic efficiency, and bootstrap distributional consistency; it also introduces a subsampled bootstrap with order-of-magnitude runtime improvement (Hao et al., 2021).

The differentiable-function-approximation analysis reaches a similar inferential conclusion by a different route. It derives asymptotic normality for $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$8, valid vanilla and multiplier bootstrap schemes performed by episodes, and a Cramér–Rao lower bound matched by the FQE asymptotic variance. The resulting confidence intervals are asymptotically valid, and the variance formula depends explicitly on the stagewise Jacobians, residual covariance structure, and target-policy sensitivity vectors $$Q_h^\pi(s,a) := \mathbb E^\pi\!\left[\sum_{h'=h}^{H} r_{h'}(s_{h'},a_{h'}) \,\middle|\, s_h=s,a_h=a\right],$$9 (Zhang et al., 2022).

Hyperparameter selection is itself treated as an estimation problem. The approximate hyperparameter selection framework defines a criterion $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$0 satisfying

$Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$1

and then seeks an operator $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$2 whose induced $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$3 is $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$4-optimal. The paper derives four methods: Regret Minimization, Kernel Loss Minimization, and their fixed-point analogues RM-FP and KLM-FP. Its conclusion is that RM and RM-FP are the first completely hyperparameter-free FQE selection methods with formal error guarantees, while KLM and KLM-FP trade extra kernel tuning for potentially tighter modeling of the Bellman residual (Miyaguchi, 2022).

A recurrent practical misconception is that any resampling or tuning procedure can be attached to FQE without regard to its dependence structure or Bellman geometry. The episode-wise bootstrap and AHS results indicate otherwise.

6. Robust, distributional, and application-specific extensions

Robust variants modify the Bellman target itself. Under sequentially exogenous unobserved confounders and a marginal sensitivity model, robust fitted-Q-evaluation replaces the nominal target with a closed-form robust Bellman expression involving a truncated conditional expectation or CVaR-like functional. The computational pipeline becomes behavior-policy estimation, quantile regression, and pseudo-outcome regression, iterated backward in time. An orthogonalized pseudo-outcome cancels the first-order effect of quantile-estimation error, so the impact of estimating the quantile enters quadratically rather than linearly (Bruns-Smith et al., 2023).

Distributional off-policy evaluation extends the fitted template from scalar $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$5-functions to return laws. “Fitted distributional evaluation” defines

$Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$6

where $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$7 is a return distribution and $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$8 is a discrepancy between distributions. The paper’s guiding principles require a contractive metric, a correct population minimizer, and loss-to-metric control, and it identifies functional Bregman divergences as the key family ensuring that the population minimizer is exactly the Bellman backup (Hong et al., 24 Jun 2025).

Application papers show how widely the fitted-Q template travels. In image-based offline policy evaluation, deep FQE with CNNs on CartPole exhibits performance similar across high- and low-resolution observations, consistent with dependence on intrinsic state structure rather than pixel dimension (Ji et al., 2022). In visual servoing, a regularized fitted Q-iteration algorithm learns effective control with just 20 training trajectory samples for reinforcement learning and reports an improvement in sample-efficiency of more than two orders of magnitude over standard model-free deep reinforcement learning algorithms (Lee et al., 2017). In option hedging and pricing, the QLBS fitted-Q method matches the dynamic-programming result in the on-policy case, with the RL/FQI price reported as $Q_h^\pi = \mathcal T_h^\pi Q_{h+1}^\pi.$9, identical to the DP result within MC error, and remains stable under suboptimal actions (Halperin, 2018).

Across these variants, the defining feature is unchanged: a fitted-Q-evaluation-style estimator replaces exact Bellman evaluation with backward regression on Bellman targets computed from logged or simulated transitions. What varies is the target—nominal, robust, anchored, or distributional—the function class, and the geometry used to control approximation and distribution shift.