---
title: Fitted Occupancy-Ratio Evaluation (FORE)
url: https://www.emergentmind.com/topics/fitted-occupancy-ratio-evaluation-fore
type: topic
---

# Fitted Occupancy-Ratio Evaluation (FORE)

Searching arXiv for FORE and closely related offline RL occupancy-ratio / FQE papers.
Searching arXiv: "Fitted Occupancy-Ratio Evaluation without Bellman Completeness"
Searching arXiv: "A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models"
Fitted Occupancy-Ratio Evaluation (FORE) is an offline reinforcement-learning method for off-policy evaluation that estimates the **discounted occupancy ratio** rather than beginning with Bellman regression for a value function. In the formulation introduced in "Fitted Occupancy-Ratio Evaluation without Bellman Completeness" [2607.05375], FORE is a fitted fixed-point method that characterizes the discounted occupancy ratio through an **adjoint Bellman recursion** and, at each iteration, solves a **single-level density-ratio objective** on one-step-transition data by projecting the adjoint Bellman image onto a **log-ratio class** in **Kullback--Leibler (KL) divergence**. The resulting fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation, and the central theoretical claim is that **discounted occupancy-ratio realizability** is sufficient for offline policy evaluation **without any completeness assumptions** [2607.05375].

## 1. Offline evaluation target and the discounted occupancy ratio

In offline reinforcement learning, the evaluation problem is to estimate the value of a target policy \(\pi\) from data generated by a different behavior policy, often unknown. The central technical difficulty is **distribution shift** between the offline data distribution and the state-action distribution induced by the target policy. FORE addresses this shift through the discounted occupancy ratio [2607.05375].

The discounted occupancy ratio is defined as
\[
\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},
\]
where \(d_{\pi,\gamma}\) is the normalized discounted state-action occupancy measure induced by the target policy and \(\nu\) is the offline data distribution. Knowledge of \(\omega^\pi\) allows transforming sample averages under \(\nu\) into expectations under \(d_{\pi,\gamma}\). In this sense, the ratio is the correction factor that transports offline-data expectations to target-policy expectations [2607.05375].

This choice of object distinguishes FORE from value-centric fitted methods. The motivation given for the method is that fitted Q-evaluation (FQE) regresses Bellman targets onto a value-function class, but such regression can be unstable unless a **Bellman completeness** condition holds, namely that the value-function class is closed under the Bellman operator’s action. The detailed summary characterizes this as a strong and often unrealistic requirement, particularly under function approximation and distribution shift. FORE therefore shifts attention from Bellman-closed value classes to direct estimation of the occupancy ratio itself [2607.05375].

## 2. Adjoint Bellman characterization

FORE is built on a fixed-point characterization of the discounted occupancy ratio through an adjoint Bellman equation. The normalized discounted occupancy measure satisfies
\[
d_{\pi,\gamma} = (1-\gamma) d_0 + \gamma d_{\pi,\gamma} P_\pi,
\]
where \(d_0\) is the initial distribution and \(P_\pi\) is the target-policy transition kernel. Taking Radon--Nikodym derivatives with respect to \(\nu\) yields
\[
\omega = (1-\gamma)\omega_0 + \gamma \frac{d[(\omega \nu) P_\pi]}{d\nu}.
\]

The associated **adjoint Bellman operator** is
\[
B\omega := (1-\gamma)\omega_0 + \gamma \frac{d[(\omega \nu) P_\pi]}{d\nu},
\]
and the discounted occupancy ratio is characterized by the fixed-point equation
\[
\omega^\pi = B\omega^\pi.
\]

The significance of this formulation is methodological as well as analytical. Existing primal-dual and minimax ratio estimators are described as enforcing occupancy-balance moments over a critic class, whereas FORE treats occupancy-ratio estimation as a **fitted fixed-point recursion** for the adjoint Bellman operator. This replaces balance-residual minimax fitting by repeated application of a forward map followed by information projection. A plausible implication is that the key approximation burden is no longer closure of an auxiliary class under Bellman updates, but representability of the ratio itself.

## 3. KL-projected fitted recursion

FORE parameterizes a normalized positive ratio through a log-ratio hypothesis class,
\[
\mathcal{W} = \{\omega_h : h \in \mathcal{H}\}, \qquad
\omega_h(x) = \exp(h(x) - \Lambda_\nu(h)), \qquad
\Lambda_\nu(h) = \log E_\nu e^{h(X)}.
\]
The use of \(\exp(h - \Lambda_\nu(h))\) makes the projection geometry explicitly KL-based.

At the population level, FORE performs two steps at each iteration. First, it applies the adjoint Bellman map \(B\omega\). Second, it projects the result onto \(\mathcal{W}\) by KL divergence:
\[
\Pi_{\mathcal{W}}^{\mathrm{KL}} u
= \arg\min_{v \in \mathcal{W}} D_\nu(u \| v).
\]
The resulting recursion is
\[
\omega^{(k+1)} := \mathcal{T}_{\mathcal{W}}^{\mathrm{KL}} \omega^{(k)}
= \Pi_{\mathcal{W}}^{\mathrm{KL}} B\omega^{(k)}.
\]

Empirically, each step is implementable as a single-level optimization over the log-ratio class:
\[
\arg\min_{h \in \mathcal{H}}
\left\{
\Lambda_\nu(h)
- (1-\gamma) E_{d_0}[h(X)]
- \gamma E_\nu[\omega^{(k)}(X) h(X^+)]
\right\}.
\]
The detailed summary states that this objective uses offline-data samples and one-step transitions under the target policy, and that the step is **convex for linear classes**. It further describes the procedure as ordinary supervised learning for linear classes and as easily implementable for neural networks or trees, with **no nested loop or saddle-point optimization** [2607.05375].

The geometric alignment is central. The adjoint Bellman map acts in KL geometry, and the projection is also taken in KL divergence. The detailed summary attributes the stability of the fitted iteration to this alignment, together with the Pythagorean property of KL for exponential families.

## 4. Approximation conditions and statistical guarantees

The principal theoretical departure of FORE is that its central approximation condition is **realizability of the discounted occupancy ratio itself**, rather than Bellman completeness of a value-function class or projected-operator stability. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio because the adjoint Bellman operator is a KL contraction [2607.05375].

The basic contraction inequality is
\[
D_\nu(B\omega \| B\tilde{\omega}) \leq \gamma D_\nu(\omega \| \tilde{\omega}).
\]
After projection, the recursion remains contractive up to approximation error:
\[
D_\nu(\mathcal{T}_{\mathcal{W}}^{\mathrm{KL}} \omega \| \omega^\pi)
\leq
\gamma D_\nu(\omega \| \omega^\pi)
+
C_{\mathrm{app}} \,\varepsilon_{\mathrm{KL}}^2,
\]
where
\[
\varepsilon_{\mathrm{KL}}
:=
\inf_{v \in \mathcal{W}}
\|\log \omega^\pi - \log v\|_{L^2(\nu)}.
\]
The detailed summary emphasizes the interpretation: convergence is dictated solely by how well \(\log \omega^\pi\) can be represented in the hypothesis class, **not** by how well the entire sequence of adjoint Bellman images can be represented.

For the empirical recursion, the reported finite-sample guarantee is a regret-type bound of the form
\[
D_\nu^{\mathrm{gen}}(\widehat{\omega}^{(K)} \| \omega^\pi)
=
O\!\left(
\left(\frac{1+\gamma}{2}\right)^K
D_\nu^{\mathrm{gen}}(\widehat{\omega}^{(0)} \| \omega^\pi)
+
\frac{\varepsilon_{\mathrm{KL}}^2}{1-\gamma}
+
\frac{\mathrm{Statistical\ Error}}{(1-\gamma)^2}
\right).
\]
The statistical term is governed by the complexity of the log-ratio class; the detailed summary states that for **parametric models** the statistical rates are \(\tilde{O}(d/n)\), while for **nonparametric classes** such as Sobolev or Hölder balls the rates are standard. It also records two implementation-relevant points: **no splitting of data between steps** and **uniform error control across iterations** [2607.05375].

## 5. Relation to fitted Q-evaluation and the role of occupancy ratios

FORE is best understood against the background of recent analysis of fitted Q-evaluation. "A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models" [2406.10438] studies FQE under both parametric and nonparametric models and shows that, under **completeness** of \(Q\)-functions, FQE can attain the optimal \(n^{-1/2}\) rate for value estimation in both parametric and sufficiently smooth nonparametric settings. It also establishes that, without special ratio-function modeling, the first-order horizon dependence is
\[
O\!\left(\sqrt{\frac{T^3 \kappa}{n}}\right),
\]
that is, \(T^{1.5}/\sqrt{n}\), and that with an additional realizability assumption on the occupancy ratio \(w_t^\pi\), the dependence improves to \(T/\sqrt{n}\), matching the sharpest known tabular bound [2406.10438].

That analysis is directly relevant because it identifies the **occupancy ratio** as a fundamental object for improving off-policy evaluation rates. In that paper, the occupancy ratio is
\[
w_t^\pi(s,a)=\frac{d_t^\pi(s,a)}{d_t^b(s,a)},
\]
and the summary states that FQE can be re-expressed as a kind of marginal importance sampling estimator in which these weights appear. It further states that when \(w_t^\pi\) is well approximated in the same sieve basis as \(Q\), the horizon dependence improves from \(T^{1.5}/\sqrt{n}\) to \(T/\sqrt{n}\), and that the improvement arises **without explicitly modeling** \(w_t^\pi\) [2406.10438].

FORE takes a different route. Rather than treating occupancy ratios as a latent analytic device inside FQE, it makes the discounted occupancy ratio the **primary fitted object**. The abstract of [2607.05375] explicitly contrasts this with analyses of FQE that require value-function realizability together with Bellman completeness or projected-operator stability. This suggests a conceptual reorientation: the sufficient structural assumption for offline policy evaluation is not Bellman completeness of a value class, but realizability of the discounted occupancy ratio in a log-ratio class.

## 6. Value estimators built on the fitted ratio

The fitted ratio produced by FORE supports several downstream estimators. The first is **direct value estimation by reward reweighting**:
\[
V_\pi(r) = E_\nu[\omega_{\mathrm{fit}}(X) r(X)].
\]
Here the ratio estimate directly transports reward expectations from the offline distribution \(\nu\) to the target discounted occupancy distribution.

The second is **doubly robust estimation**, which combines a fitted ratio with a fitted \(Q\)-function:
\[
\Psi_{\mathrm{DR}}(\omega,Q)
=
(1-\gamma) E_{d_0}[Q(X)]
+
E_\nu\!\left[
\omega(X)\left\{
r(X) + \gamma E_{\pi}[Q(S',A') \mid S'] - Q(X)
\right\}
\right].
\]
The detailed summary states that the value error is **doubly robust** in the sense that it vanishes if either the ratio or the \(Q\)-function is consistently estimated, with a product-form error bound [2607.05375].

The third is **occupancy-weighted fitted Q-evaluation**, or **FORE-weighted FQE**:
\[
Q^{(k+1)}
=
\arg\min_{q \in \mathcal{Q}}
E_\nu\!\left[
\omega_{\mathrm{fit}}(X)
\left(
r(X) + \gamma E_\pi[Q^{(k)}(S',A')] - q(X)
\right)^2
\right].
\]
The associated guarantee is described as depending only on **ratio realizability** and **value-function approximation error**, with **Bellman completeness not required** [2607.05375].

Taken together, these constructions make FORE more than a ratio estimator in isolation. It is a framework in which ratio fitting, direct reweighting, weighted Bellman regression, and doubly robust estimation are all organized around the same object: the discounted occupancy ratio. The main theoretical conclusion reported in [2607.05375] is therefore not merely algorithmic but structural: **discounted occupancy-ratio realizability** is a sufficient condition for offline policy evaluation without completeness assumptions.

Source: https://www.emergentmind.com/topics/fitted-occupancy-ratio-evaluation-fore