Fitted Occupancy-Ratio Evaluation (FORE)

• FORE is a reinforcement learning method that estimates the discounted occupancy ratio using fixed-point adjoint Bellman recursion, eliminating the need for Bellman completeness.
• It employs KL-divergence projection onto a log-ratio class, ensuring a stable and supervised learning–like fitting process for accurate ratio estimation.
• FORE supports downstream estimators such as direct reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust methods, with strong statistical guarantees.

Searching arXiv for FORE and closely related offline RL occupancy-ratio / FQE papers. Searching "Fitted Occupancy-Ratio Evaluation without Bellman Completeness" Searching "A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models" Fitted Occupancy-Ratio Evaluation (FORE) is an offline reinforcement-learning method for off-policy evaluation that estimates the discounted occupancy ratio rather than beginning with Bellman regression for a value function. In the formulation introduced in "Fitted Occupancy-Ratio Evaluation without Bellman Completeness" (Laan et al., 6 Jul 2026), FORE is a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion and, at each iteration, solves a single-level density-ratio objective on one-step-transition data by projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. The resulting fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation, and the central theoretical claim is that discounted occupancy-ratio realizability is sufficient for offline policy evaluation without any completeness assumptions (Laan et al., 6 Jul 2026).

1. Offline evaluation target and the discounted occupancy ratio

In offline reinforcement learning, the evaluation problem is to estimate the value of a target policy $\pi$ from data generated by a different behavior policy, often unknown. The central technical difficulty is distribution shift between the offline data distribution and the state-action distribution induced by the target policy. FORE addresses this shift through the discounted occupancy ratio (Laan et al., 6 Jul 2026).

The discounted occupancy ratio is defined as

$\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$

where $d_{\pi,\gamma}$ is the normalized discounted state-action occupancy measure induced by the target policy and $\nu$ is the offline data distribution. Knowledge of $\omega^\pi$ allows transforming sample averages under $\nu$ into expectations under $d_{\pi,\gamma}$. In this sense, the ratio is the correction factor that transports offline-data expectations to target-policy expectations (Laan et al., 6 Jul 2026).

This choice of object distinguishes FORE from value-centric fitted methods. The motivation given for the method is that fitted Q-evaluation (FQE) regresses Bellman targets onto a value-function class, but such regression can be unstable unless a Bellman completeness condition holds, namely that the value-function class is closed under the Bellman operator’s action. The detailed summary characterizes this as a strong and often unrealistic requirement, particularly under function approximation and distribution shift. FORE therefore shifts attention from Bellman-closed value classes to direct estimation of the occupancy ratio itself (Laan et al., 6 Jul 2026).

2. Adjoint Bellman characterization

FORE is built on a fixed-point characterization of the discounted occupancy ratio through an adjoint Bellman equation. The normalized discounted occupancy measure satisfies

$$d_{\pi,\gamma} = (1-\gamma) d_0 + \gamma d_{\pi,\gamma} P_\pi,$$

where $d_0$ is the initial distribution and $P_\pi$ is the target-policy transition kernel. Taking Radon--Nikodym derivatives with respect to $\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$0 yields

$\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$1

The associated adjoint Bellman operator is

$\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$2

and the discounted occupancy ratio is characterized by the fixed-point equation

$\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$3

The significance of this formulation is methodological as well as analytical. Existing primal-dual and minimax ratio estimators are described as enforcing occupancy-balance moments over a critic class, whereas FORE treats occupancy-ratio estimation as a fitted fixed-point recursion for the adjoint Bellman operator. This replaces balance-residual minimax fitting by repeated application of a forward map followed by information projection. A plausible implication is that the key approximation burden is no longer closure of an auxiliary class under Bellman updates, but representability of the ratio itself.

3. KL-projected fitted recursion

FORE parameterizes a normalized positive ratio through a log-ratio hypothesis class,

$\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$4

The use of $\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$5 makes the projection geometry explicitly KL-based.

At the population level, FORE performs two steps at each iteration. First, it applies the adjoint Bellman map $\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$6. Second, it projects the result onto $\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$7 by KL divergence: $\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$8 The resulting recursion is

$\omega^\pi(X) = \frac{d_{\pi,\gamma}(X)}{\nu(X)},$9

Empirically, each step is implementable as a single-level optimization over the log-ratio class: $d_{\pi,\gamma}$0 The detailed summary states that this objective uses offline-data samples and one-step transitions under the target policy, and that the step is convex for linear classes. It further describes the procedure as ordinary supervised learning for linear classes and as easily implementable for neural networks or trees, with no nested loop or saddle-point optimization (Laan et al., 6 Jul 2026).

The geometric alignment is central. The adjoint Bellman map acts in KL geometry, and the projection is also taken in KL divergence. The detailed summary attributes the stability of the fitted iteration to this alignment, together with the Pythagorean property of KL for exponential families.

4. Approximation conditions and statistical guarantees

The principal theoretical departure of FORE is that its central approximation condition is realizability of the discounted occupancy ratio itself, rather than Bellman completeness of a value-function class or projected-operator stability. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio because the adjoint Bellman operator is a KL contraction (Laan et al., 6 Jul 2026).

The basic contraction inequality is

$d_{\pi,\gamma}$1

After projection, the recursion remains contractive up to approximation error: $d_{\pi,\gamma}$2 where

$d_{\pi,\gamma}$3

The detailed summary emphasizes the interpretation: convergence is dictated solely by how well $d_{\pi,\gamma}$4 can be represented in the hypothesis class, not by how well the entire sequence of adjoint Bellman images can be represented.

For the empirical recursion, the reported finite-sample guarantee is a regret-type bound of the form

$d_{\pi,\gamma}$5

The statistical term is governed by the complexity of the log-ratio class; the detailed summary states that for parametric models the statistical rates are $d_{\pi,\gamma}$6, while for nonparametric classes such as Sobolev or Hölder balls the rates are standard. It also records two implementation-relevant points: no splitting of data between steps and uniform error control across iterations (Laan et al., 6 Jul 2026).

5. Relation to fitted Q-evaluation and the role of occupancy ratios

FORE is best understood against the background of recent analysis of fitted Q-evaluation. "A Fine-grained Analysis of Fitted Q-evaluation: Beyond Parametric Models" (Wang et al., 2024) studies FQE under both parametric and nonparametric models and shows that, under completeness of $d_{\pi,\gamma}$7-functions, FQE can attain the optimal $d_{\pi,\gamma}$8 rate for value estimation in both parametric and sufficiently smooth nonparametric settings. It also establishes that, without special ratio-function modeling, the first-order horizon dependence is

$d_{\pi,\gamma}$9

that is, $\nu$0, and that with an additional realizability assumption on the occupancy ratio $\nu$1, the dependence improves to $\nu$2, matching the sharpest known tabular bound (Wang et al., 2024).

That analysis is directly relevant because it identifies the occupancy ratio as a fundamental object for improving off-policy evaluation rates. In that paper, the occupancy ratio is

$\nu$3

and the summary states that FQE can be re-expressed as a kind of marginal importance sampling estimator in which these weights appear. It further states that when $\nu$4 is well approximated in the same sieve basis as $\nu$5, the horizon dependence improves from $\nu$6 to $\nu$7, and that the improvement arises without explicitly modeling $\nu$8 (Wang et al., 2024).

FORE takes a different route. Rather than treating occupancy ratios as a latent analytic device inside FQE, it makes the discounted occupancy ratio the primary fitted object. The abstract of (Laan et al., 6 Jul 2026) explicitly contrasts this with analyses of FQE that require value-function realizability together with Bellman completeness or projected-operator stability. This suggests a conceptual reorientation: the sufficient structural assumption for offline policy evaluation is not Bellman completeness of a value class, but realizability of the discounted occupancy ratio in a log-ratio class.

6. Value estimators built on the fitted ratio

The fitted ratio produced by FORE supports several downstream estimators. The first is direct value estimation by reward reweighting: $\nu$9 Here the ratio estimate directly transports reward expectations from the offline distribution $\omega^\pi$0 to the target discounted occupancy distribution.

The second is doubly robust estimation, which combines a fitted ratio with a fitted $\omega^\pi$1-function: $\omega^\pi$2 The detailed summary states that the value error is doubly robust in the sense that it vanishes if either the ratio or the $\omega^\pi$3-function is consistently estimated, with a product-form error bound (Laan et al., 6 Jul 2026).

The third is occupancy-weighted fitted Q-evaluation, or FORE-weighted FQE: $\omega^\pi$4 The associated guarantee is described as depending only on ratio realizability and value-function approximation error, with Bellman completeness not required (Laan et al., 6 Jul 2026).

Taken together, these constructions make FORE more than a ratio estimator in isolation. It is a framework in which ratio fitting, direct reweighting, weighted Bellman regression, and doubly robust estimation are all organized around the same object: the discounted occupancy ratio. The main theoretical conclusion reported in (Laan et al., 6 Jul 2026) is therefore not merely algorithmic but structural: discounted occupancy-ratio realizability is a sufficient condition for offline policy evaluation without completeness assumptions.