Papers
Topics
Authors
Recent
Search
2000 character limit reached

Fitted Occupancy-Ratio Evaluation without Bellman Completeness

Published 6 Jul 2026 in stat.ML and cs.LG | (2607.05375v1)

Abstract: Occupancy ratios correct distribution shift in offline reinforcement learning and are central to off-policy evaluation. Existing primal-dual and minimax methods typically estimate these ratios by enforcing occupancy-balance moments over a critic class. We propose fitted occupancy-ratio evaluation (FORE), a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion. At each iteration, FORE solves a single-level density-ratio objective on one-step-transition data, thereby projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence. Unlike analyses of fitted Q-evaluation, which typically require value-function realizability together with Bellman completeness or projected-operator stability, our central approximation condition is just realizability of the discounted occupancy ratio itself. Under this condition, the population KL-projected recursion contracts in relative entropy toward the true ratio by virtue of the adjoint Bellman operator being a KL-contraction. For the empirical recursion, we establish finite-sample regret bounds that yield convergence in KL up to log-ratio approximation error and a statistical error governed by the complexity of the ratio hypothesis class. The fitted ratio supports direct value estimation by reward reweighting, occupancy-weighted fitted Q-evaluation, and doubly robust estimation that combines the fitted ratio with a fitted Q-function. Together, these results identify discounted occupancy-ratio realizability as a sufficient condition for offline policy evaluation without any completeness assumptions.

Summary

  • The paper introduces FORE, a fixed-point approach to estimate occupancy ratios without requiring Bellman completeness.
  • It leverages the contraction property of the adjoint Bellman operator and KL-projection to achieve geometric convergence.
  • The method offers rigorous finite-sample guarantees and improved error scaling for long-horizon RL tasks.

Fitted Occupancy-Ratio Evaluation Without Bellman Completeness: A Technical Summary

Motivation and Context

Offline policy evaluation (OPE) in reinforcement learning seeks to estimate the performance of a target policy using data generated by a different (behavior) policy. This requires correcting for distribution mismatch between the offline data distribution and the discounted occupancy measure of the target policy. Classic value-based approaches, such as Fitted Q Evaluation (FQE), rely on function approximation and regression against Bellman targets. However, well-posedness and convergence of FQE typically require either Bellman completeness (ensuring closure of the value-function class under the Bellman operator), or various projection-stability conditions, which can be restrictive or fail in practice. Ratio-based OPE methods, by contrast, focus on directly estimating the density ratio between the target and behavior occupancy distributions, but standard minimax/primal-dual approaches place heavy requirements on the critic class or demand dual/adjoint completeness.

The paper "Fitted Occupancy-Ratio Evaluation without Bellman Completeness" (FORE) (2607.05375) develops a fitted fixed-point approach—Fitted Occupancy-Ratio Evaluation (FORE)—to directly estimate the discounted state-action occupancy ratio using a contraction based on the adjoint Bellman operator in Kullback–Leibler divergence geometry, sidestepping the completeness conditions that afflict FQE and minimax ratio estimation.

FORE: Methodological Contributions

The core contribution is a fitted-iteration procedure for occupancy-ratio estimation whereby each update consists of projecting the adjoint Bellman image onto a normalized log-ratio class with respect to the KL divergence. The recursion is informed by the following:

  • Adjoint Bellman Operator: The true discounted occupancy ratio satisfies a fixed-point equation involving an adjoint Bellman operator, which is a contraction in KL divergence. Each step brings estimates closer to the true ratio geometrically at rate γ\gamma.
  • KL Projection: Rather than solving a minimax saddle point, each iteration uses standard supervised learning (including deep learning or boosted trees) to minimize a population (or empirical) KL loss between the adjoint Bellman map's image and the candidate ratio in the log-density class.

Crucially, the only approximation requirement is that the discounted occupancy ratio itself is (approximately) expressible in the log-ratio class used, eliminating the need for Bellman, dual, or adjoint completeness.

Theoretical Guarantees

The paper provides rigorous results on the population and finite-sample behavior of FORE:

  1. Population-Level Contraction: The KL-projected adjoint Bellman operator is a γ\gamma-contraction up to the log-ratio approximation error. This yields geometric convergence to the true ratio if realizability holds, or to within an L2(ν)L^2(\nu)-log approximation error otherwise.
  2. Finite-Sample Error Bounds: The empirical recursion satisfies finite-sample regret bounds. Error decomposes into: (i) geometric decay of initialization, (ii) squared log-ratio approximation error, and (iii) a bounded statistical term controlled tightly by the Rademacher complexity of the log-ratio class.
  3. Horizon Dependence: For RL problems with long effective horizons, the direct approximation error scales as 1/1−γ1/\sqrt{1-\gamma}, in contrast to the 1/(1−γ)1/(1-\gamma) dependence typical under Bellman-completeness-based analysis. The statistical error remains at the expected 1/(1−γ)21/(1-\gamma)^2 scaling.

The key theoretical innovation is the avoidance of any completeness assumption: Given enough expressive power in the log-ratio class, global contraction in KL between iterates and the truth is ensured, regardless of the adjoint Bellman images' representability.

Applications to Off-Policy Evaluation

FORE provides a versatile tool for policy evaluation in several respects:

  • Reward Reweighting: The fitted occupancy ratio can be used for direct reward reweighting, yielding unbiased value estimates under the target policy.
  • Doubly Robust Estimation: By coupling the fitted ratio with a Q-function estimate, a doubly robust estimator is formed—guaranteed to be consistent if either the ratio or Q-estimate is consistent, with explicit product-form error bounds.
  • Occupancy-Weighted FQE: The FORE-estimated ratio can serve as reweighting for Bellman projections in FQE, restoring contraction even when Bellman completeness fails. The authors derive explicit finite-sample and bias bounds for this fully off-policy variant.

Distinguishing Point: In contrast to existing minimax/primal-dual density ratio estimators (e.g., DualDICE, MWL), FORE never requires optimizing over critic classes; there is no saddle point, only a sequence of single-level KL projections.

Empirical Results

The paper presents controlled experiments highlighting the theoretical claims:

  • Baird-Style Finite MRP: In a classical scenario where value-class realizability holds but Bellman completeness fails, standard linear FQE diverges. By contrast, FORE (and FORE-reweighted FQE) converge robustly (Figure 1). Figure 1

    Figure 1: Baird-style finite MRP—policy-value errors and L1(ν)L^1(\nu) occupancy-ratio errors for various methods. FORE and FORE-reweighted FQE avoid the divergence exhibited by standard FQE.

  • Linear-Gaussian Policy Evaluation: Under a continuous-state MDP with a value-class that is not Bellman-complete, FORE exhibits geometric contraction and decreasing RMSE with sample size, while linear FQE errors do not vanish and can even diverge (Figures 2 and 3). Figure 2

    Figure 2: Population recursion in linear-Gaussian MDP—FORE stably contracts in both policy-value error and occupancy-ratio error compared to FQE.

    Figure 3

    Figure 3: RMSE of value estimates across sample size, with FORE and FORE-reweighted FQE exhibiting favorable convergence compared to FQE.

  • Analysis Across Horizon and Estimation Approaches: Comparing direct reward reweighting, MWL, DualDICE, and FORE shows that FORE leads to the smallest density-ratio error. The gap widens as effective horizon grows (Figures 4 and 5). Figure 4

    Figure 4: Side-by-side: Value RMSE from direct reward reweighting and L2(ν)L^2(\nu) error of the fitted density ratio for different estimators in the linear-Gaussian setting.

    Figure 5

    Figure 5: Value RMSE versus effective horizon (1−γ)−1(1-\gamma)^{-1}, highlighting favorable scaling for FORE-based estimators.

Implications and Limitations

FORE establishes that the central challenge in off-policy evaluation lies in the realizability of the discounted occupancy ratio. If this assumption is met within the chosen log-ratio class, stable and convergent estimation is possible even with nonlinear function approximation and without Bellman completeness, offering a more robust basis for offline RL.

Practical implications include:

  • Robust OPE in non-tabular settings where closure assumptions for FQE cannot be ensured.
  • Flexible value estimation pipelines, including direct, doubly robust, and occupancy-weighted approaches.
  • Avoidance of dual/critic/adjoint completeness and the complexities of minimax optimization.

Limitations are inherent: FORE requires standard support overlap between the offline distribution and the target occupancy; if this fails, estimation is statistically intractable. Moreover, in high-dimensional regimes or with nearly deterministic target policies, occupancy-ratio modeling may be as hard as value-function learning. The statistical and approximation errors remain governed by the expressiveness and complexity of the log-ratio class.

Future Directions

Potential directions stemming from this framework include:

  • Adaptive Model Selection: Customizing the log-ratio class size (e.g., via regularization or complexity penalization) to balance bias-variance tradeoffs in practice.
  • Extension to Average-Reward/Undiscounted Regimes: Generalizing the contraction arguments to accommodate average-reward MDPs, possibly leveraging alternative strong data processing inequalities.
  • Integration with Policy Optimization: Using FORE-estimated ratios for constraint satisfaction or improved trust region methods in batch RL and imitation learning.
  • Scalable Implementations and Benchmarking: More extensive empirical evaluation in high-dimensional or large-scale real-world RL benchmarks.

Conclusion

Fitted Occupancy-Ratio Evaluation (FORE) resolves a key stability and approximation gap in off-policy evaluation by shifting the focus of realizability and estimation away from Bellman completeness to the learnability of the discounted occupancy ratio itself. By leveraging the KL contraction property of the adjoint Bellman operator and simple supervised-learning methods, FORE delivers robust OPE guarantees under minimal structural assumptions, opening the door for more reliable and practical batch RL in broad settings (2607.05375).

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

What is this paper about?

This paper is about a safe way to judge how good a new decision-making rule (a “policy”) would be, using only past data, without running the policy in the real world. This task is called offline policy evaluation in reinforcement learning.

There’s a big problem in offline evaluation: the past data comes from how someone acted before (the “logged” or “behavior” policy), but we want to know how a different, target policy would do. The authors introduce a new method, called FORE (Fitted Occupancy-Ratio Evaluation), that focuses on estimating a “ratio” that corrects this mismatch so we can fairly compare.

What questions are the authors asking?

In simple terms, they ask:

  • How can we reweight the past data so it looks like it came from the policy we actually care about?
  • Can we do this reweighting in a stable, reliable way that doesn’t rely on strong, hard-to-satisfy assumptions?
  • Will this approach give good theoretical guarantees and work with common learning tools?

Key ideas explained simply

To follow the paper, it helps to understand three ideas:

  • Occupancy ratio: Think of the states and actions in a game (or process) as places you can be and moves you can make. The “occupancy” tells us how often you visit each place–move pair when following a policy, giving more credit to visits soon and less to far-future ones (this is controlled by a discount number γ between 0 and 1). The occupancy ratio compares “how often under the target policy” to “how often in the logged data.” If the ratio is 2 at a certain place–move, we should count that data point twice; if it’s 0.5, count it half as much. Using this ratio lets us estimate what would happen under the target policy by reweighting the logged data.
  • KL divergence (a measure of difference): This is a way to measure how different two “weighting rules” are. Think of it like: how surprised would you be if you expected one pattern but saw another?
  • Bellman updates (step-by-step reasoning): In reinforcement learning, Bellman equations describe how things relate from one step to the next. This paper uses a special “adjoint Bellman” update tailored for occupancy ratios. You can think of it as a rule for improving your guess of the ratio by looking one step ahead.

What does FORE do and how does it work?

FORE is a fitted, repeat-until-good method to learn the occupancy ratio:

  • Start with a simple guess for the ratio (like “1 everywhere,” meaning no reweighting).
  • Repeat the following:
    • Use the “adjoint Bellman” step to say what the ratio should look like after one step.
    • Because we only have samples (not the full world), project this suggestion back into a family of easy-to-learn ratio functions by solving a standard supervised learning problem. The projection uses KL divergence, so we’re always moving closer in the same “distance” the Bellman step naturally shrinks.
    • Update the ratio and go again.

Important features:

  • Single-level optimization at each step: Each FORE update is like solving a regular machine-learning problem (e.g., using neural networks or gradient-boosted trees), not a complicated saddle-point or min–max game.
  • No extra “critic” needed: Many past methods needed a second helper function (a critic) and strong assumptions to work. FORE avoids this.
  • Stability from the right geometry: The adjoint Bellman step naturally shrinks KL divergence by a factor of γ (the discount). FORE’s projection uses the same measure (KL), so the whole process stays stable and converges.

Everyday analogy: Imagine you’re trying to match a new music playlist (target policy) using a library of old songs (logged data). The occupancy ratio is like a “play count multiplier” that tells you how much to upvote or downvote each song to mimic the target playlist. FORE updates these multipliers step by step: it looks at how your current multipliers would play out one song ahead and then adjusts them with a simple, consistent rule that keeps you from overcorrecting.

What are the main findings?

The paper offers both conceptual and mathematical results:

  • Convergence without strong assumptions: If the true occupancy ratio can be represented (or well approximated) by the kind of functions you’re learning with (e.g., a neural net class), then FORE will converge toward the true ratio. Crucially, you do not need “Bellman completeness” (a common and restrictive assumption in other methods that says the function class must be closed under Bellman updates).
  • A clean error breakdown:
    • Iteration error: Shrinks quickly with each update (roughly by γ each step).
    • Approximation error: Depends on how well your chosen function class can represent the true ratio. If your class can represent it exactly, this term disappears.
    • Statistical error: Depends on how much data you have and how complex your function class is (more data and simpler models make this smaller).
  • Works with standard learners: Each step can be implemented with ordinary supervised learning tools, making the method practical.
  • Useful downstream tools:
    • Direct value estimation: Reweight observed rewards by the learned ratio to estimate how good the target policy is.
    • Doubly robust estimation: Combine the learned ratio with a learned value function (Q-function) for extra safety—if either is good, you get a good estimate.
    • Better fitted Q-evaluation: Use the learned ratio as weights to make value-function learning more stable, without needing Bellman completeness.

Why this matters: Earlier methods often needed extra helper networks (critics) and/or strong assumptions that are hard to ensure in practice. FORE shows you can get stability and guarantees by focusing on the right object (the occupancy ratio) and matching the update and projection measures (both in KL), which naturally contract.

What is the impact of this research?

  • More reliable offline evaluation: FORE gives researchers and practitioners a principled, stable way to evaluate new policies using only logged data—important in safety-critical areas like healthcare, education, or recommendation systems.
  • Fewer restrictive assumptions: By avoiding tough conditions like Bellman completeness or rich critics, FORE can be applied more broadly and more simply.
  • Building block for better methods: The learned ratio can directly estimate values, power doubly robust methods, and stabilize value-function learning. This can make the whole offline reinforcement learning toolkit more robust and easier to use.
  • Practical and flexible: Since each step reduces to a standard supervised learning problem, FORE can plug into existing ML workflows and benefit from modern models and optimization tools.

In short, FORE shows that if you focus on the right correction (the occupancy ratio) and update it in the right “distance” (KL divergence), you can get a simple, practical, and provably stable method for offline policy evaluation—without leaning on heavy assumptions.

View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

The paper establishes a fitted fixed-point framework (FORE) for discounted occupancy-ratio estimation with strong population and finite-sample guarantees under specific assumptions. The following open issues remain unaddressed or only partially explored:

  • Availability and estimation of initial-state moments: The method requires estimating the initial expectation term (via P^0h\widehat P_0 h). How to proceed when no samples from d0d_0 are available and the state marginal of ν\nu does not match μ0\mu_0? Can one consistently recover (1−γ)Ed0h(1-\gamma)E_{d_0}h from transition data alone (e.g., using backward recursion or auxiliary ratio models) and how does this affect guarantees?
  • Dependence on strong coverage assumptions: Conditions (C1) and (C4) require absolute continuity of d0d_0 and νPπ\nu P_\pi with respect to ν\nu and bounded density ratios (∥dd0/dν∥∞≤B0\|dd_0/d\nu\|_\infty\le B_0, ∥d(νPπ)/dν∥∞≤B+\|d(\nu P_\pi)/d\nu\|_\infty\le B_+). What is the behavior of FORE under weak overlap or when these bounds are large or unknown? Develop theory and algorithms (e.g., clipping, regularization, pessimistic penalties) that remain stable with partial or weak coverage.
  • Positivity requirement on the target ratio: Condition (C5) imposes ωπ≥m⋆>0\omega^\pi \ge m_\star > 0 ν\nu-a.e. This excludes target policies whose occupancy has zeros on the support of ν\nu. How to relax this (e.g., via support restriction, barrier smoothing, or truncation) while preserving convergence and rates?
  • Assumptions on the hypothesis class: The analysis requires H\mathcal H to be convex, closed, totally bounded in L2(ν)L^2(\nu), and uniformly bounded (centered sup-norm ≤R\le R). These properties fail for common deep network classes. Extend guarantees to nonconvex, unbounded, or scale-sensitive classes (e.g., via margin-based or norm-based capacity control, self-concordant bounds, or Lipschitz/covering-number conditions tailored to neural networks).
  • Optimization error and inexact ERM: Each step assumes exact minimization in H\mathcal H. Practical training uses SGD/Adam with finite steps and nonconvex objectives. Develop contraction and error bounds that explicitly account for optimization error per iteration and quantify its accumulation.
  • Empirical normalization mismatch: The estimator normalizes ehe^{h} using the empirical log-partition under the offline sample. Analyze the bias introduced by empirical normalization relative to ν\nu, its impact on value-function estimation, and whether cross-fitting or alternative normalization strategies reduce this mismatch.
  • Additional stochasticity from target action sampling: For continuous actions the method approximates (πh)(S′)(\pi h)(S') via Monte Carlo. Provide finite-sample analyses that account for this extra randomness and guide the choice of the number of target-action draws per transition.
  • Data dependence and trajectory sampling: The theory assumes i.i.d. (Xi,Si′)(X_i, S_i') with Xi∼νX_i\sim\nu. Offline RL datasets commonly comprise dependent trajectories. Extend concentration and convergence results to β\beta-mixing or Markov-dependent samples and episodic resets.
  • Complexity of multiplier class with modern function approximators: The finite-sample term relies on local Rademacher complexities of HΔ\mathcal H_\Delta and the product class G×\mathcal G_\times. Derive actionable complexity bounds for practical architectures (e.g., path-norm, spectral norm, NTK, or margin-based complexity for deep nets) to yield concrete rates.
  • Numerical stability of exponential parameterization: The exp⁡(h)\exp(h) parameterization can produce extreme weights and numerical instability. Develop stabilized objectives (e.g., clipped exponentials, temperature scaling, Fenchel–Young alternatives) and analyze their effect on contraction and approximation properties.
  • Sensitivity to large weights and variance control: Even with bounded hh, variance can be high in regions of weak overlap. Analyze and design robust procedures (e.g., weight clipping, soft-normalization, variance penalization) and quantify bias–variance trade-offs while retaining the fixed-point contraction.
  • Extension to undiscounted average-reward settings (γ=1\gamma=1): The main theory treats γ<1\gamma<1; an appendix is mentioned for “mixed contraction.” Provide a full development for average-reward MDPs, including uniqueness, contraction/stability properties, and finite-sample rates.
  • Beyond KL: The adjoint Bellman operator contracts any ff-divergence, but the empirical projection is developed only for KL with exponential families. Construct tractable projections for other divergences (e.g., χ2\chi^2, reverse-KL) and study whether they offer improved finite-sample performance or robustness.
  • Minimax optimality and horizon dependence: The statistical term scales as (1−γ)−2(1-\gamma)^{-2} in generalized KL, translating to (1−γ)−1(1-\gamma)^{-1} at the value scale; determine whether this is minimax optimal for ratio estimation and OPE under FORE-like procedures, and establish lower bounds.
  • Statistical inference for values: The paper provides high-probability error bounds but no inference tools. Develop asymptotic normality, valid confidence intervals, and bootstrap methods for VπV_\pi using FORE (alone or doubly robust), including the impact of estimated weights and potential cross-fitting.
  • Adaptive model selection and hyperparameter tuning: Provide data-driven selection of H\mathcal H (or regularization) to balance εKL\varepsilon_{\rm KL} and statistical error (e.g., oracle inequalities, structural risk minimization, aggregation) and analyze their effect on the fitted recursion.
  • Sample reuse and overfitting: FORE, DR value estimation, and FORE-weighted FQE may reuse the same data for fitting and evaluation. Assess overfitting bias and develop cross-fitting or sample-splitting strategies tailored to iterative fixed-point procedures, with accompanying guarantees.
  • Robustness to heavy-tailed or heteroskedastic rewards: DR estimation uses observed YY. Analyze robustness and propose heavy-tail–resistant variants (e.g., Huberization, Catoni losses) and their interplay with FORE-induced weights.
  • Multi-step variance reduction: Investigate whether using multi-step lookahead or control variates (beyond single-step X+X^+) reduces variance in the KL loss and how this affects contraction and finite-sample guarantees.
  • Practical benchmarking and ablations: The paper lacks empirical evaluation. Systematically compare FORE to DualDICE/GenDICE, FQE (with/without occupancy weighting), and weighted TD across coverage regimes, horizons, and function classes to assess stability, variance, and accuracy.
  • Policy optimization and control: Explore using FORE within policy improvement (e.g., distribution matching or occupancy-regularized objectives) without auxiliary critics, and analyze convergence and stability in policy iteration.
  • Extensions beyond fully observed MDPs: Generalize the adjoint Bellman and FORE to POMDPs, continuous-time settings, or constrained RL, and study identifiability and contraction properties in those settings.
  • Handling confounding in offline data: FORE assumes unconfounded MDP dynamics. In observational settings with unobserved confounders, investigate sensitivity analyses and robustification strategies for occupancy-ratio estimation and OPE.
  • FORE-weighted FQE theory with estimated weights: The paper claims separation of ratio error, value-function approximation error, and finite-iteration error but does not fully detail the finite-sample analysis for weighted FQE using estimated ω\omega. Provide complete bounds that account for weight estimation error, nonconvex value classes, and optimization error, and delineate when weighted FQE avoids Bellman completeness in practice.
  • Estimating Ed0hE_{d_0}h from transitions when μ0\mu_0 is unknown: Beyond the two special cases given (direct d0d_0 sampling or νS=μ0\nu_S=\mu_0), propose and analyze estimators of (1−γ)Ed0h(1-\gamma)E_{d_0}h using only transition data (e.g., backward ratios or learned initial-state models) and quantify their effect on FORE’s rates.
View Paper Prompt  View All Prompts 

Practical Applications

Overview

The paper introduces FORE (Fitted Occupancy-Ratio Evaluation), a practical, fitted fixed-point method for estimating discounted occupancy ratios in offline reinforcement learning (RL). FORE uses one-step transitions and an adjoint Bellman recursion with KL-projected updates to estimate how a target policy shifts the state–action distribution relative to logged data. It:

  • Requires only realizability (or approximation) of the true discounted occupancy ratio in a chosen log-ratio function class, not Bellman completeness.
  • Works with standard supervised learners (e.g., gradient-boosted trees, neural networks).
  • Produces a ratio usable for:
    • Direct value estimation by reward reweighting.
    • Doubly robust (DR) estimators when combined with a fitted Q-function.
    • Stabilized, occupancy-weighted fitted Q-evaluation (FQE) without value-function Bellman completeness.

Below are practical, real-world applications grouped by immediacy, with sector links, tools/workflows, and feasibility dependencies.

Immediate Applications

These can be deployed with current logged-transition datasets, standard function approximators, and basic MLOps.

  • Offline A/B gating and policy rollout decisions
    • Sectors: software platforms, recommender systems, search, advertising, marketplaces, fintech apps, edtech.
    • Use: Evaluate the impact of a new policy (ranking, pricing, bidding, tutoring strategy) on key rewards (CTR, revenue, engagement) offline, before any or minimal online exposure.
    • How: Fit FORE on logged one-step transitions; compute Eν[ω(X)r(X)] for reward r or use the DR estimator with a fitted Q. Use bounds to set decision thresholds.
    • Tools/workflows: Python/R library implementing Algorithm 1; integration into offline evaluation services; bootstrap for CIs; policy registry to sample A+ from the target policy.
    • Assumptions/dependencies:
    • Logged (S, A, S′) transitions and access to the target policy π to sample actions A+ at S′ or compute policy averages if actions are discrete.
    • Coverage: d0 and one-step target-policy distribution under π must be absolutely continuous wrt data distribution ν (bounded density ratios).
    • Ratio realizability: log ωπ,γ is approximable by the chosen function class (e.g., GBDT/NN).
    • Discount γ < 1 and stable environment/MDP assumptions over logged data.
  • Safer offline evaluation in healthcare from EHR logs
    • Sectors: healthcare.
    • Use: Estimate expected outcomes for new treatment or triage policies using historical patient trajectories without requiring behavior policy propensities.
    • How: Fit FORE on (state, action, next-state) extracted from EHR; estimate discounted outcome value; optionally combine with a clinical Q-estimator via DR for robustness.
    • Tools/workflows: Clinical feature engineering; causal validation/diagnostics; conservative deployment thresholds.
    • Assumptions/dependencies:
    • Approximate Markov transitions over chosen state representation; sufficient policy support/coverage; careful treatment of confounding and nonstationarity.
  • Offline evaluation of robotics and autonomous systems policies
    • Sectors: robotics, autonomous driving, industrial automation, warehousing.
    • Use: Pre-deployment evaluation of policies using logs or simulators to avoid risky online tests.
    • How: Fit FORE on offline transitions (sim or real logs); compute value and safety constraint functionals (e.g., collision penalties) using Eν[ω(X)g(X)].
    • Tools/workflows: Dataset curation pipeline; simulators to augment transitions; DR with Q estimated via occupancy-weighted FQE for stability.
    • Assumptions/dependencies:
    • MDP fidelity of simulator/logs; adequate overlap between logged data and target trajectories; stability of learned representations.
  • Advertising and bidding strategy evaluation from auction logs
    • Sectors: digital advertising, real-time bidding (RTB), marketing tech.
    • Use: Offline assessment of bidding or budget-allocation policies under auction dynamics.
    • How: Apply FORE on auction-transition logs to correct for policy-induced occupancy shift; estimate expected ROI/conversions.
    • Tools/workflows: Integration with experimentation platforms; post-estimation DR for lower variance.
    • Assumptions/dependencies:
    • Stationarity over evaluation window; availability of one-step transitions with bid outcomes; coverage for relevant price/action space.
  • Occupancy-weighted FQE to stabilize value learning
    • Sectors: any domain using offline RL (recsys, ops, robotics).
    • Use: Improve stability and validity of fitted Q-evaluation by projecting Bellman regressions with FORE-estimated weights (estimated target-occupancy norm), avoiding Bellman completeness on the value class.
    • How: Two-stage workflow: (1) Fit ω via FORE. (2) Run FQE with weighted regression using ω as fixed weights. Use DR for final value.
    • Tools/workflows: Unified OPE pipeline; cross-fitting for ω/Q; early stopping; model selection via validation metrics.
    • Assumptions/dependencies:
    • FORE assumptions above; for FQE, consistent function class for Q and adequate coverage under ω; careful handling if ω has high variance.
  • Compliance, safety, and auditing for policy deployment
    • Sectors: regulated industries (healthcare, finance), safety-critical systems.
    • Use: Provide auditable, distribution-shift-corrected value estimates with finite-sample error controls to support risk assessments.
    • How: Package FORE with diagnostics (coverage checks, weight distribution, bootstrap intervals), sensitivity analysis on γ and function-class choices.
    • Tools/workflows: Reporting dashboards; automated data/coverage checks; thresholds for deployment gating.
    • Assumptions/dependencies:
    • Documentation of data lineage; stability of logged environment; sufficiency of coverage; interpretability of function approximators if required.
  • Estimating non-reward target-occupancy functionals
    • Sectors: fairness/safety monitoring, operations, education, content policy.
    • Use: Estimate probability of visiting risky/undesirable states or fairness metrics under a target policy using Eν[ω(X)g(X)].
    • How: Fit ω with FORE; compute occupancy-based metrics (e.g., visit rates to sensitive states, constraint violation frequencies).
    • Tools/workflows: Metric library for g functions (constraints/fairness targets); CI via bootstrap.
    • Assumptions/dependencies:
    • Clear definition of g; coverage of states where g is relevant; stable logging and consistent reward/cost semantics.
  • Contextual bandit special case (γ=0)
    • Sectors: recommendations, ads, news feeds, promos.
    • Use: FORE collapses to density-ratio estimation for one-step off-policy evaluation without behavior propensities; can replace IPS in propensities-sparse logs.
    • How: Set γ=0; solve single convex objective per iteration (trivial contraction); use ω for weighted averages of immediate rewards.
    • Tools/workflows: Existing bandit OPE pipelines; low-variance plug-in estimators.
    • Assumptions/dependencies:
    • One-step independence assumptions; target-policy actions supported by logged actions.

Long-Term Applications

These require further research, scaling, or integration beyond current practice.

  • End-to-end off-policy RL with FORE-guided actor–critic
    • Sectors: large-scale personalization, robotics, operations research.
    • Vision: Use FORE to supply occupancy corrections to stabilize critic learning and to inform policy gradients in offline-to-online improvement loops.
    • Needed advances:
    • Theoretical/empirical study of combining FORE with policy optimization; controlling variance and bias in iterative policy improvement.
    • Adaptive updates of ω as π changes, with warm-starts and incremental training.
  • Robust policy optimization under constraints and risk
    • Sectors: finance, healthcare, safety-critical operations.
    • Vision: Use FORE to estimate occupancy of constraint-violating states, CVaR-type objectives, or risk-sensitive costs; optimize policies with explicit safety/fairness constraints.
    • Needed advances:
    • Tight high-confidence bounds and concentration for tail/risk objectives; safe optimization with learned ratios; principled treatment of coverage gaps.
  • Nonstationary and partially observed environments
    • Sectors: ad markets, healthcare with evolving practices, dynamic supply chains.
    • Vision: Extend FORE to settings with time-varying dynamics or POMDPs (e.g., via representation learning and time-indexed ω).
    • Needed advances:
    • Methods to handle covariate shift over time, reweighting across episodes; identification under partial observability.
  • Scaling to high-dimensional perception and control
    • Sectors: vision-based robotics, autonomous driving, AR/VR systems.
    • Vision: Use representation learning (contrastive, self-supervised) to improve log-ratio realizability; couple with large neural function classes and distributed training.
    • Needed advances:
    • Practical training heuristics for stability; regularization for ω to avoid weight spikes; on-device evaluation pipelines.
  • Dataset design and adaptive logging for coverage
    • Sectors: platforms with continuous experimentation and logging.
    • Vision: Actively plan logging policies to improve coverage where FORE diagnostics indicate deficiencies, reducing worst-case ratio bounds.
    • Needed advances:
    • Coverage-aware data collection strategies; feedback loops between evaluation diagnostics and logging policies.
  • Formal auditing, certification, and standardization of offline evaluation
    • Sectors: policy/regulatory, standards bodies, enterprise AI governance.
    • Vision: Build standards around KL-based occupancy-ratio evaluation, coverage reporting, and evaluation checklists for offline RL deployment.
    • Needed advances:
    • Reference implementations, benchmarks, and reporting protocols; guidance for acceptable coverage thresholds and error bars.
  • Confidence intervals and uncertainty quantification tailored to FORE
    • Sectors: any with decision accountability.
    • Vision: Develop asymptotic/bootstrapped CIs that respect the iterative fixed-point nature and multiplier complexities in the estimator.
    • Needed advances:
    • Theory for valid post-fitting inference under function approximation; scalable resampling strategies and sensitivity analyses.
  • Extensions to undiscounted or average-reward criteria
    • Sectors: operations/logistics, continuing control.
    • Vision: Apply mixed-contraction variants and stationarity corrections for average-reward settings.
    • Needed advances:
    • Stable implementations and guarantees for γ→1 regimes; handling of long horizons and bias–variance trade-offs.
  • Multi-agent and strategic environments
    • Sectors: marketplaces, auctions, multi-robot systems.
    • Vision: Generalize FORE to estimate occupancy corrections under interacting policies and game dynamics.
    • Needed advances:
    • Adjoint operators and KL projections for multi-agent transitions; identification under strategic behavior and sparse support.

Cross-cutting assumptions and dependencies affecting feasibility

  • Coverage: The initial distribution d0 and one-step target-policy distribution under π must have bounded densities relative to the logging distribution ν; lack of support leads to unavoidable bias or instability.
  • Ratio realizability: The discounted occupancy ratio’s log must be in (or well-approximated by) the chosen function class; representation learning may be required in high dimensions.
  • Logged data quality: Availability of reliable one-step transitions (S, A, S′) and rewards; consistent environment dynamics across logging and target policy evaluation.
  • Target policy access: Ability to compute or sample π(a|s′) for A+ generation or to evaluate policy-averaged h when actions are discrete.
  • Discount factor: γ < 1 for discounted settings; separate treatment needed for average-reward/undiscounted problems.
  • Training stability: While each FORE update is a convex objective in linear classes, neural parameterizations are nonconvex; practical success may require regularization, early stopping, and variance control.
  • Variance control and diagnostics: Monitor weight distributions (ω tails), effective sample sizes, and coverage diagnostics; consider truncation/smoothing if needed with awareness of induced bias.

Potential tools and products that might emerge

  • FOREEstimator library: A Python package exposing:
    • fit_ratio(transitions, policy, gamma, class), predict(x), evaluate(functionals).
    • DoublyRobustValue(omega, Q) and OccupancyWeightedFQE modules.
  • OPE pipeline integration:
    • Data checks (coverage, support, logging quality), ratio fitting, DR estimation, CI via bootstrap, deployment gating dashboards.
  • Platform plugins:
    • Extensions for RLlib, ACME, or internal RL stacks to add FORE-based OPE and weighted FQE.
  • Diagnostics suite:
    • Visualizations for ω distribution, overlap metrics, sensitivity analyses (function-class choices, γ), and stability monitors during training.
View Paper Prompt  View All Prompts 

Glossary

  • adjoint Bellman operator: The linear operator mapping a candidate ratio to a new ratio by propagating it one step under the target policy and mixing with the initial distribution; it characterizes the discounted occupancy ratio as a fixed point. "define the adjoint Bellman operator \citep{ueharaEtAl2021FiniteSampleMinimax}"
  • adjoint Bellman recursion: The iterative application of the adjoint Bellman operator whose fixed point equals the discounted occupancy ratio. "a fitted fixed-point method that characterizes the discounted occupancy ratio through an adjoint Bellman recursion."
  • absolute continuity: A coverage condition requiring that one measure does not assign positive mass where another assigns zero, enabling density ratios to exist. "The measures d0d_0 and areabsolutelycontinuouswithrespectto are absolutely continuous with respect to \nu$.&quot;</li> <li><strong>Bellman completeness</strong>: An assumption that the image of the value class under the Bellman operator remains in the class, restoring contraction for projected iterations. &quot;This condition is called Bellman completeness, a key assumption in analyses of FQE&quot;</li> <li><strong>Bellman residual</strong>: The discrepancy between a candidate value function and its Bellman update, used in error analysis and doubly robust corrections. &quot;the standard doubly robust Bellman-residual correction&quot;</li> <li><strong>concentrability</strong>: A condition bounding how much the target-policy distributions can differ from the offline data distribution. &quot;together with coverage or concentrability conditions relating target-policy distributions to the offline data&quot;</li> <li><strong>contraction (KL-contraction)</strong>: A property that an operator brings distributions closer in a divergence metric by a factor less than one. &quot;the adjoint Bellman operator being a KL-contraction.&quot;</li> <li><strong>critic class</strong>: A set of test functions used to detect and penalize violations of moment or balance conditions in minimax estimators. &quot;require the critic to detect the relevant residuals, through conditions such as critic richness&quot;</li> <li><strong>data processing inequality</strong>: An information-theoretic principle stating that applying a Markov kernel cannot increase divergence between distributions. &quot;By joint convexity of KL and the data processing inequality for the Markov kernel $P_$&quot;</li> <li><strong>doubly robust estimator</strong>: An estimator combining a model for the outcome (e.g., Q-function) and for the distribution correction (ratio), consistent if either is correct. &quot;a doubly robust estimator with a product-form error bound&quot;</li> <li><strong>dual realizability</strong>: A condition ensuring the auxiliary (dual) function class is rich enough to capture residuals in saddle-point formulations. &quot;require critic richness, dual realizability, or completeness conditions&quot;</li> <li><strong>exponential family (normalized)</strong>: A family of densities parameterized by log-ratios with a normalizing log-partition term, used here for ratio modeling. &quot;Since $W$ is a normalized exponential family with convex natural-parameter space&quot;</li> <li><strong>f-divergence</strong>: A broad class of divergences between probability measures defined via convex functions; includes KL as a special case. &quot;the adjoint Bellman map is a contraction with respect to any $f$-divergence.&quot;</li> <li><strong>fitted iteration</strong>: An algorithmic paradigm that alternates between computing targets via a dynamic-programming operator and projecting them back onto a function class. &quot;fitted $Q$-evaluation (FQE) is the canonical fitted-iteration approach&quot;</li> <li><strong>fixed point</strong>: A point that remains unchanged under an operator; here, the occupancy ratio solves $\omega=B\omega$. "the occupancy ratio is characterized by the fixed-point equation"
  • generalized KL divergence: An extension of KL that allows comparing non-normalized functions, used when empirical normalizations differ. "we measure its error using the generalized KL divergence"
  • inherent Bellman error: The approximation error introduced when the Bellman image cannot be represented within the chosen function class. "small inherent Bellman error"
  • information projection (I-projection): Projection of a distribution onto a model class by minimizing KL divergence, enjoying a Pythagorean property. "the Pythagorean inequality for information projections"
  • importance weighting: Reweighting samples to correct for distribution shift between behavior and target distributions. "importance ratios, while doubly robust estimators combine importance weighting with value-function estimates"
  • Kullback–Leibler (KL) divergence: A measure of dissimilarity between probability distributions, equal to relative entropy. "projecting the adjoint Bellman image onto a log-ratio class in Kullback--Leibler (KL) divergence."
  • local Rademacher complexity: A data-dependent measure of function class richness around the target, governing statistical error rates. "a statistical error governed by local Rademacher critical radii for the log-ratio class"
  • log-partition function: The log of the normalizing constant ensuring an exponential-family density integrates to one. "and the normalizing log-partition function"
  • log-ratio class: A hypothesis class parameterizing density ratios via exponentiated log functions that are normalized under the offline distribution. "thereby projecting the adjoint Bellman image onto a log-ratio class"
  • marginalized importance sampling: Importance sampling that reweights marginal occupancies rather than products of stepwise likelihood ratios. "Marginalized importance sampling avoids products of trajectory ratios by correcting marginal state or state--action occupancies"
  • Markov kernel: A conditional transition distribution mapping current state–action pairs to next state–action distributions. "For an action distribution η\eta, let PηP_\eta denote the induced state--action transition kernel"
  • minimax estimation: Optimizing a model against the worst-case critic over a function class to enforce balance or moment conditions. "primal-dual and minimax methods that seek a ratio function minimizing empirical imbalance over a class of critic functions"
  • multiplier class: A function class formed by products of learned weights and test functions, appearing in empirical process bounds. "and the induced multiplier class"
  • occupancy Bellman moment identity: A set of balance equations relating expectations under the offline distribution of functions and their one-step rollouts under the target policy. "this becomes the occupancy Bellman moment identity"
  • occupancy measure: The distribution over state–action pairs induced by a policy over time, possibly discounted. "the discounted occupancy measure satisfies the Bellman equation"
  • occupancy ratio: The density ratio between the target policy’s (discounted) occupancy measure and the offline data distribution. "The discounted occupancy ratio is the density of the target policy's discounted state--action occupancy measure relative to the offline data distribution."
  • occupancy-weighted FQE: A version of fitted Q-evaluation that projects in the target-occupancy norm using learned occupancy ratios. "occupancy-weighted fitted QQ-evaluation"
  • Pythagorean inequality (for KL): A decomposition property whereby the KL divergence to the truth splits into the sum of divergences to and within the model class. "the Pythagorean inequality for information projections"
  • Radon–Nikodym derivative: The density relating two measures when one is absolutely continuous with respect to the other. "Taking Radon--Nikodym derivatives in \eqref{eqn::occbell} gives"
  • Rademacher critical radius: The smallest radius where local Rademacher complexity is dominated by the squared radius, controlling ERM rates. "a statistical error governed by local Rademacher critical radii for the log-ratio class"
  • realizability: The assumption that the true target function lies within the chosen hypothesis class. "our central approximation condition is just realizability of the discounted occupancy ratio itself."
  • relative entropy: Another name for KL divergence, quantifying discrepancy between distributions. "contracts in relative entropy toward the true ratio"
  • Riesz representer: A function representing a linear functional in a Hilbert space, used to interpret policy-value functionals. "as minimax estimators of the Riesz representer of the policy-value functional"
  • saddle-point formulation: A min–max optimization structure used to enforce moment or stationarity conditions by adversarial critics. "through saddle-point, minimax, or temporal-difference objectives"
  • semiparametric efficiency: Statistical theory identifying nuisance functions needed to attain optimal estimation rates without fully parametric models. "Semiparametric theory likewise identifies the occupancy ratio, together with a value function, as the pair of nuisance functions underlying efficient and doubly robust off-policy evaluation"
  • stationary-distribution correction: Adjusting learning or evaluation procedures to account for differences between behavior and target stationary distributions. "stationary-distribution correction"
  • successor representation: A representation that predicts expected discounted future state occupancies, useful for density-ratio and policy optimization. "Other occupancy-correction methods use successor representations"
  • Tikhonov penalty: An L2 (ridge) regularization term added to stabilize optimization in function approximation. "Regularization, such as a Tikhonov penalty, can be added to the same objective."
  • value-function realizability: The assumption that the true value function can be perfectly represented within the chosen function class. "value-function realizability alone does not ensure stable FQE"
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Tweets

Sign up for free to view the 2 tweets with 67 likes about this paper.

Sign Up for Free