Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching

Abstract: We present a novel theoretical framework, Q-MMR, for off-policy evaluation in finite-horizon MDPs. Q-MMR learns a set of scalar weights, one for each data point, such that the reweighted rewards approximate the expected return under the target policy. The weights are learned inductively in a top-down manner via a moment matching objective against a value-function discriminator class. Notably, and perhaps surprisingly, a data-dependent finite-sample guarantee for general function approximation can be established under only the realizability of $Q^π$, with a dimension-free bound -- that is, the error does not depend on the statistical complexity of the function class. We also establish connections to several existing methods, such as importance sampling and linear FQE. Further theoretical analyses shed new light on the nature of coverage, a concept of fundamental importance to offline RL.

• The paper introduces Q-MMR, a recursive reweighting algorithm for off-policy evaluation that avoids explicit function class complexity penalties.
• It provides dimension-free finite-sample error guarantees and unifies classical estimators such as Monte Carlo and importance sampling.
• The method enables transparent uncertainty quantification, facilitating practical model selection and robust offline RL evaluation.

Q-MMR: Off-Policy Evaluation via Recursive Reweighting and Moment Matching

Problem Context and Motivation

Off-policy evaluation (OPE) in reinforcement learning seeks to estimate the expected return of a target policy using data generated by a different behavior policy. This problem is central in offline RL, especially when function approximation is required to handle large or continuous state-action spaces. Classical algorithms such as Fitted-Q Evaluation (FQE) and temporal-difference methods provide solutions, but standard guarantees based on general function approximation incur explicit dependence on the complexity (e.g., $\log|F|$) of the function class, which may not be tight in all regimes. Moreover, the disconnect between general and specialized analyses (notably, dimension-free guarantees in linear/tabular cases) has raised the question of whether these complexity dependencies are inherent or artifacts of analysis or algorithmic structure.

Algorithmic Framework: Q-MMR

Q-MMR (Q-function-based Moment-Matched Reweighting) introduces a top-down, recursive algorithm for OPE in finite-horizon MDPs. At each step $h$, Q-MMR computes non-parametric scalar weights $w_h^{(i)}$ for each trajectory in the dataset, minimizing a moment-matching objective based on an IPM over a chosen value function class $F_h$. The weights are constructed recursively:

• At $h=0$, initialization uses the fixed initial state, requiring no reweighting ($w_0^{(i)} \equiv 1$).
• For $h=1,\dots,H$, the algorithm selects $w_h^{[n]}$ to minimize the supremum, over $f\in F_h$, of the discrepancy between the $w_h^{(i)}$-weighted empirical mean of $h$0 at $h$1 and the $h$2-weighted empirical mean of $h$3 at $h$4. This enforces moment-matching for the value distribution under reweighted datasets.

This level-wise moment matching avoids parameterizing the importance-ratio directly and does not require additional realizability or Bellman-completeness assumptions beyond the expressivity of $h$5 for $h$6.

Theoretical Results and Guarantees

Dimension-Free Finite-Sample Guarantee

A principal claim of the paper is the establishment of a high-probability, data-dependent finite-sample bound for general function approximation that is explicitly dimension-free in the dominant term:

$h$7

where $h$8 is the empirical matching loss and $h$9 scales with the empirical 2nd moment $w_h^{(i)}$0. Notably, the leading term does not include a statistical complexity penalty depending on function class size or covering numbers; only the higher-order data-independent bound contains such a dependence, and then only in a lower-order term.

Connections to Monte Carlo, FQE, and MIS

Q-MMR encompasses and strictly generalizes several canonical OPE estimators:

• Monte Carlo estimator: If the data is on-policy and $w_h^{(i)}$1, the matching loss vanishes; the empirical estimate coincides with MC, and the dimension-free bound is tight.
• Importance Sampling: A variant of the algorithm with modified targets recovers step-wise importance sampling as the solution that yields zero matching loss, though potentially with high variance.
• FQE/Tabular Model-based Estimation: In the linear or tabular setting, the weights are in closed form, and Q-MMR reduces to linear FQE and, when data coverage is complete, to tabular certainty equivalence. The bound specializes to previously established dimension-free guarantees in the linear case [duan2020minimax; yin2021towards].

Empirical Uncertainty Quantification

Q-MMR’s bound is "what you see is what you get": both empirical moment-matching loss and the 2nd moment of $w_h^{(i)}$2 are computable from the data, enabling transparent uncertainty quantification. This facilitates robust application and model selection in practice without the need for held-out data or double-sampling solutions.

Statistical and Computational Properties

Population and Empirical Tracking

The paper provides a careful treatment of the population limit of Q-MMR under regularity conditions, establishing that the empirical weights $w_h^{(i)}$3 track their population minimizers $w_h^{(i)}$4 (which always lie within the closure of the span of $w_h^{(i)}$5), with deviations captured by complexity/covering number terms only in higher order.

A key result is that the instability (i.e., potential exponential blow-up of weights) is ruled out if either the operator norm of a projected Bellman operator sequence or Bellman completeness is controlled. Thus, the Q-MMR framework tightly characterizes when variance will remain bounded—refining prior notions of coverage.

Theoretical Implications for Coverage

Contrary to traditional intuition (that coverage means $w_h^{(i)}$6 should directly cover $w_h^{(i)}$7 or its features), Q-MMR reveals that the actual object requiring sufficient coverage is, in general, a projected feature evolution dictated by the data distribution and transition operators—not the true forward dynamics [amortila2026unifying]. Only under Bellman completeness do these coincide.

Computational Implementation

Where $w_h^{(i)}$8 is linear, Q-MMR is computationally tractable via least-squares regression; for nonlinear $w_h^{(i)}$9, moment-matching can be efficiently approximated using minimax optimization or no-regret online learning and optimization oracles.

Comparison to Existing Approaches

Q-MMR offers a middle ground between parameterized marginalized importance weighting (MIS) approaches [uehara2019minimax; liu2018breaking] and direct fitted value methods. Its level-wise, top-down approach obviates the need for additional parametric realizability for density ratio function classes, and avoids the double-sampling pitfalls of simultaneously optimizing all weights. Compared to model selection methods which require tournaments over finite candidate sets, Q-MMR provides computationally tractable uncertainty estimation for rich function classes.

Limitations and Directions for Future Research

There are two central limitations identified:

• The analysis requires i.i.d. trajectory data; extending to adaptive or dependent data (e.g., in the infinite-horizon case) is nontrivial due to loss of conditional independence.
• The recursive, level-wise construction of weights is locally greedy and does not consider downstream impacts; simultaneously optimizing all levels brings back double-sampling challenges unless parametric restrictions are imposed.

Future work may focus on extending Q-MMR to non-i.i.d./dependent sampling settings, more global optimization of weight vectors, and richer online RL data collection protocols.

Conclusion

Q-MMR establishes a unified, theoretically robust framework for OPE in finite-horizon MDPs with general function approximation, accompanied by a dimension-free data-dependent error bound. The method generalizes and connects classical OPE estimators, clarifies the true nature of coverage, and enables practical, uncertainty-quantified OPE in both linear and nonlinear settings. These insights have theoretical implications for RL theory and practical consequences for offline RL evaluation and model selection, and motivate further research in extending high-confidence, scalable OPE beyond standard assumptions.

---

References

• [duan2020minimax], [uehara2019minimax], [amortila2026unifying], [yin2021towards], [liu2018breaking], [ernst2005tree], [chen2019information], [le2019batch], etc.