---
title: 'Q-MMR: Moment-Matched Reweighting'
url: https://www.emergentmind.com/topics/q-mmr
type: topic
---

# Q-MMR: Moment-Matched Reweighting

Q-MMR, or **Q-function based Moment-Matched Reweighting**, is a theoretical framework for off-policy evaluation in finite-horizon Markov decision processes. Introduced by Xiang Li and Nan Jiang, it learns one scalar weight for each data point so that reweighted rewards approximate the expected return of a target policy. Its defining mechanism is an inductive, top-down moment-matching procedure against a value-function discriminator class, and its main theoretical result is a data-dependent finite-sample guarantee under only the realizability of \(Q^\pi\), with a dimension-free leading term that does not depend on the statistical complexity of the function class [2605.06474].

## 1. Formal setting and estimation target

Q-MMR is formulated for a finite-horizon MDP with disjoint state layers \(S_0,\ldots,S_H\), action set \(A\), transition kernel \(P(s' \mid s,a)\) supported on the next layer, and reward distribution \(R(r \mid s,a)\) with \(r \in [0,R_{\max}]\). The setup includes a fixed initial dummy pair \((s_0,a_0)\) and reward \(r_0 \equiv 0\). For a target policy \(\pi : S \to \Delta(A)\), trajectories have the form
\[
(s_0,a_0,r_0,s_1,a_1,r_1,\ldots,s_H,a_H,r_H),
\]
with occupancy
\[
d_h^\pi(s,a)=\Pr_\pi[s_h=s,a_h=a],
\]
and return
\[
J(\pi)=E_\pi\!\left[\sum_{h=1}^H r_h\right]\in[0,H\cdot R_{\max}] \, .
\]
The available data are \(n\) i.i.d. trajectories collected under a behavior policy \(\pi_D\), written as
\[
\{(s_h^{(i)},a_h^{(i)},r_h^{(i)},s_{h+1}^{(i)}) : i=1\ldots n,\ h=0\ldots H\},
\]
with empirical marginal \(d_h^D\) and expectation \(E_D[\cdot]\). The off-policy evaluation objective is to estimate \(J(\pi)\) using the behavior data and a function class \(F=\{F_1,\ldots,F_H\}\) under the standard realizability assumption that, for each \(h\), the true action-value function
\[
Q_h^\pi(s,a)=E_\pi\!\left[\sum_{h'=h}^H r_{h'} \mid s_h=s,a_h=a\right]
\]
lies in \(F_h\) [2605.06474].

The estimator used by Q-MMR has the form
\[
\widehat{J}(\pi)=\sum_{h=1}^H \frac{1}{n}\sum_{i=1}^n w_h^{(i)} r_h^{(i)},
\]
where the \(w_h^{(i)}\) are learned scalar weights. The intuitive comparison point is the marginalized importance ratio \(d_h^\pi(s,a)/d_h^D(s,a)\), which would yield unbiased reweighting if known. Q-MMR replaces explicit density-ratio estimation by a recursive empirical moment-matching construction [2605.06474].

## 2. Recursive moment-matched reweighting

For each level \(h\), given the previous weights \(w_{h-1}^{[n]}=\{w_{h-1}^{(i)}\}_{i=1}^n\), Q-MMR defines the empirical matching loss
\[
L_h\bigl(w;w_{h-1}\bigr)
=
\sup_{f\in F_h}
\left|
\frac{1}{n}\sum_{i=1}^n w^{(i)} f(s_h^{(i)},a_h^{(i)})
-
\frac{1}{n}\sum_{i=1}^n w_{h-1}^{(i)} f(s_h^{(i)},\pi)
\right|,
\]
where
\[
f(s,\pi)=\sum_a \pi(a\mid s)\, f(s,a).
\]
Q-MMR chooses \(w_h^{[n]}\) to approximately minimize \(L_h(w;w_{h-1}^{[n]})\), and among minimizers selects the one with smallest \(\ell_2\)-norm \(\|w\|_2\). Because \(F_h\) is convex, for example in the linear case, this is a convex-concave minimax problem in \((w,f)\) [2605.06474].

The recursion is initialized with \(w_0^{(i)}\gets 1\) for all \(i\). For \(h=1,\ldots,H\), the algorithm solves
\[
w_h^{[n]}=\arg\min_{w\in\mathbb{R}^n} L_h(w;w_{h-1}^{[n]})
\]
optionally subject to \(w\ge 0\), and then forms the final estimate \(\widehat J(\pi)\) by summing the reweighted rewards across levels. Although the procedure never explicitly computes \(Q^\pi\), the analysis shows that each matching step controls the Bellman-error term
\[
Q_h^\pi(s_h^{(i)},a_h^{(i)}) - r_h^{(i)} - Q_{h+1}^\pi(s_{h+1}^{(i)},\pi)
\]
through the requirement that, for all \(f\in F_{h+1}\),
\[
\frac{1}{n}\sum_i w_h^{(i)} f(s_{h+1}^{(i)},a_{h+1}^{(i)})
\approx
\frac{1}{n}\sum_i w_{h-1}^{(i)} f(s_{h+1}^{(i)},\pi).
\]
This induces a top-down propagation of Bellman corrections from one level to the next [2605.06474].

A plausible implication is that Q-MMR shifts the main estimation burden from explicit transition or occupancy modeling to a sequence of local compatibility constraints with the discriminator classes \(F_h\). In the terminology of the paper, the method learns weights inductively in a top-down manner via moment matching against a value-function discriminator class [2605.06474].

## 3. Finite-sample theory and dimension-free control

The central finite-sample result states that, with probability at least \(1-\delta\), for any choice of weights \(w_h^{[n]}\) such that each depends only on the first \(h\) levels of data,
\[
\bigl|\widehat J(\pi)-J(\pi)\bigr|
\le
\sum_{h=1}^H L_h(w_h^{[n]};w_{h-1}^{[n]})
+
\sum_{h=0}^H \varepsilon_h,
\]
where
\[
\varepsilon_h
=
V_{\max}\,\|w_h^{[n]}\|_2
\sqrt{\frac{2\ln(2(H+1)/\delta)}{n}}.
\]
The leading term depends on the empirical matching losses \(L_h\), not on the covering number, dimension, or other standard complexity measure of \(F_h\). This is the sense in which the guarantee is dimension-free [2605.06474].

The proof sketch in the paper proceeds by a Bellman-residual telescoping argument. First, \(r_h^{(i)}\) is replaced by the Bellman difference \(Q_h^\pi-Q_{h+1}^\pi\), with the resulting empirical-variance error bounded by \(\varepsilon_h\). Second, the matching loss \(L_{h+1}\) is used to replace \(w_h Q_{h+1}^\pi(s_{h+1},\pi)\) by \(w_{h+1} Q_{h+1}^\pi(s_{h+1},a_{h+1})\). Third, the resulting expression telescopes exactly to \(J(\pi)\). The final error is therefore the sum of the moment-matching losses and the higher-order \(\ell_2\)-norm penalties [2605.06474].

The paper emphasizes three consequences of this result. First, the guarantee requires only realizability of \(Q^\pi\), not full Bellman completeness. Second, the bound is data-dependent: the quantities \(\sum_h L_h + \sum_h \varepsilon_h\) can be computed on hold-out data, yielding what the authors describe as a “WYSIWYG” uncertainty measure. Third, the complexity of the function class enters only through the higher-order norm terms rather than the leading approximation term [2605.06474].

## 4. Relation to importance sampling, FQE, and tabular OPE

Q-MMR is explicitly positioned as a unifying reweighting framework with connections to several established off-policy evaluation methods. In the on-policy case, when \(\pi_D=\pi\) and one chooses \(w_h\equiv 1\), the matching losses satisfy \(L_h=0\), and \(\widehat J\) reduces to the Monte Carlo estimator with a dimension-free Hoeffding bound. For step-wise importance sampling, if the second term of \(L_h\) is modified to use \(\pi(a_h^{(i)}\mid s_h^{(i)})/\pi_D(a_h^{(i)}\mid s_h^{(i)})\), then the choice
\[
w_h=\prod_{h'\le h}\pi/\pi_D
\]
makes \(L_h=0\), but with exponentially large variance. In this comparison, Q-MMR trades variance against exact matching by constraining the moments only through the discriminator classes \(F_h\) [2605.06474].

When \(F_h\) is linear,
\[
F_h=\{\phi(s,a)\cdot \theta : \|\theta\|\le B\},
\]
the paper states that Q-MMR with exact matching and minimum-\(\ell_2\) weights exactly reproduces linear FQE. In the tabular case, Q-MMR coincides with computing empirical occupancy-ratio weights
\[
w_h(i)=\frac{d_h^\pi(s_h^{(i)},a_h^{(i)})}{d_h^D(s_h^{(i)},a_h^{(i)})},
\]
and therefore with certainty-equivalence model-based evaluation [2605.06474].

| Method | Q-MMR specialization | Consequence |
|---|---|---|
| On-policy Monte Carlo | \(\pi_D=\pi\), \(w_h\equiv 1\) | \(L_h=0\), recovers MC |
| Step-wise Importance Sampling | Product ratios \(\prod_{h'\le h}\pi/\pi_D\) | \(L_h=0\), exponentially large variance |
| Linear FQE | Linear \(F_h\), exact matching, min-\(\ell_2\) weights | Exactly reproduces linear FQE |
| Tabular OPE | Empirical occupancy-ratio weights | Coincides with certainty-equivalence model-based evaluation |

This structure suggests that Q-MMR is best understood not as a competitor to a single baseline, but as a reweighting template whose limiting cases recover several familiar estimators. The paper also notes a conceptual link to joint multi-level reweighting methods such as DualDICE and MWL, but frames simultaneous multi-level minimax optimization as difficult in the non-parametric regime because of double-sampling issues [2605.06474].

## 5. Coverage, population weights, and projected feature geometry

A substantial part of the theory concerns the nature of coverage. The population weight functions \(w_h^\star(s,a)\) are defined as minimizers of the population matching loss
\[
\sup_{f\in F_h}\left|E_D[w(s,a)f(s,a)] - E_D[w_{h-1}(\cdot)f(\cdot,\pi)]\right|.
\]
When \(F_h\) is a reproducing-kernel or linear class, the paper gives the representation
\[
w_h^\star(s,a)=\phi(s,a)^\top \Sigma_h^{-1}\psi_h,
\qquad
\psi_h = B_{h-1}^\pi\cdots B_1^\pi\,E_D[\phi(s_1,\pi)],
\]
with
\[
\Sigma_h=E_D[\phi\phi^\top].
\]
The associated coverage parameter is
\[
\|w_h^\star\|_{2,d_h^D}
=
\sqrt{E_{d_h^D}\bigl[(w_h^\star)^2\bigr]} \, .
\]
Under Bellman completeness, \(\psi_h=E_{d_h^\pi}[\phi]\) [2605.06474].

The interpretation given in the paper is that \(\|w_h^\star\|\) measures how well the dataset covers the “projected next-step features” needed to evaluate the target policy. This reformulates coverage away from raw occupancy overlap and toward a feature-space quantity determined by the discriminator classes. The same parameter controls higher-order terms in the random-design bound and the sample size required for \(L_h(w_h)=0\) with high probability [2605.06474].

A plausible implication is that Q-MMR distinguishes between two kinds of mismatch: lack of support in the raw state-action space, and lack of support only after projection through \(F_h\). The framework treats the latter as the operative obstruction for estimation error, which is consistent with its reliance on moment constraints rather than full density-ratio recovery.

## 6. Advantages, limitations, and open problems

The advantages emphasized for Q-MMR are tightly tied to its theorem. The leading term in the finite-sample bound is dimension-free; the uncertainty decomposition is data-dependent and can be monitored on hold-out data; the framework recovers Monte Carlo, importance sampling, linear FQE, and tabular certainty-equivalence as special cases; and the main guarantee requires only realizability of \(Q^\pi\), not Bellman completeness [2605.06474].

The limitations are equally explicit. The current analysis assumes i.i.d. trajectories; adaptive or online data collection violates the conditional-independence structure used in the concentration arguments. Extending the framework to Markov-chain samplers or infinite-horizon discounted MDPs is identified as open. The algorithm is also greedy and local in the sense that each \(w_h\) optimizes only the level-\(h\) matching objective without directly accounting for downstream effects. Finally, for general function classes \(F_h\), Q-MMR requires a minimax optimization, and the paper notes that practical implementations rely on no-regret oracles together with best responses over \(F_h\), making efficient realization for large neural classes or RKHSs an implementation challenge [2605.06474].

Within offline reinforcement learning, these points place Q-MMR at the intersection of theory-driven reweighting, value-function realizability, and coverage analysis. Its contribution is less a new heuristic estimator than a framework for understanding off-policy evaluation through recursive moment matching, with error terms that are explicitly decomposed across time, approximation, and reweighting variance [2605.06474].

Source: https://www.emergentmind.com/topics/q-mmr