Future Off-Policy Evaluation

• Future Off-Policy Evaluation (F-OPE) is a method that estimates a policy's future value using past data while accounting for non-stationary, shifting environments.
• It employs time-feature decomposition and doubly robust estimation to leverage recurring temporal patterns and mitigate biases caused by distributional shifts.
• Demonstrated by the OPFV estimator, F-OPE offers theoretical guarantees and practical improvements in applications like recommendation systems and digital advertising.

Future Off-Policy Evaluation (F-OPE) refers to the estimation of a policy’s value at future time points using only historical data, where the underlying environment and data distributions may shift with time. Unlike standard off-policy evaluation (OPE), which assumes stationarity, F-OPE explicitly targets non-stationary environments common in real-world applications such as recommendation systems, digital advertising, and healthcare—domains where context and reward dynamics evolve and future data is unavailable at evaluation time.

1. Motivation and Problem Formulation

F-OPE arises when one must predict how a candidate policy will perform in a future period (e.g., an upcoming month or season), given that all available data was collected in the past under possibly different distributional conditions and perhaps different policies. Formally, given logged data $\mathcal{D} = \{(x_i, t_i, a_i, r_i)\}_{i=1}^n$ up to time $T$ and a policy $\pi_e$, the objective is to estimate: $$V_{t'}(\pi_e) := \mathbb{E}_{p(x|t')\,\pi_e(a|x, t')}[q(x, t', a)],$$ where $t' > T$ is a future time point, $q(x, t', a)$ is the mean reward for $(x, a)$ at time $t'$, and $p(x|t')$ is the future context distribution.

This formulation is critical whenever distributions exhibit predictable changes (e.g., seasonality) or abrupt shifts (e.g., policy changes, holiday effects). Standard OPE methods typically assume $p(x|t') = p(x|t)$ and $q(x, t', a) = q(x, t, a)$ for all $t < t'$, which is rarely realistic in deployed systems.

2. Challenges in F-OPE

Deploying F-OPE in non-stationary settings presents several unique challenges:

• Absence of Future Data: There is no logged data from the future period $t'$ when evaluation is needed, precluding straightforward estimation or direct model fitting to $p(x|t')$ and $q(x, t', a)$.
• Non-stationarity Bias: Covariate and reward distributions may change between the logging and target periods, inducing significant bias when applying conventional OPE estimators (such as IPS or DR) that rely on the i.i.d. stationarity assumption.
• Variance–Bias Tradeoffs: Partitioning logged data by period to isolate temporal shifts leads to smaller effective sample sizes, increasing estimation variance.
• Inadequate Extrapolation: Regression-based extrapolation (as in some prior approaches) can be unreliable, particularly for extended horizons or abrupt distributional changes.
• Lack of Mechanisms to Exploit Temporal Patterns: Typical methods do not leverage recurring patterns (e.g., weekly, seasonal effects), nor do they support sharing information across time periods with similar structure.

3. The OPFV Estimator: Key Methodology

The Off-Policy Estimator for the Future Value (OPFV) is specifically designed to address F-OPE by exploiting time series structure in the historical log. Its core innovations include:

• Time Feature Decomposition: The expected reward is represented as

$q(x, t, a) = g(x, \phi(t), a) + h(x, t, a),$

where $\phi(t)$ extracts a (possibly multidimensional) time feature (e.g., weekday, seasonal segment) that summarises repeated temporal effects. The function $g$ captures the component predictable from $\phi(t)$, while $h$ models residual effects.

• Doubly Robust Estimation with Time-Feature Importance Weighting: The estimator applies importance weighting not only over actions but also over time features, seeking samples from the historical log that match the feature $\phi(t')$ of the future time of interest. For each logged sample $(x_i, t_i, a_i, r_i)$,

$$\hat{V}^{\mathrm{OPFV}}_{t'} (\pi_e) = \frac{1}{n} \sum_{i=1}^n \left[ \frac{\mathbb{I}_{\phi}(t_i, t')}{p(\phi(t'))} \frac{\pi_e(a_i | x_i, t')}{\pi_0(a_i | x_i, t_i)} (r_i - \hat{f}(x_i, t_i, a_i)) + \mathbb{E}_{\pi_e(a|x_i, t')}[\hat{f}(x_i, t', a)] \right],$$

where $\mathbb{I}_\phi(t, t') = 1$ if $\phi(t) = \phi(t')$, and $p(\phi(t'))$ is the empirical probability of $\phi(t')$ in the historical data.

• Handling Non-stationarity and Recurring Patterns: By leveraging samples with matching temporal features, OPFV makes it possible to generalize historical signals into the future without the need for direct future samples, under the assumption that the effect of $\phi(t)$ is semistationary.
• Doubly Robust Structure: The approach combines a regression adjustment term ($\hat{f}$, a reward model fit on historical data) with importance weighting for both policy and time feature, mitigating both action distribution and known temporal distributional shift.

4. Theoretical Properties and Bias–Variance Tradeoffs

OPFV delivers rigorous statistical guarantees that underpin its application in F-OPE:

• Bias: The estimator is unbiased for the future policy value when both the reward function is fully captured by the time feature and the modeling error by the regression $\hat{f}$ is negligible, and the future time feature is observed in the historical log (satisfying common support).
• Variance: Bias decreases as the time feature $\phi$ becomes more fine-grained, bringing historical and future samples closer in distribution, while variance generally increases due to fewer samples per matched time feature.
• Tradeoff: There exists a fundamental bias-variance tradeoff in the granularity of the time feature. Coarse features induce bias if $g(\cdot, \phi(t), \cdot)$ fails to distinguish changes, while fine features increase estimator variance.

Theoretical analysis provides a non-asymptotic bias expression: $$\text{Bias}(\hat{V}_{t'}^{\mathrm{OPFV}}) = \mathbb{E}\left[ \frac{\mathbb{I}_{\phi}(t, t')}{p(\phi(t'))} \big( \Delta_q(x, t, t', a) - \Delta_{\hat f}(x, t, t', a) \big) \right],$$ where $\Delta_q(x, t, t', a) = q(x, t, a) - q(x, t', a)$.

5. Experimental Validation and Practical Implications

Empirical evaluations are conducted on both synthetic and real-world (e-commerce recommendation) datasets:

• Synthetic Experiments: These show that when recurring calendar effects (e.g., seasonal rewards) are present and the time feature $\phi$ matches the underlying structure, OPFV attains near-unbiased future policy value estimation, outperforming standard IPS, DR, and regression extrapolation baselines by large margins.
• Real-world Analysis (KuaiRec): In weekly recommendation data with observable day-of-week effects, OPFV achieves the lowest future policy evaluation error compared to alternatives, underlining its practical applicability.
• Policy Optimization Extension: The estimator is naturally extended to policy gradient-based OPE in non-stationary settings, providing effective optimization of deployment policies aimed at future periods.

6. Broader Impact and Methodological Extensions

F-OPE and OPFV introduce paradigms for data-driven policy evaluation and policy learning that are acutely tailored for non-stationary, temporally structured environments:

• Practical Deployment: Applications requiring robust deployment under dynamics that shift with time—such as online retail and digital recommendations—stand to benefit directly from F-OPE methods.
• Automated Feature Selection: Accurate execution depends on careful specification or learning of temporal features $\phi$. Future directions involve feature engineering, automated selection, or representation learning for time series.
• Integration with Covariate Shift and External Validity: While OPFV targets temporal/covariate non-stationarity, it is complementary to methods addressing covariate shift, confounding, and external validity more broadly.

7. Future Research Directions

• Learning Temporal Structure: The development of automated, possibly nonparametric methods for feature learning (for $\phi$) is anticipated to significantly enhance F-OPE.
• Adaptation to Abrupt and Unknown Shifts: While the framework supports both smooth and abrupt changes, further refinements could improve adaptability to complex, high-frequency non-stationarity.
• Extensions to Other Domains: Applications to healthcare, finance, and broader RL deployment settings, where future-targeted evaluation is critical and non-stationarity is pervasive.

---

Feature	Standard OPE	F-OPE/OPFV Approach
Handles non-stationarity	No	Yes (explicit)
Uses future time features	No	Yes
Benefits from recurring structure	No	Yes
Bias–variance tradeoff	Action-only	Time + action

F-OPE, exemplified by OPFV, establishes a methodological and practical foundation for safe and accurate policy evaluation under non-stationarity, a condition intrinsic to many deployed decision systems.