Optimal and Adaptive Off-policy Evaluation in Contextual Bandits (1612.01205v2)

Abstract: We study the off-policy evaluation problem---estimating the value of a target policy using data collected by another policy---under the contextual bandit model. We consider the general (agnostic) setting without access to a consistent model of rewards and establish a minimax lower bound on the mean squared error (MSE). The bound is matched up to constants by the inverse propensity scoring (IPS) and doubly robust (DR) estimators. This highlights the difficulty of the agnostic contextual setting, in contrast with multi-armed bandits and contextual bandits with access to a consistent reward model, where IPS is suboptimal. We then propose the SWITCH estimator, which can use an existing reward model (not necessarily consistent) to achieve a better bias-variance tradeoff than IPS and DR. We prove an upper bound on its MSE and demonstrate its benefits empirically on a diverse collection of data sets, often outperforming prior work by orders of magnitude.

• The paper establishes minimax lower bounds for off-policy evaluation, proving that IPS and DR estimators are optimal under agnostic reward conditions.
• It introduces the switch estimator, which adaptively blends direct methods with IPS/DR to better control bias and variance.
• The findings offer practical insights for improving policy evaluation in applications like personalized recommendations and medical decision-making.

Optimal and Adaptive Off-policy Evaluation in Contextual Bandits

This paper investigates the problem of off-policy evaluation (OPE) within the framework of contextual bandits, focusing on estimating the value of a target policy using data collected through another policy. The paper aims at understanding optimality in OPE, drawing attention to the significant challenges associated with it, especially in the absence of a consistent model of rewards. The contributions of the authors can be outlined in two key areas: theoretical bounds for OPE and the development of a novel estimator with practical performance benefits.

Theoretical Foundations and Minimax Limits

A central part of the paper is the establishment of a minimax lower bound on the mean squared error (MSE) for OPE under the contextual bandit model, without consistent reward model assumptions. Notably, this bound highlights the complexity of OPE in the agnostic contextual settings compared to scenarios with access to consistent reward models, such as in multi-armed bandits. The authors demonstrate that the inverse propensity scoring (IPS) and doubly robust (DR) estimators match this lower bound up to constant factors. Given non-degenerate context distributions, these estimators are shown to achieve the minimax risk for OPE, underscoring them as fundamentally unimprovable in the worst-case scenarios without additional assumptions.

The Switch Estimator

Addressing the limitations in bias and variance trade-offs of traditional estimators, the paper introduces the switch estimator. This estimator leverages available reward models to improve empirical performance on finite samples, offering a better bias-variance tradeoff than both IPS and DR. The switch estimator's design makes it robust to large importance weights, a known issue leading to large variance in IPS and DR. The switch estimator adapts by interpolating between a direct method (DM) and DR (or IPS), based on the size of importance weights, which is particularly valuable when reward models are imperfect but informative.

Formally, the switch estimator utilizes a threshold parameter to separate contexts into those with high and low importance weights, applying DM to the former and DR or IPS to the latter. The empirical utility of the switch estimator is supported by a thorough evaluation across numerous datasets where it often outperforms existing methods by significant margins.

Implications and Speculations

Practically, this paper's findings have substantial implications for contexts where direct reward models are potentially inconsistent but still valuable. This form of estimator selection could steer applications, such as personalized recommendations or medical decision-making, towards more efficient policy evaluation strategies, thereby enhancing decision-making processes.

Theoretically, these findings contribute to the broader understanding of OPE, marking the divergences between agnostic and consistent model settings. The minimax perspective enriches the literature on OPE with a sound risk framework, while empirically driven enhancements like the switch estimator illustrate the potential for adaptivity in highly variable or structured environments.

Future Directions

This research opens several avenues for further exploration. High-probability upper bounds on the MSE of switch estimators represent one promising direction. Moreover, the exploration of alternate reward model integration techniques or estimators beyond DR and IPS could yield new insights into managing the bias-variance tradeoff more effectively. From an application-based view, evaluating the flexibility of such estimators in real-world bandit settings across different industrial domains will inform their utility and limitations.

In conclusion, this paper sheds light on the fundamental difficulties in off-policy evaluation and offers practical advancements through the switch estimator, promising both theoretical insights and empirical enhancements for contextual bandit applications.