Bayesian Optimization for Policy Search via Online-Offline Experimentation (1904.01049v2)

Abstract: Online field experiments are the gold-standard way of evaluating changes to real-world interactive machine learning systems. Yet our ability to explore complex, multi-dimensional policy spaces - such as those found in recommendation and ranking problems - is often constrained by the limited number of experiments that can be run simultaneously. To alleviate these constraints, we augment online experiments with an offline simulator and apply multi-task Bayesian optimization to tune live machine learning systems. We describe practical issues that arise in these types of applications, including biases that arise from using a simulator and assumptions for the multi-task kernel. We measure empirical learning curves which show substantial gains from including data from biased offline experiments, and show how these learning curves are consistent with theoretical results for multi-task Gaussian process generalization. We find that improved kernel inference is a significant driver of multi-task generalization. Finally, we show several examples of Bayesian optimization efficiently tuning a live machine learning system by combining offline and online experiments.

• The paper proposes a multi-task Gaussian process framework that fuses offline simulator data with online experiments to efficiently explore policy spaces.
• The methodology employs an ICM kernel to model inter-task relationships, mitigating simulator bias and significantly improving prediction accuracy.
• Empirical tests on recommendation systems show that the approach reduces costly online tests while achieving superior policy optimization performance.

An Overview of Bayesian Optimization for Policy Search via Online-Offline Experimentation

This paper presents an innovative methodology of leveraging Bayesian optimization to effectively explore and optimize policy spaces within interactive machine learning systems by combining online and offline experimentation. The critical challenge addressed here is the limitation posed by the low throughput of online tests, particularly in complex policy environments characteristic of large, multi-dimensional systems such as recommendation engines.

Methodological Approach

The paper employs a multi-task Gaussian process model (MTGP) to integrate simulator-based data with real-world online experiments. By doing so, it aims to exploit the computational efficiency of simulators while adjusting for biases inherent in their predictions. The paper uses a naïve simulator to provide offline evaluations based on historic data and event prediction models, then juxtaposes these results with those from traditional online field experiments.

A critical component of this methodology is the ICM kernel, a covariance function tasked with modeling inter-task relationships while assuming a shared spatial covariance. This kernel allows the optimization process to adaptively learn and correct simulator biases against the online data, thus facilitating a more reliable approximation of real-world policy outcomes.

Empirical Evaluation

In a series of empirical tests conducted on a live recommendation system's value model, the paper highlights the MTGP's proficiency in significantly improving model predictions and optimization outcomes. Across a series of experiments involving optimization over parameter spaces with dimensions ranging from 10 to 20, the multi-task model exhibited considerable predictive improvements over single-task counterparts that rely solely on online evaluations. Key empirical findings underscore not only the efficacy of simulator data in enhancing predictive performance but also demonstrate the ability of the MTGP to identify and exploit patterns in simulator bias.

For instance, the results depicted that, using the MTGP, even biased simulator data could lead to substantial improvements in prediction quality. Furthermore, iterative optimizations mediated by MTGP were able to achieve better overall performance metrics with fewer online tests compared to completely online methods, showcasing the potential to reduce the computational and logistical burden of policy tuning in real-world applications.

Theoretical Insights

The theoretical framework presented sheds light on MTGP learning behaviors and the relative value of simulator data in optimizing policy inferences. A key conclusion drawn from the theoretical exploration is that the MTGP's proficiency hinges heavily on the inter-task correlation ($\rho^2$). This correlation serves as an effective indicator of the extent to which simulator outputs can be integrated with and enhance predictions from online data. For policies with a high $\rho^2$, simulator data leveraged by MTGP markedly accelerates learning and reduces prediction error.

Implications and Future Directions

The implications of this research are manifold. Practically, it offers a structured, empirically-verified approach to expediting the policy search process, especially within high-dimensional systems facing constrained experimental throughput. Theoretically, it validates the use of the MTGP model within policy optimization frameworks and sets a precedent for further exploration into the mitigation of bias in simulation-based optimization.

Looking forward, future developments may include refining the acquisition strategy to optimally balance offline and online experimentation, exploring alternative kernels and model architectures to capture even more complex inter-task dynamics, and applying these methods across broader domains where similar simulation biases are observed.

In summary, this work provides a comprehensive methodology and empirical proof of concept for integrating simulated experiences with online experimentation in efficient policy search, enabling more agile and informed decision-making within complex machine learning infrastructures.