A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning (1012.2599v1)

Abstract: We present a tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions. Bayesian optimization employs the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function. This permits a utility-based selection of the next observation to make on the objective function, which must take into account both exploration (sampling from areas of high uncertainty) and exploitation (sampling areas likely to offer improvement over the current best observation). We also present two detailed extensions of Bayesian optimization, with experiments---active user modelling with preferences, and hierarchical reinforcement learning---and a discussion of the pros and cons of Bayesian optimization based on our experiences.

Bayesian Optimization for Expensive Cost Functions: Applications and Insights

The paper "A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning" by Eric Brochu, Vlad M. Cora, and Nando de Freitas, provides a comprehensive tutorial on Bayesian optimization (BO) techniques and their practical applications in active user modeling and hierarchical reinforcement learning (HRL). This essay will distill the key methodologies, results, and implications from the paper, geared towards fellow researchers in the field.

Overview of Bayesian Optimization

Bayesian optimization is designed to efficiently find the extrema of expensive cost functions, particularly useful when evaluations of the objective function are costly, derivatives are unavailable, or the problem is non-convex. BO uses a probabilistic model, typically a Gaussian Process (GP), to estimate the objective function and an acquisition function to decide where to sample next based on a trade-off between exploration and exploitation.

Core Components of BO

1. Gaussian Process Priors: GPs are favored for their flexibility in modeling complex, unknown functions. Key elements of a GP include the mean function (often assumed zero) and the covariance function (or kernel). Common kernels include the squared exponential and the Matérn kernel, which determine the smoothness and general characteristics of the functions being modeled.
2. Acquisition Functions: Various acquisition functions like Probability of Improvement (PI), Expected Improvement (EI), and Upper Confidence Bound (UCB) guide the selection of the next sampling point. These functions balance the trade-off between exploration—sampling regions of high uncertainty—and exploitation—sampling areas expected to yield high values of the objective function.

Applications Demonstrated in the Paper

Active User Modeling with Preferences

Problem Setup: Traditional user modeling often involves direct scalar ratings, which are unreliable due to user inconsistency and cognitive burdens. Bayesian optimization addresses this by modeling user preferences using a probit model, which interprets user comparisons between pairs of instances.

Methodology:

• Probit Model: Utilizes a Gaussian likelihood to model pairwise preferences, allowing integration with GPs to maintain a probabilistic framework over user preferences.
• Laplace Approximation: Used to approximate the posterior distribution of the latent utility function, which provides a robust method to infer user preferences from limited and noisy data.

Results: The approach showed significant efficiency in modeling user preferences for applications like finding desired material properties (BRDF) through a preference gallery. Empirical results highlighted that the method using expected improvement (EI) significantly reduced the number of iterations needed to find the target compared to random and maximum variance sampling strategies.

Hierarchical Reinforcement Learning

Problem Setup: Hierarchical control problems such as navigating a complex environment or controlling a vehicle involve a mix of discrete and continuous decisions. Traditional HRL faces efficiency issues due to the need for exploring large state-action spaces.

Methodology:

• Hierarchically Decomposed Tasks: Tasks are structured into a hierarchy, with high-level decisions like route planning broken down into simpler subtasks such as local navigation.
• Bayesian Optimization for Task Learning:
  • Active Policy Optimization: A parameterized policy for lower-level tasks, optimized using the expected improvement criterion to minimize the number of costly function evaluations.
  • Active Value Learning: GPs are used for the value function approximation in discrete map navigation tasks, leveraging BO to focus exploration on the most relevant parts of the state space.

Results: The approach successfully integrated Bayesian active exploration with the MAXQ HRL framework. The optimized method significantly accelerated the learning process, evidenced by efficient navigation of a simulated city environment based on a topological map of Vancouver, BC.

Implications and Future Work

Practical Implications:

• Interactive Systems: Bayesian optimization shows great promise in applications requiring human interaction, significantly reducing user burden and improving system responsiveness.
• Reinforcement Learning: In HRL, BO can address the exploration-exploitation dilemma more efficiently than traditional random or heuristic approaches, particularly in high-dimensional and continuous state spaces.

Theoretical Implications:

• Scalability: The techniques must evolve to handle higher-dimensional spaces more robustly, possibly through advanced kernel methods or dimensionality reduction techniques.
• Sequential Optimization: Extending BO to handle multi-step optimization and batch sampling remains a critical area for enhancing applicability in dynamic and real-time settings.

In conclusion, Bayesian optimization presents a compelling framework for efficiently solving complex optimization problems where function evaluations are expensive. The paper by Brochu, Cora, and de Freitas not only elucidates the core principles of BO but also demonstrates its practical efficacy in diverse applications such as user modeling and hierarchical control, paving the way for future advancements and broader applications in the field.