A Tutorial on Thompson Sampling (1707.02038v3)

Abstract: Thompson sampling is an algorithm for online decision problems where actions are taken sequentially in a manner that must balance between exploiting what is known to maximize immediate performance and investing to accumulate new information that may improve future performance. The algorithm addresses a broad range of problems in a computationally efficient manner and is therefore enjoying wide use. This tutorial covers the algorithm and its application, illustrating concepts through a range of examples, including Bernoulli bandit problems, shortest path problems, product recommendation, assortment, active learning with neural networks, and reinforcement learning in Markov decision processes. Most of these problems involve complex information structures, where information revealed by taking an action informs beliefs about other actions. We will also discuss when and why Thompson sampling is or is not effective and relations to alternative algorithms.

• The paper demonstrates Thompson Sampling’s effectiveness as a Bayesian method for balancing exploration and exploitation in online decision tasks.
• The paper details approximate posterior sampling techniques, such as Gibbs sampling and Laplace approximations, to overcome computational challenges in complex models.
• The paper explores practical enhancements, addressing nonstationarity and contextual complexities in bandit, path optimization, and reinforcement learning applications.

Overview of "A Tutorial on Thompson Sampling"

This paper, "A Tutorial on Thompson Sampling," authored by Daniel J. Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen, serves as an extensive tutorial on Thompson Sampling (TS). It articulates the algorithm's applicability to various online decision-making problems, provides thorough theoretical insight, and backs its claims with practical examples and computational results.

Introduction to Thompson Sampling

Thompson Sampling is an online decision-making algorithm that balances the exploration-exploitation trade-off by sampling from posterior distributions. This inherently Bayesian approach has enjoyed vigorous interest due to its computational efficiency and predictable performance across a broad class of problems, including multi-armed bandits, shortest path problems, product recommendation, and reinforcement learning tasks.

The Mechanics of Thompson Sampling

TS operates by maintaining a posterior distribution over a model's parameters and sampling from this distribution to choose actions. This approach ensures that all actions are explored while favoring those considered optimal based on the current knowledge. The algorithm repeatedly updates the posterior distribution as it observes outcomes from chosen actions, hence refining its action-selection strategy over time.

Application to Bernoulli Bandits

The paper elucidates the application of TS using the Bernoulli bandit problem, a canonical example traditionally used to illustrate exploration-exploitation trade-offs. With an in-depth walkthrough, the authors demonstrate TS's superiority over naive exploration strategies like $\epsilon$-greedy methods. Empirical results highlight that TS effectively learns to select the best arm, minimizing the cumulative regret even in scenarios with multiple arms.

Generalizing Thompson Sampling

Beyond Bernoulli bandits, the paper extends the discussion to problems with more complex structures, such as path optimization under uncertain travel times. It demonstrates how TS can adapt to these problems, capturing interdependencies and leveraging information structures to efficiently guide exploration.

Approximate Posterior Sampling Methods

Due to computational constraints, exact Bayesian inference is often infeasible in complex models. The paper reviews several approximate methods for posterior sampling, such as Gibbs sampling, Langevin Monte Carlo, Laplace approximations, and bootstrapping. Each method's effectiveness is validated through computational experiments, emphasizing scenarios where approximate approaches still confer significant performance advantages.

Practical Considerations and Extensions

In practical applications, several nuances affect the implementation of TS:

• Prior Specification: The choice of prior distributions significantly impacts TS's performance. The paper underscores empirical methods for choosing informative priors based on historical data.
• Time-Varying Constraints and Contextual Decisions: Real-world problems may involve evolving constraints and contextual information. TS naturally extends to these cases, demonstrating flexibility and robustness.
• Nonstationarity: TS adapts to nonstationary systems by design, but the paper suggests modifications to ensure continued efficacy even when system dynamics change over time.

Additional Applications

TS's utility is illustrated through diverse examples:

• Product Assortment: An optimal subset of products to offer is dynamically chosen to maximize profit based on observed demand.
• News Recommendation: Context-sensitive recommendations are made by leveraging user features and feedback to refine article selections.
• Active Learning with Neural Networks: TS guides deep learning approaches, ensuring effective exploration alongside performance optimization in reinforcement learning.

Theoretical Foundations

The paper provides a comprehensive theoretical backdrop for TS, discussing its asymptotic optimality under specific conditions and extending regret analysis to more complex problems. Theoretical analyses indicate why TS works effectively in structured environments and highlight scenarios where it might underperform.

Conclusion

"A Tutorial on Thompson Sampling" is a rigorously detailed resource that not only explains the foundational principles of TS but also guides its practical application across varied and complex domains. The paper's thorough treatment of numerical methods, theoretical justifications, and contextual modifications makes it an invaluable reference for researchers and practitioners aiming to leverage Bayesian methods for online decision-making problems. Moreover, it emphasizes the importance of context, prior information, and adaptive strategies, thereby charting a comprehensive pathway for future research and application development in the domain of intelligent decision-making systems.