Introduction to Multi-Armed Bandits (1904.07272v8)

Abstract: Multi-armed bandits a simple but very powerful framework for algorithms that make decisions over time under uncertainty. An enormous body of work has accumulated over the years, covered in several books and surveys. This book provides a more introductory, textbook-like treatment of the subject. Each chapter tackles a particular line of work, providing a self-contained, teachable technical introduction and a brief review of the further developments; many of the chapters conclude with exercises. The book is structured as follows. The first four chapters are on IID rewards, from the basic model to impossibility results to Bayesian priors to Lipschitz rewards. The next three chapters cover adversarial rewards, from the full-feedback version to adversarial bandits to extensions with linear rewards and combinatorially structured actions. Chapter 8 is on contextual bandits, a middle ground between IID and adversarial bandits in which the change in reward distributions is completely explained by observable contexts. The last three chapters cover connections to economics, from learning in repeated games to bandits with supply/budget constraints to exploration in the presence of incentives. The appendix provides sufficient background on concentration and KL-divergence. The chapters on "bandits with similarity information", "bandits with knapsacks" and "bandits and agents" can also be consumed as standalone surveys on the respective topics.

• The paper presents core MAB algorithms such as UCB, Thompson Sampling, and Exp3, demonstrating near-optimal log-regret performance in both IID and adversarial settings.
• The paper transitions from basic IID rewards to complex contextual bandit models, offering structured insights for practical applications.
• The paper bridges theoretical rigor with real-world use cases in recommendation systems and economics while outlining avenues for future research.

Introduction to Multi-Armed Bandits

The foundational monograph "Introduction to Multi-Armed Bandits" by Aleksandrs Slivkins provides comprehensive coverage of the multi-armed bandit (MAB) framework, a central area in sequential decision-making under uncertainty. The text serves as both an introduction and an in-depth treatment of various aspects of MABs, targeting a wide range of researchers from beginners to seasoned experts.

Structure and Content

Slivkins organizes the book into several coherent chapters that cater to different models and applications of MABs. The progression from basic IID models to more complex scenarios such as adversarial rewards and contextual bandits ensures a structured learning path.

1. Basic IID Rewards: The initial chapters cover the basic model with independent and identically distributed (IID) rewards, emphasizing fundamental results such as UCB and Thompson Sampling algorithms. These chapters lay out the initial assumptions and provide elementary proofs that are robust enough to guide further developments in the book.
2. Adversarial Rewards: Subsequent chapters delve into adversarial settings where rewards can vary in possibly adversarial ways. This includes adaptations of basic algorithms and novel methods like Exp3, specifically designed to handle adversarial contexts.
3. Contextual Bandits: Highlighting a middle-ground approach, the book dedicates a chapter to contextual bandits where the reward distributions are influenced by observable contexts. This model is particularly relevant for real-world applications like recommendation systems and online advertising.
4. Connections to Economics: The final chapters bridge MABs with economic theories, exploring applications in auctions, pricing, and other economic paradigms. These sections also cover advanced concepts such as bandits with supply/budget constraints.

Numerical Results and Claims

Throughout the text, the author makes several noteworthy claims backed by rigorous proofs:

• Optimal Regret Bounds: The concepts of upper confidence bounds (UCBs) and Thompson Sampling are highlighted for their near-optimal regret properties in stochastic settings. It's shown that these algorithms can achieve log-regret in certain scenarios, which are optimal up to constant factors.
• Adversarial Environments: For adversarial settings, the Exp3 algorithm is showcased with regret bounds that are tight up to logarithmic factors. This presents a strong result with implications for robustness in uncertain environments.
• Contextual Bandits: The text details how contextual bandit algorithms can be augmented with linear or Lipschitz assumptions to achieve efficient learning. Specifically, the LinUCB algorithm is emphasized for its effectiveness in handling linear reward functions with contextual information.

Implications and Future Directions

By thoroughly covering these varying aspects of MABs, Slivkins lays a robust groundwork for both theoretical exploration and practical implementation.

• Practical Implications: Algorithms like UCB and Thompson Sampling are not only theoretically sound but have significant practical relevance in fields like web optimization, personalized recommendations, and economic decision-making.
• Theoretical Directions: The work opens numerous avenues for theoretical advancements. For instance, it raises questions about tighter lower bounds in more complex settings or the extension of MABs to decentralized environments.

Conclusion

"Introduction to Multi-Armed Bandits" is an indispensable reference that encapsulates the rich and multi-faceted nature of bandit problems. By systematically tackling both fundamental and advanced topics, the monograph stands out as a critical resource for researchers looking to deepen their understanding or apply MAB models to real-world problems. The treatments of IID, adversarial, and contextual bandits culminate in a text that is both broad and deep, ensuring its place in the canon of sequential decision-making literature.