Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learning (2110.00871v1)

Abstract: Thompson Sampling has been widely used for contextual bandit problems due to the flexibility of its modeling power. However, a general theory for this class of methods in the frequentist setting is still lacking. In this paper, we present a theoretical analysis of Thompson Sampling, with a focus on frequentist regret bounds. In this setting, we show that the standard Thompson Sampling is not aggressive enough in exploring new actions, leading to suboptimality in some pessimistic situations. A simple modification called Feel-Good Thompson Sampling, which favors high reward models more aggressively than the standard Thompson Sampling, is proposed to remedy this problem. We show that the theoretical framework can be used to derive Bayesian regret bounds for standard Thompson Sampling, and frequentist regret bounds for Feel-Good Thompson Sampling. It is shown that in both cases, we can reduce the bandit regret problem to online least squares regression estimation. For the frequentist analysis, the online least squares regression bound can be directly obtained using online aggregation techniques which have been well studied. The resulting bandit regret bound matches the minimax lower bound in the finite action case. Moreover, the analysis can be generalized to handle a class of linearly embeddable contextual bandit problems (which generalizes the popular linear contextual bandit model). The obtained result again matches the minimax lower bound. Finally we illustrate that the analysis can be extended to handle some MDP problems.

• The paper proposes Feel-Good Thompson Sampling, a modified exploration mechanism that leverages historical optimism to improve frequentist regret bounds.
• It integrates an additional exploration term that aligns the empirical success of Thompson Sampling with theoretical minimax lower bounds for contextual bandit problems.
• The approach extends to reinforcement learning by adapting to linearly embeddable MDPs, offering practical benefits for applications like recommendation systems and online advertising.

Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learning

The paper "Feel-Good Thompson Sampling for Contextual Bandits and Reinforcement Learning" by Tong Zhang aims to enhance the theoretical understanding and practical efficacy of Thompson Sampling, particularly within the frequentist framework. Thompson Sampling is widely known for its adaptability to contextual bandit problems, yet its frequentist regret analysis has lacked sophistication. This work introduces a modified approach, dubbed Feel-Good Thompson Sampling, which addresses this gap by proposing a more aggressive exploration mechanism than standard Thompson Sampling.

Analysis of Thompson Sampling

The paper first delineates the limitations inherent in the traditional Thompson Sampling approach, particularly its insufficiency in exploring new actions decisively under frequentist regret formulations. Traditional Thompson Sampling assumes a probabilistic model that permits the choice of actions based on posterior distributions, yet can sometimes result in subpar exploration strategies, especially when the reward models are pessimistic. Consequently, it can lead to linear worst-case regret bounds, which are not optimal.

Feel-Good Exploration

To redress these shortcomings, Feel-Good Thompson Sampling introduces an additional exploration term to favor high-reward models more promptly. This exploration term encourages historical optimism, an idea that helps drive the sampling process toward models that promise maximal rewards based on historical data. By integrating this mechanism, the approach employs the principles of optimism in the face of uncertainty, similar to strategies like UCB (Upper Confidence Bound), within the context of Thompson Sampling.

Regret Bound Analysis

Feel-Good Thompson Sampling leads to improved frequentist regret bounds that match the minimax lower bound for finite-action contextual bandit problems. The theoretical framework facilitates transforming the bandit regret problem into an online least squares regression estimation. Specifically, it ensures that both Bayesian and frequentist regret bounds can be derived within the same framework, leveraging decoupling coefficient techniques to separate action choices from value estimations.

Extending to Reinforcement Learning

The paper also extends its evaluation to certain reinforcement learning settings. By considering deterministic transitions within the contextual episodic Markov decision processes (MDPs), the analysis demonstrates how Feel-Good Thompson Sampling can be generalized to the reinforcement learning domain. The framework maintains its efficacy by adapting to linearly embeddable MDP structures, allowing contextually dependent non-linear embeddings of linear functions.

Implications and Future Directions

The modifications proposed by Feel-Good Thompson Sampling provide a step forward in aligning the empirical success of Thompson Sampling with its theoretical backing in frequentist paradigms. The implications of this work extend beyond theoretical reinforcement, impacting practical applications in areas like recommendation systems and online advertising, where contextual bandit models are prevalent. Future developments in AI may further exploit this theoretical groundwork, potentially extending the framework to more complex and structured bandit problems beyond the settings considered in this paper.

In summary, Feel-Good Thompson Sampling enriches the exploration process of Thompson Sampling, bridging the long-standing gap in its frequentist regret analysis. The theoretical contributions offer robust mechanisms for both contextual bandit and reinforcement learning problems with computational methods feasible for practical implementations.