Bayesian Bandit Frameworks Overview

• Bayesian bandit frameworks are probabilistic models that apply Bayesian inference to balance exploration and exploitation in sequential decision-making.
• They employ algorithms like Thompson Sampling and Bayesian UCB, achieving rigorous regret guarantees in diverse settings including stochastic, contextual, and adversarial environments.
• Approximate inference methods, using techniques such as α-divergences and particle filters, enhance scalability and robustness in high-dimensional and decentralized applications.

Bayesian bandit frameworks provide a probabilistically grounded approach to sequential decision-making problems characterized by exploration–exploitation trade-offs. By representing uncertainty using posterior distributions and leveraging Bayesian inference, these frameworks have enabled a plethora of algorithms—spanning Thompson Sampling, Bayesian UCBs, nonparametric Bayesian bandits, high-dimensional models, and decentralized and hierarchical extensions—with rigorous regret guarantees in stochastic, contextual, non-stationary, and even adversarial or multi-agent environments.

1. Problem Formulations and Bayesian Posterior Construction

Bayesian bandit models formalize the sequential decision process as follows: At each time step $t=1,\ldots,T$, the learner observes a context (possibly empty), selects an action $a_t$ from an action set (arms or action set $X_t$), and then receives a reward $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$ or $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$ depending on context and model assumptions (Huang et al., 2024, Urteaga et al., 2017).

The unknown parameters $\theta^*$ (for linear/contextual bandits) or $\{\theta_a\}$ (for multi-armed) are assigned prior distributions (e.g., Gaussian, Beta, or nonparametric Dirichlet Process mixtures), which summarize domain beliefs before data. After each observation, Bayes’ rule is used to update the posterior $$p(\theta|D_t) \propto p(\theta)\prod_{\tau=1}^t f_{a_\tau}(y_\tau|\theta)$$.

Key features:

• For stochastic linear bandits, the action space is $X_t \subset \mathbb{R}^d$, $\|\theta^*\|\leq S$, $a_t$0, with noise $a_t$1 being zero-mean and $a_t$2-sub-Gaussian (Huang et al., 2024).
• In nonparametric settings, each arm has a DP mixture prior $a_t$3 with Gaussian base measures (Urteaga et al., 2018).

2. Bayesian Algorithms: Decisions and Posterior Approximation

Canonical Bayesian bandit algorithms select actions by leveraging the posteriors in different ways:

• Thompson Sampling (TS): At each round, sample parameters $a_t$4 from the posterior (exact or approximate), select $a_t$5 (for linear bandits) or $a_t$6 (for MABs) (Santosh et al., 2021, Huang et al., 2024).
• Bayesian UCB (Bayes-UCB, LinBUCB): Compute a high quantile (e.g., posterior $a_t$7 quantile) over possible rewards and select the maximizer (Huang et al., 2024).
• Information-Geometric/Belief-Manifold Algorithms (BelMan): Alternate between geometric barycenters of joint beliefs and rewards, performing I-projection for action selection and rI-projection for global update (Basu et al., 2018).

Posterior approximation is vital when closed forms are inaccessible:

• Approximate Inference via $a_t$8-divergences: When true posteriors are intractable, sample from approximations $a_t$9 whose discrepancy from the true $X_t$0 is quantifiable by bounded $X_t$1-divergence measures, covering KL, Hellinger, and $X_t$2 (Huang et al., 2024).
• Sequential Monte Carlo (SMC): Weighted particle filters approximate evolving posterior measures; especially powerful for nonstationary, contextual, and nonlinear/reward models (Cherkassky et al., 2013, Urteaga et al., 2018).

Hardware-efficient approximations for resource-constrained environments rely on methods such as order-statistics sampling for Beta distributions (Santosh et al., 2021).

3. Regret Guarantees and Theoretical Analyses

Regret for Bayesian bandit frameworks is most commonly measured in the (frequentist) expected regret sense: $X_t$3 For MABs, this reduces to $X_t$4. Key findings include:

• Exact Inference Regret: LinTS achieves $X_t$5, LinBUCB matches the minimax-optimal $X_t$6 (Huang et al., 2024). Classical TS and Bayes-UCB in MABs give $X_t$7 (Urteaga et al., 2017).
• Approximate Inference Penalties: As long as the $X_t$8-divergences between true and approximate posteriors are bounded, LinTS and LinBUCB retain their original regret rates up to larger constant factors (Huang et al., 2024).
• Nonparametric Bandits: Dirichlet-process mixture TS achieves $X_t$9 regret, where $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$0 depends on DP base tail regularity (Urteaga et al., 2018).
• Decentralized: In networked settings, decentralized TS matches centralized regret up to $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$1 and network-mixing-dependent constants, with regret scaling as $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$2 (Lalitha et al., 2020).

Universal themes are the preservation of optimal regret scaling under proper approximation error control and the inherent robustness of Bayesian sampling methods to model uncertainty.

4. Approximate Inference: $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$3-Divergences and Key Assumptions

The theoretical framework for analyzing Bayesian bandit algorithms with approximate inference relies on quantifying the deviation $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$4 for each round $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$5 (Huang et al., 2024). The Tsallis-style $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$6-divergence generalizes Kullback–Leibler and other divergences: $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$7 Assumption 2.4 of (Huang et al., 2024) requires that for appropriate choices of $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$8, $r_{t+1} = x_t^\top \theta^* + \xi_{t+1}$9, the divergences $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$0 and $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$1 are uniformly bounded.

Key implications:

• If the approximate inference mechanism (e.g., variational, MCMC, SMC, Laplace) ensures these bounds, regret guarantees hold.
• These results are algorithm-agnostic: the structure applies to both LinTS and LinBUCB and, under mild regularity, even to general Bayesian bandit algorithms.

5. Role and Definition of Well-Behaved Distributions

A new notion introduced in (Huang et al., 2024) is that of "well-behaved" distributions to formalize favorable conditions on the data-generating process that permit minimax-optimal rate achievement for LinBUCB. The well-behavedness criterion involves anti-concentration and concentration properties on the rescaled posterior, specifically that for any direction $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$2 and quantile $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$3,

$r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$4

satisfies lower and upper bounds (see Assumption 2.3 in (Huang et al., 2024)). The main significance is:

• Enforcing this condition enables LinBUCB, even under approximate inference, to accelerate LinTS regret from $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$5 to the optimal $r_{t+1} = f_{a_t}(y_{t+1}|\theta_{a_t})$6.
• The definition is agnostic to the actual Bayesian machinery, requiring only concentration control under the adopted posterior/rescaled measure.

6. Algorithmic Variants and Practical Implementation

Frameworks in the literature employ a spectrum of algorithmic choices:

• Double Sampling: Adaptively adjusts the number of sampled arms per round in proportion to estimated uncertainty, thereby interpolating between TS and greedy play as more is learned (Urteaga et al., 2017).
• Nonparametric Bandits: Per-arm Dirichlet-process Gaussian mixture priors, with sequential Gibbs sampling or SMC filter-based updates to approximate complex, unknown reward distributions (Urteaga et al., 2018, Urteaga et al., 2018).
• Reconfigurable System-on-Chip Implementations: Hardware-amenable TS and UCB variants utilizing PRNG-based Beta sampling and aggregation logic for algorithm selection in real-time edge scenarios (Santosh et al., 2021).
• Posterior Particle Filtering: SMC/particle filters maintain an online weighted set of parameter particles for each arm, providing scalable non-Gaussian updates and robust adaptation in nonstationary (restless) or high-dimensional settings (Cherkassky et al., 2013, Urteaga et al., 2018).
• Distributed Bayesian Bandits: Agents on a network run local Bayesian updates, communicate (via “mixing” of Bayesian parameters), and achieve global near-centralized regret with decentralized computation (Lalitha et al., 2020).

7. Extensions: Lifelong Learning, Hierarchical & Meta-Bandits

Recent work extends Bayesian bandit frameworks into higher-order learning regimes:

• Lifelong/Lifelong PAC-Bayes Bandits: Hierarchical (hyper-)priors are learned across multiple independent bandit tasks, optimizing explicit PAC-Bayes reward lower bounds for transfer (Flynn et al., 2022).
• Empirical Bayesian Multi-Bandit Models: Hierarchical Gaussian models with empirical-Bayes covariance estimation across tasks yield Thompson/UCB-style algorithms and regret bounds scaling in task and arm count (Jiang et al., 30 Oct 2025).
• Adversarial and Robust Design: Bayesian principles now underwrite regret-optimal algorithms in adversarial, stochastic, and nonstationary regimes via information-theoretic criteria without need for true priors (Xu et al., 2023).

Across these settings, Bayesian bandit algorithms maintain their core advantages: coherent uncertainty quantification, adaptability to data/model mismatch, transfer learning through hierarchical priors, and theoretically justified decision rules.

---

Selected References

• Bayesian Bandit Algorithms with Approximate Inference in Stochastic Linear Bandits (Huang et al., 2024)
• Nonparametric Gaussian Mixture Models for the Multi-Armed Bandit (Urteaga et al., 2018)
• Multi-armed Bandit Algorithms on System-on-Chip: Go Frequentist or Bayesian? (Santosh et al., 2021)
• Bayesian bandits: balancing the exploration-exploitation tradeoff via double sampling (Urteaga et al., 2017)
• Empirical Bayesian Multi-Bandit Learning (Jiang et al., 30 Oct 2025)
• Bayesian Algorithms for Decentralized Stochastic Bandits (Lalitha et al., 2020)
• Bayesian Design Principles for Frequentist Sequential Learning (Xu et al., 2023)