Batch Ensemble for Variance Dependent Regret in Stochastic Bandits

Abstract: Efficiently trading off exploration and exploitation is one of the key challenges in online Reinforcement Learning (RL). Most works achieve this by carefully estimating the model uncertainty and following the so-called optimistic model. Inspired by practical ensemble methods, in this work we propose a simple and novel batch ensemble scheme that provably achieves near-optimal regret for stochastic Multi-Armed Bandits (MAB). Crucially, our algorithm has just a single parameter, namely the number of batches, and its value does not depend on distributional properties such as the scale and variance of the losses. We complement our theoretical results by demonstrating the effectiveness of our algorithm on synthetic benchmarks.

• The paper introduces a Batch Ensemble algorithm that optimizes regret in stochastic MAB by using a novel, optimistic mean estimator.
• It splits observed losses into multiple batches and selects the minimum batch average to guide exploration without needing explicit distribution models.
• The analysis delivers variance-dependent regret bounds that rival state-of-the-art methods while reducing parameter tuning and computational overhead.

This paper, "Batch Ensemble for Variance Dependent Regret in Stochastic Bandits" (2409.08570), introduces a simple and practical algorithm for stochastic @@@@1@@@@ Bandits (MAB) that achieves near-optimal, variance-dependent regret bounds without requiring explicit knowledge of arm distributions or extensive parameter tuning. The core idea is inspired by ensemble methods used in deep reinforcement learning to encourage exploration.

Problem Addressed:

The standard stochastic MAB problem involves an agent repeatedly selecting one of $K$ arms, each yielding a loss drawn from an unknown, fixed distribution. The goal is to minimize cumulative loss over $T$ rounds, which is equivalent to minimizing regret (the difference between the agent's cumulative loss and that of an oracle always choosing the best arm). Many existing algorithms, like UCB or KL-UCB, rely on constructing confidence bounds based on assumptions about the arm distributions (e.g., bounded support, specific parametric family like Bernoulli or Gaussian) and often require careful parameter tuning or prior knowledge of these distributions. This can be problematic when distributions are misspecified or heterogeneous. The paper aims to develop a simpler, more robust algorithm that adapts to the true concentration properties (like variance) of the distributions without explicit modeling.

Proposed Solution: Batch Ensemble Algorithm

The proposed solution is the Batch Ensemble algorithm (Algorithm 1). It operates based on a novel optimistic mean estimator constructed by splitting observed samples for each arm into multiple batches.

Here's the breakdown of the mechanism:

1. For each arm $a$, maintain a history of observed losses.
2. At each time step $t$, given $n_a(t-1)$ samples collected for arm $a$, these samples are conceptually split into $l_t$ batches (where $l_t$ is the number of batches at time $t$). A round-robin scheme is suggested for simplicity.
3. For each batch $l'$, an empirical average loss $\bar{\mu}_{n_a(t-1), a, l'}$ is computed based on the samples assigned to that batch. A small constant (2) is added to the denominator of the average (i.e., $\frac{\sum \text{losses}}{\text{samples}+2}$) to ensure a strong optimism property, especially with few samples.
4. The Batch Ensemble estimator for arm $a$ at time $t$, denoted $\tilde{\mu}_{t, n_a(t-1), a}$, is defined as the minimum of the empirical averages across all batches: $$\tilde{\mu}_{t,n,a} = \min_{l' \in [l_t]} \bar{\mu}_{t,n,a,l'}$$. Since the problem is defined with losses (where smaller is better), taking the minimum estimate is an optimistic strategy – it's more likely to underestimate the true expected loss.
5. The algorithm then chooses the arm with the minimum Batch Ensemble estimate: $$a_t \in \argmin_{a \in [K]} \tilde{\mu}_{t, n_a(t-1), a}$$.
6. Observe the loss for $a_t$ and update the sample counts and history.

The number of batches, $l_t$, is the algorithm's single parameter. The paper shows that setting $l_t$ based on the time horizon $T$ (e.g., $l = (7/2) \log (2T / \delta)$ for a high-probability bound or $l_t = 8 \log t$ for an anytime expected regret bound) yields strong guarantees.

Algorithm Implementation Details (Algorithm 1):

def batch_ensemble_mab(K, T, get_l_t_func): """ K: number of arms T: time horizon (or maximum expected time for anytime version) get_l_t_func: function that returns the number of batches l_t for time t """ sample_history = {a: [] for a in range(K)} # Store losses for each arm pull_counts = {a: 0 for a in range(K)} # Number of times each arm is pulled for t in range(1, T + 1): l_t = get_l_t_func(t) ensemble_estimates = {} for a in range(K): n_a = pull_counts[a] losses_a = sample_history[a] batch_estimates = [] # Split samples into l_t batches (round-robin) and compute estimates for l_prime in range(l_t): batch_samples_indices = [i for i, loss in enumerate(losses_a) if i % l_t == l_prime] batch_losses = [losses_a[i] for i in batch_samples_indices] batch_size = len(batch_losses) if batch_size == 0: # Convention for empty batch sum is 0 in paper, adjust denominator batch_estimate = 0 # This should be handled carefully, maybe based on definition with +2 # Paper uses sum / (count + 2). Empty sum is 0, empty count is 0. So 0/(0+2) = 0. else: batch_estimate = sum(batch_losses) / (batch_size + 2) batch_estimates.append(batch_estimate) # Compute the minimum estimate across batches for this arm ensemble_estimates[a] = min(batch_estimates) if batch_estimates else float('inf') # Handle case with 0 batches/samples # Choose arm with the minimum ensemble estimate chosen_arm = min(ensemble_estimates, key=ensemble_estimates.get) # Simulate pulling the chosen arm and observing loss # In a real environment, you would interact with the environment here observed_loss = simulate_loss(chosen_arm) # This is environment interaction # Update sample history and pull counts sample_history[chosen_arm].append(observed_loss) pull_counts[chosen_arm] += 1 # After T steps, calculate total regret or return pull_counts # Regret calculation omitted for brevity of algorithm pseudocode pass

Implementation Considerations:

• Computational Cost: At each time step $t$, for each arm $a$, the algorithm iterates through $l_t$ batches. Computing the batch mean takes $O(n_a(t-1)/l_t)$ time. Summing over all arms, the total computation per step is approximately $O(K \cdot l_t \cdot (T/l_t)) = O(KT)$ naively, but since each sample belongs to only one batch, the samples are distributed. The total time per step is dominated by computing batch means for arms pulled up to time $t-1$, which is $O(K l_t)$. With $l_t$ growing logarithmically in $T$, this is efficient ($O(K \log T)$ per step). The "Efficient" variant mentioned in experiments reuses batch calculations, further improving efficiency.
• Memory: The algorithm needs to store the sample history for each arm, which is $O(T)$. If only sample sums and counts per batch are stored, memory can be reduced to $O(K l_t)$.
• Parameter Tuning: The primary parameter is the number of batches, $l_t$. The theory provides guidance on how to set this based on $T$ and $\delta$, eliminating the need to tune based on unknown distribution properties.
• Distributed View: The paper suggests interpreting the algorithm as $l_t$ independent, simple bandit agents, each receiving a subset of samples and proposing an arm. The main agent selects the arm proposed by the agent with the best (minimum) empirical estimate. This view aligns with practical ensemble methods in RL and suggests potential for scaling.

Theoretical Guarantees:

The paper provides high-probability regret bounds (Theorem 1) and anytime expected regret bounds (Theorem 2, in Appendix). Crucially, these bounds depend on the variance $\sigma_a^2$ of the sub-optimal arms $\Delta_a$, exhibiting a variance-dependent term $$\sum_{a \neq a_*} \frac{\sigma_a^2}{\Delta_a} \log^2 T$$. For Bernoulli arms, $\sigma_a^2 = \mu_a(1-\mu_a)$, which can be much smaller than the worst-case variance of $1/4$ for a $[0,1]$ bounded distribution, especially when the mean is close to 0 or 1. The bounds also include standard terms like $\sum_{a \neq a_*} \frac{1}{\Delta_a} \log^2 T$. The bounds are near-optimal and match the best-known variance-dependent bounds achieved by more complex algorithms like UCB-V or KL-UCB, but the Batch Ensemble algorithm achieves this without using explicit variance estimates or distribution knowledge within the algorithm itself. The dependence on variance arises naturally from the use of Bernstein's inequality in the analysis of the batch means.

Mechanism for Optimism (Lemma 1):

The key to the algorithm's performance is the optimistic property of the minimum empirical estimate. For a single batch average $\bar{\mu}_{n,a,l'}$, the probability of underestimating the true mean $\mu_a$ is bounded below by a constant (e.g., 1/4 for Bernoulli, or 1/2 for symmetric distributions, using anti-concentration properties). By taking the minimum across $l$ independent batch estimates, the probability that at least one estimate is below the true mean $\mu_a$ becomes high. Specifically, $$\Pr(\min_{l'} \bar{\mu}_{n,a,l'} \le \mu_a) = 1 - \Pr(\forall l', \bar{\mu}_{n,a,l'} > \mu_a) \ge 1 - (1 - 1/4)^l \ge 1 - e^{-l/4 \times 2} = 1 - e^{-2l/7}$$ for Bernoulli (using a slightly refined bound than $1/4$). This guarantees that with high probability, the Batch Ensemble estimate for any arm is below its true mean, making it an optimistic choice rule.

Extensions Beyond Bernoulli Arms:

The paper discusses how the algorithm can work for various distributions, even though the theoretical proof is detailed for Bernoulli. The algorithm itself does not rely on the distribution type. The analysis, however, requires concentration and anti-concentration properties.

• Bounded Distributions: The concentration bound (Lemma 3, based on Bernstein) holds for any bounded distribution.
• Anti-Concentration: The optimism property (Lemma 1) relies on anti-concentration, the property that the empirical mean is likely to be below the true mean. This holds for Bernoulli distributions (Lemma 2). The paper argues it holds for symmetric distributions and distributions with lower-bounded variance after a sufficient warmup phase, based on Berry-Esseen theorem. A conjecture is made that it might hold for all bounded distributions, supported by experiments on Exponential arms.
• Bernoulli-fication: Any $[0,1]$ distribution can be converted to a Bernoulli one by replacing samples $x$ with $\text{Ber}(x)$. This allows applying the Bernoulli analysis, but the variance dependency might change.
• Scaled Bernoulli: The analysis can be adapted for distributions scaled by different factors $b_a$, showing that the regret depends on $b_a$ but only for sub-optimal arms, not the optimal one.

Experimental Results:

Experiments on synthetic Bernoulli, Gaussian, and Exponential MAB environments demonstrate the practical effectiveness of the Batch Ensemble algorithm.

• It performs comparably or better than UCB-KL, which is generally considered state-of-the-art for instance-dependent bounds, especially in scenarios with low-variance optimal arms (Test Case 1) or high-variance arms (Test Case 4 & 5).
• It significantly outperforms standard UCB and UCB-V in variance-sensitive scenarios.
• MARS, a related method based on random subsampling, shows much higher regret in the tested scenarios, often exhibiting near-linear regret initially.
• The "Efficient" implementation variant shows similar or slightly improved performance with better computational characteristics.
• The performance on Exponential arms supports the conjecture that the algorithm works well even for non-symmetric distributions without special warm-up phases.

The experiments validate that the Batch Ensemble approach is not only theoretically sound with variance-dependent guarantees but also competitive and computationally efficient in practice compared to existing methods.