Adversarial Bandit over Bandits: Hierarchical Bandits for Online Configuration Management

Abstract: Motivated by dynamic parameter optimization in finite, but large action (configurations) spaces, this work studies the nonstochastic multi-armed bandit (MAB) problem in metric action spaces with oblivious Lipschitz adversaries. We propose ABoB, a hierarchical Adversarial Bandit over Bandits algorithm that can use state-of-the-art existing "flat" algorithms, but additionally clusters similar configurations to exploit local structures and adapt to changing environments. We prove that in the worst-case scenario, such clustering approach cannot hurt too much and ABoB guarantees a standard worst-case regret bound of $O\left(k^{\frac{1}{2}}T^{\frac{1}{2}}\right)$, where $T$ is the number of rounds and $k$ is the number of arms, matching the traditional flat approach. However, under favorable conditions related to the algorithm properties, clusters properties, and certain Lipschitz conditions, the regret bound can be improved to $O\left(k^{\frac{1}{4}}T^{\frac{1}{2}}\right)$. Simulations and experiments on a real storage system demonstrate that ABoB, using standard algorithms like EXP3 and Tsallis-INF, achieves lower regret and faster convergence than the flat method, up to 50% improvement in known previous setups, nonstochastic and stochastic, as well as in our settings.

• The paper introduces the ABoB algorithm that clusters action configurations and applies hierarchical adversarial bandit strategies to reduce regret.
• The method achieves worst-case regret bounds of O(√k T^0.5) and up to 91% improvement in simulation, leveraging clustering in metric spaces.
• Empirical validations on real storage systems confirm ABoB's practical effectiveness in enhancing optimization speed and convergence in dynamic setups.

Adversarial Bandit over Bandits: Hierarchical Bandits for Online Configuration Management

Introduction

"Adversarial Bandit over Bandits: Hierarchical Bandits for Online Configuration Management" focuses on optimizing dynamic parameter configurations in large action spaces, particularly in adversarial environments. The paper introduces the ABoB algorithm which hierarchically clusters action configurations and employs advanced multi-armed bandit (MAB) strategies to efficiently navigate adversarial settings while exploiting local structures.

Algorithmic Framework

The ABoB algorithm introduces a hierarchical adversarial bandit approach that clusters arms into groups and applies adversarial MAB algorithms, such as EXP3, both at the cluster and individual arm levels. This hierarchical framework allows the use of local clustering structures and adapts to dynamic environments. Theoretically, ABoB achieves a regret bound of $O(\sqrt{k} T^{0.5})$ in worst-case scenarios, which matches traditional flat methods. Under favorable conditions, a tighter bound $O(\sqrt[4]{k} T^{0.5})$ is achieved, leveraging the advantages of clustering in metric spaces with Lipschitz properties.

Figure 1: A high-level description of Adversarial Bandit over Bandits (ABoB), illustrating system optimization under dynamic contexts through parameter clustering and adversarial MAB selection.

Theoretical Implications

The paper provides analytical bounds demonstrating that ABoB holds robust worst-case regret performance, akin to non-hierarchical methods like EXP3. Specifically, the clustering does not significantly increase regret while enabling improved bounds when the clusters exhibit Lipschitz continuity. This presents an advantage in structured environments where similar configurations yield close performance outcomes, providing substantial performance gains.

Empirical Validation

Simulation studies and experiments on real-world storage systems validate ABoB's theoretical predictions. In synthetic environments, ABoB displayed up to 91% improvement in regret over traditional MAB approaches, confirming the algorithm's efficiency in both stochastic and adversarial scenarios. The empirical evaluations in a genuine storage system further demonstrate ABoB's practical effectiveness, showcasing its ability to enhance optimization speed and convergence significantly.

Figure 2: Results from real storage systems illustrating cumulative regret and empirical validation of the Lipschitz constant estimation in the arm configurations.

Future Prospects

The ABoB algorithm opens avenues for future research in adaptive clustering techniques and exploring multilevel hierarchical bandit approaches. Potential extensions could involve dynamic partition strategies or enhancing computational efficiency in parallel sampling settings, driving forward optimization capabilities in complex, evolving systems.

Conclusion

By effectively clustering action configurations and applying hierarchical bandit strategies, ABoB offers a promising solution for optimizing performance in adversarial environments with large configuration spaces. Its ability to achieve lower regret and faster convergence affirms its utility in practical applications, paving the way for further advancements in algorithmic development and deployment in dynamic contexts.