Hierarchical Multi-Armed Bandits

• Hierarchical multi-armed bandits are decision frameworks that structure actions in tree-like or multi-level hierarchies to efficiently explore vast, structured action spaces.
• They apply algorithmic techniques such as layered UCB, hierarchical Thompson sampling, and beam search to share information and achieve low regret.
• These methods drive scalable, safe learning in applications like recommendation systems, curriculum learning in reinforcement learning, and intelligent tutoring.

The hierarchical multi-armed bandit (MAB) problem generalizes standard bandit and contextual bandit formulations to settings in which the action or decision space is structured according to a hierarchy. Such hierarchies may encode tree-structured relationships among arms, nested task decompositions, multi-level constraints, or latent similarity among tasks. This class of problems enables efficient exploration and learning in domains characterized by combinatorially large, highly structured action spaces or where cross-task, cross-arm, or multi-level information sharing is fundamental. Rigorous algorithmic and theoretical developments in hierarchical MAB are motivated by diverse applications including large-scale recommendation, curriculum learning in reinforcement learning, hierarchical online learning, intelligent tutoring, distributed resource allocation, and hardware-aware optimization.

1. Formal Definitions and Canonical Models

In hierarchical MAB, the arms are organized according to a tree, layered, or otherwise nested structure. The precise specification varies across research directions:

• In tree-structured bandits, such as the deep Bayesian hierarchy in multi-label or contextual settings, the arms correspond to leaves of a rooted tree, with internal nodes aggregating sets of child arms (Hong et al., 2022, Hong et al., 2022, Guo et al., 2022, Guo et al., 2022).
• In multi-level curriculum or bilevel resource settings, arms may correspond to action pairs across two or more levels, such as (cluster, sub-arm) (Peng et al., 6 Feb 2025, Peng et al., 6 Feb 2025), or macro/micro agents (Shen et al., 2023, Shen et al., 2023).
• In meta-bandit or multi-task frameworks, each “arm” at a higher level selects a task-specific lower-level bandit agent, whose arms are then selected at the next level (Wan et al., 2021, Wan et al., 2021, Hong et al., 2022, Hong et al., 2022).

Formally, a hierarchical MAB process typically proceeds as follows:

1. At each round, a context (possibly multi-dimensional) is observed.
2. The agent descends the hierarchy, sequentially selecting nodes/arms at each level, sometimes subject to level-specific constraints or routing classifiers (Sen et al., 2021, Sen et al., 2021, Baheri, 22 Oct 2024, Baheri, 22 Oct 2024).
3. Upon traversing to a leaf or terminal node, an action is instantiated and a (possibly vectorial) reward is realized.
4. Feedback may be received for all, some, or only the chosen arms, depending on application and feedback model.

Key technical features include additivity or independence assumptions across arms, context-dependent routing or arm grouping, and sharing of statistical information across related arms via hierarchical priors or models.

2. Algorithmic Approaches

Multiple algorithmic frameworks have been developed to address the unique challenges of hierarchical MAB:

• Hierarchical UCB (Upper Confidence Bound): In competitive environments with replication or adversarial structure, a layered version of UCB is used, where an upper-level expert (agent, cluster, or controller) chooses among lower-level experts, which themselves may run UCB strategies on underlying arms. For example, in strategic registration, a two-level H-UCB deters replication by allocating O(ln T) exploration per agent, regardless of arm multiplicity (Shin et al., 2021, Shin et al., 2021).
• Hierarchical Thompson Sampling and Posterior Propagation: For settings with hierarchical Bayesian priors (e.g., correlated arm rewards), algorithms such as HierTS perform fast, exact hierarchical posterior updates, sampling parameters at each node and leveraging cross-arm dependencies for maximum information sharing (Hong et al., 2022, Hong et al., 2022).
• Beam Search and Contextual Reduction: To address computational bottlenecks in “extreme” high-arm cardinality settings, such as multi-label ranking, hierarchical bandit algorithms employ tree-based beam search to reduce the effective arm set to a logarithmic subset per context, enabling tractable regret-optimal selection (Sen et al., 2021, Sen et al., 2021).
• Path-Planning and Monte Carlo Tree Search: For multivariate bandits with large combinatoric layouts, arm selection is decomposed into sequence-of-decisions along a graph or tree path, combining TS or UCB within this latent structure for scalable search (Nie et al., 2019, Nie et al., 2019).
• Hierarchical Constraint Management: In scenarios with multi-level cost or safety constraints, algorithms such as HC-UCB conduct level-wise optimistic selection while maintaining constraint satisfaction through conservative lower-confidence bounds at each level (Baheri, 22 Oct 2024, Baheri, 22 Oct 2024).
• Meta-Bandit, Multi-Task, and Off-Policy Hierarchies: Hierarchical Bayesian models, both parametric and nonparametric, are used to tie together multi-task bandits, enabling cross-task learning and efficient exploration via hierarchical prior induction and Thompson sampling or “pessimism under uncertainty” principles (Wan et al., 2021, Wan et al., 2021, Hong et al., 2022, Hong et al., 2022).
• Curriculum and Bilevel Bandits: Hierarchical bandits allocate training curricula or scenarios for RL agents, optimizing over clusters (task classes) and sub-tasks (arms), often with Exp3.S-style adversarial bandit updates (Peng et al., 6 Feb 2025, Peng et al., 6 Feb 2025).

3. Regret Analysis and Theoretical Guarantees

Regret and suboptimality analyses for hierarchical MAB algorithms reveal intricate dependencies on the structure and identification of hierarchy:

• In flat bandits with replication or adversarially constructed hierarchies, naively layering can result in regret growing linearly with the number of levels, R, or with the total number of arms (Guo et al., 2022, Guo et al., 2022). However, with coordinated exploration, e.g., unique low-exploration-parameter UCB agents per layer, regret can be reduced to O(ln T) independent of R, matching the flat bandit rate (Shin et al., 2021, Shin et al., 2021, Guo et al., 2022, Guo et al., 2022).
• In deep Bayesian hierarchies, regret admits a decomposition along the hierarchical prior, with multiplicative improvements (e.g., O(√(log_b K)) over flat TS) depending on tree balance and prior widths. Regret scales as O(√{n |V|}) (n: rounds, |V|: total nodes), with statistical efficiency gained by sharing information across related actions (Hong et al., 2022, Hong et al., 2022).
• In extremely large arm spaces, hierarchical partitioning algorithms achieve exponential reduction in sample and computational complexity. For the top-k contextual bandit, beam search over a tree of arms yields a regret bound of O(k√[(log A − k + 1) T log (|𝔽|T)]) with O(log A) per-round computational complexity, compared to O(A√T) in flat settings (Sen et al., 2021, Sen et al., 2021).
• In hierarchical multi-task settings, hierarchical Thompson sampling achieves multi-task regret rates of O(√{MdKT}) or better, where d is feature dimension, K arms per task, and M tasks, thus outperforming independent task learning when M ≫ d (Wan et al., 2021, Wan et al., 2021). For off-policy contextual bandits, hierarchical models yield per-task suboptimality gains, with the hyper-parameter uncertainty diminishing as the number of tasks increases (Hong et al., 2022, Hong et al., 2022).
• In hierarchical settings with constrained optimization, HC-UCB guarantees sublinear regret O(√{d T log (1+ T/(λd))}), high-probability constraint satisfaction at all levels, and near-minimax lower bounds Ω(√{d H T}), with d the dimensionality and H the number of levels (Baheri, 22 Oct 2024, Baheri, 22 Oct 2024).

4. Hierarchy-Induced Computational Efficiency

Structural exploitation in the hierarchy frequently leads to dramatic computational gains:

• Hierarchical bandit models with tree partitions enable replacing exponential-in-arm-number updates with O(log A) or O(M log N) per-round complexity (Neyshabouri et al., 2016, Neyshabouri et al., 2016, Sen et al., 2021, Sen et al., 2021).
• In LLM-based kernel optimization, KernelBand utilizes behavioural clustering to compress the arm space and a hierarchical UCB with hardware-awareness; empirical results show the approach scales to very large search spaces, outperforming beam-search-based multi-agent methods and maintaining increasing returns as computational budgets grow (Ran et al., 24 Nov 2025, Ran et al., 24 Nov 2025).
• Hierarchical multi-agent bandits for LEO satellite resource allocation decompose a combinatorial macro-micro resource allocation problem into two levels of MAB, yielding rapid throughput convergence and resilience without requiring explicit channel-state information (Shen et al., 2023, Shen et al., 2023).
• Intelligent tutoring systems implement two-level MABs (concept → problem), incorporating content difficulty via per-arm scaling and dynamic belief state updates, yielding higher student mastery rates than flat or random sequencing (Castleman et al., 10 Aug 2024, Castleman et al., 10 Aug 2024).

5. Application Domains

Hierarchical multi-armed bandit methodologies have been foundational in several domains:

Domain	Hierarchy Type	Representative Citation
Large-scale recommendation/ranking	Tree-based arm groupings	(Sen et al., 2021, Hong et al., 2022)
RL curriculum learning	Bilevel curriculum scheduling	(Peng et al., 6 Feb 2025)
Multi-agent resource allocation	Macro–micro agent hierarchy	(Shen et al., 2023)
Automated code optimization	Kernel→strategy (clustering)	(Ran et al., 24 Nov 2025)
Multi-task/Meta learning	Task-level graphical hierarchy	(Wan et al., 2021, Hong et al., 2022)
Constrained online learning	Multi-level constraints	(Baheri, 22 Oct 2024)
Intelligent Tutoring	Concept→problem/skill tree	(Castleman et al., 10 Aug 2024)

The practical significance of these structures lies in their ability to encode inductive biases, mirror the combinatorial or logical constraints of the application, and exploit cross-group or cross-task knowledge for both computational and statistical gains.

6. Strategic, Adversarial, and Robust Hierarchies

A nuanced aspect of hierarchical MAB is their behaviour under adversarial, strategic, or partially observed settings:

• In presence of strategic agents who may replicate arms to exploit flat bandit exploration, layered-banded algorithms (H-UCB, RH-UCB) enforce equilibrium and replication resistance via top-layer agent selection and inner-layer arm exploration (Shin et al., 2021, Shin et al., 2021).
• In hierarchical expert settings, the naive composition of multiple layers can multiplicatively inflate regret, unless coordinated exploration—such as unique “lean” experts per layer with fast exploration—is guaranteed (Guo et al., 2022, Guo et al., 2022).
• For constrained or safe hierarchical exploration (e.g., robotics, autonomous driving), level-wise UCB with layer-specific lower-confidence bounds ensures that exploration does not violate safety or other operational constraints (Baheri, 22 Oct 2024, Baheri, 22 Oct 2024).
• Curriculum bandit scheduling for RL (BiMAB) employs parallel target networks and Exp3.S updates to dynamically shift learning towards more informative but harder tasks, increasing success rates and generalization across unseen scenarios (Peng et al., 6 Feb 2025, Peng et al., 6 Feb 2025).

7. Open Issues and Extensions

Outstanding directions in hierarchical MAB research include lower bounds under various structural, adversarial, or regret-optimality constraints; scalable inference in non-Gaussian nonparametric hierarchies; efficient contextualization in deep or arbitrary graphs; dynamic/task-adaptive hierarchy design; and optimal hierarchy width/depth trade-offs for specific application domains (Neyshabouri et al., 2016, Hong et al., 2022, Guo et al., 2022). A plausible implication is that future research will focus on meta-learned or adaptive hierarchies, theory for bandit-structured multi-agent coordination under communication bottlenecks, and robust algorithms beyond the realizable setting.

---

References

• "Top-$k$ eXtreme Contextual Bandits with Arm Hierarchy" (Sen et al., 2021)
• "Bilevel Multi-Armed Bandit-Based Hierarchical Reinforcement Learning for Interaction-Aware Self-Driving..." (Peng et al., 6 Feb 2025)
• "Multi-armed Bandit Algorithm against Strategic Replication" (Shin et al., 2021)
• "An Asymptotically Optimal Contextual Bandit Algorithm Using Hierarchical Structures" (Neyshabouri et al., 2016)
• "Efficient Multivariate Bandit Algorithm with Path Planning" (Nie et al., 2019)
• "Metadata-based Multi-Task Bandits with Bayesian Hierarchical Models" (Wan et al., 2021)
• "Hierarchical Upper Confidence Bounds for Constrained Online Learning" (Baheri, 22 Oct 2024)
• "Deep Hierarchy in Bandits" (Hong et al., 2022)
• "Hierarchical Multi-Armed Bandits for the Concurrent Intelligent Tutoring of Concepts and Problems..." (Castleman et al., 10 Aug 2024)
• "Hierarchical Multi-Agent Multi-Armed Bandit for Resource Allocation..." (Shen et al., 2023)
• "KernelBand: Boosting LLM-based Kernel Optimization with a Hierarchical and Hardware-aware Multi-armed Bandit" (Ran et al., 24 Nov 2025)
• "Multi-Task Off-Policy Learning from Bandit Feedback" (Hong et al., 2022)
• "Regret Analysis for Hierarchical Experts Bandit Problem" (Guo et al., 2022)