Hierarchical Multi-Armed Bandits

• Hierarchical multi-armed bandits are sequential decision models that structure arms, contexts, or priors across levels to reduce search complexity and share statistical strength.
• They leverage hierarchical action spaces, Bayesian priors, and contextual partitions to refine exploration strategies and improve convergence rates.
• Applications include intelligent tutoring, online advertising, resource allocation, and LLM-guided optimization, each demonstrating tailored regret bounds and efficiency gains.

Searching arXiv for recent and foundational papers on hierarchical multi-armed bandits and closely related formulations. arXiv search query: "hierarchical multi-armed bandit" A hierarchical multi-armed bandit is a family of sequential decision models in which the arm set, the inference model, the context representation, or the feedback structure is organized across multiple levels rather than treated as a flat collection of independent alternatives. In current arXiv usage, the term encompasses nested educational action spaces in which a tutor first selects a concept and then a problem (Castleman et al., 2024), hierarchical Bayesian bandits whose arm-specific parameters share higher-level priors (Cherkassky et al., 2013), contextual bandits that learn over hierarchical partitions of the context space (Neyshabouri et al., 2016), and tree-structured similarity models in which arms are leaves of a rooted hierarchy (Martin et al., 2022, Eldowa et al., 22 Jun 2026). This suggests that hierarchical bandits are best understood as an umbrella class of bandit models that exploit structure across levels to reduce effective search complexity, share statistical strength, or respect feasibility constraints induced by the domain.

1. Major meanings of hierarchy in bandit models

The modern literature uses “hierarchical” in several technically distinct ways. In some papers the hierarchy is the action space itself; in others it is the prior, the information flow among agents, the causal graph, the similarity structure of the arms, or the constraint system. This diversity is central to the topic because it explains why papers with very different algorithms are all described as hierarchical multi-armed bandits.

Hierarchical locus	Characteristic formulation	Representative source
Nested decision space	A concept MAB chooses a concept, then a problem MAB chooses a problem within that concept	(Castleman et al., 2024)
Hierarchical Bayesian prior	Arm-specific parameters, shared parameters, and higher-level hyperparameters are updated sequentially	(Cherkassky et al., 2013)
Hierarchical context partition	A weighted mixture is learned over region-to-arm mappings represented by a tree or splitting graph	(Neyshabouri et al., 2016)
Context/causal parent over arms	A context variable $S$ sits above arms $X_1,\dots,X_N$, inducing dependence marginally and independence conditionally	(Song et al., 2021)
Leader-follower information structure	A leader acts first and a follower responds after observing the leader’s action	(Kao et al., 2021)
Tree-structured arm similarity	Arms are leaves of a rooted tree and losses are constrained by shared ancestry	(Martin et al., 2022)
Multi-level constraints	A composite action is chosen across $H$ levels with level-wise cost constraints	(Baheri, 2024)

A plausible implication is that “hierarchical multi-armed bandit” does not denote a single canonical formalism. Instead, it denotes a class of bandit models in which some higher-level structure determines how lower-level actions are enumerated, estimated, constrained, or compared.

2. Structural formulations

A direct action-space hierarchy appears in intelligent tutoring. In "Hierarchical Multi-Armed Bandits for the Concurrent Intelligent Tutoring of Concepts and Problems of Varying Difficulty Levels" (Castleman et al., 2024), the hierarchy is two-level: a concept MAB selects a concept from the current section, and a problem MAB specific to the selected concept chooses one problem from that concept’s question bank. The paper emphasizes that a single MAB over all concept-question pairs would be insufficient because the set of feasible questions depends on the chosen concept, and the concepts themselves have prerequisite structure. The feasible set is governed by a Zone of Proximal Development frontier containing currently available activities whose prerequisites are mastered but which themselves are not yet mastered. Rewards are based on student correctness over time, the tutor maintains belief states of mastered or unmastered, and weights are normalized over the ZPD and combined with exploration through

$$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$

The same formulation adds expert-defined problem difficulty $d \in [1,5]$ and a memory-decay component from the Multiscale Context Model.

A different notion of hierarchy is statistical rather than operational. In "Sequential Monte Carlo Bandits" (Cherkassky et al., 2013), parameters are collected in

$$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$

where $\boldsymbol{\beta}$ are arm-specific parameters, $\boldsymbol{\tau}$ are parameters shared across arms, and $\boldsymbol{\phi}$ are higher-level hierarchical hyperparameters. The paper’s key “hierarchical multi-armed bandit” idea is that arms are not treated independently; their parameters can share statistical strength through the prior hierarchy. A representative example is

$\beta_{k1} \sim \mathcal{N}(\nu,\sigma_\beta^2),$

with further hierarchy possible on $X_1,\dots,X_N$0 and $X_1,\dots,X_N$1. Here the hierarchy does not constrain feasible actions; it induces posterior coupling across arms.

A third formulation places the hierarchy in the context space. "An Asymptotically Optimal Contextual Bandit Algorithm Using Hierarchical Structures" (Neyshabouri et al., 2016) studies adversarial contextual bandits by partitioning the context space into regions and learning a weighted mixture over all mappings from regions to arms. The hierarchical structure is carried by binary trees, $X_1,\dots,X_N$2-ary trees, lexicographical splitting graphs, $X_1,\dots,X_N$3-group lexicographical splitting, arbitrary position splitting, and arbitrary splitting. The recursion over node weights $X_1,\dots,X_N$4 and $X_1,\dots,X_N$5 allows the algorithm to compute the exact EXP4-style mixture over a huge expert class without explicitly storing all experts.

Causal hierarchy yields yet another interpretation. "Hierarchical Causal Bandit" (Song et al., 2021) introduces a context variable $X_1,\dots,X_N$6 above the arms, with the factorization

$X_1,\dots,X_N$7

The hierarchy is $X_1,\dots,X_N$8, and the important modeling assumption is that the arms are independent given the context. This creates marginal dependence among arms while retaining a decomposition amenable to regret analysis.

3. Algorithmic families

Hierarchical action selection often preserves a bandit core while changing the object to which exploration is applied. In the tutoring setting, exploration is applied twice: first at the concept level and then at the problem level, with difficulty-aware adaptation at the lower level. The modified MAPLE-style rule increases multipliers for harder questions after a correct response and decreases them after an incorrect one, while performing the opposite update for easier questions (Castleman et al., 2024). The hierarchy is both structural and operational because the lower-level MAB is instantiated only after a concept has been chosen.

Hierarchical Bayesian bandits typically use Thompson-style probability matching. In "Sequential Monte Carlo Bandits" (Cherkassky et al., 2013), posterior inference is performed with weighted particles

$X_1,\dots,X_N$9

and arm selection samples a parameter particle and chooses

$H$0

The hierarchy appears in the prior, while sequential Monte Carlo replaces repeated MCMC and supports contextual, static, and restless variants.

Hierarchical contextual partitioning leads to expert-mixture algorithms rather than arm-by-arm posterior updates. The HSB algorithm computes node-level quantities recursively and sets the arm probability by

$H$1

Its distinctive feature is exact aggregation over an exponentially large class of region-to-arm mappings represented compactly by a hierarchy (Neyshabouri et al., 2016).

Hierarchical information structures produce decentralized UCB variants. In the two-agent model of "Decentralized Cooperative Reinforcement Learning with Hierarchical Information Structure" (Kao et al., 2021), the leader chooses $H$2, the follower observes $H$3 and chooses $H$4, and the leader cannot observe $H$5. The leader uses a UCB-like rule whose bonus compensates for two uncertainty sources: statistical noise and the follower’s learning regret. The resulting algorithm is coordination-free because the leader never needs to infer the follower’s exact action.

Tree-structured similarity has generated layered FTRL and nested exponential-weights methods. "Nested bandits" (Martin et al., 2022) proposes a nested exponential weights algorithm that performs layered exploration of the learner’s set of alternatives based on a nested, step-by-step selection method. "Leveraging Similarities in Multi-Armed Bandits" (Eldowa et al., 22 Jun 2026) uses tree-aware regularization on subtree masses and multi-point feedback estimators based on loss differences such as $H$6. These methods are designed to exploit similarity encoded by shared ancestry rather than only observed reward means.

A distinct algorithmic line uses hierarchy to restore incentive compatibility. "Multi-armed Bandit Algorithm against Strategic Replication" (Shin et al., 2021) proposes Hierarchical UCB, which first selects an agent using an agent-level UCB index and then selects an arm within the chosen agent using arm-level UCB. Robust Hierarchical UCB adds stronger exploration over agents and random subsampling of arms to remain sublinear even with irrational replicators.

4. Regret, optimality, and lower bounds

Theoretical guarantees in hierarchical bandits are heterogeneous because the hierarchy may sit in the action space, the prior, the context space, the information structure, or the feedback model.

Setting	Representative guarantee	Source
Hierarchical contextual partitioning	$H$7 with optimized learning rate	(Neyshabouri et al., 2016)
Hierarchical causal bandit	$H$8 and a lower bound $H$9	(Song et al., 2021)
Leader-follower hierarchical bandit	Gap-independent regret $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$0 and gap-dependent $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$1	(Kao et al., 2021)
Hierarchical constrained bandit	$$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$2, with minimax lower bound $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$3	(Baheri, 2024)
Replication-proof hierarchy	H-UCB has $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$4-regret under equilibrium; RH-UCB has $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$5 and problem-independent $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$6	(Shin et al., 2021)
Tree similarity under limited feedback	One-point feedback has an $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$7 lower bound; richer feedback yields $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$8-type bounds	(Eldowa et al., 22 Jun 2026)

These results support two recurring conclusions. First, hierarchy can reduce effective complexity: HSB achieves logarithmic dependence on partition complexity rather than the naive $$w_{a,n} = \frac{w_a}{\sum_{a \in ZPD} w_a},\qquad p_a = w_{a,n}(1-\gamma) + \frac{\gamma}{|ZPD|}.$$9 enumeration cost (Neyshabouri et al., 2016), and tree-aware algorithms replace the raw number of actions by an effective number of actions $d \in [1,5]$0 under richer feedback (Eldowa et al., 22 Jun 2026). Second, hierarchy does not automatically improve regret: in hierarchical causal bandits the lower bound depends on the worst context-conditioned subproblem rather than the marginal mixture (Song et al., 2021), and under one-point feedback tree-induced similarity cannot in general be exploited (Eldowa et al., 22 Jun 2026).

A related theme is near-optimality under structured decentralization. The leader-follower hierarchical bandit matches the fully coordinated single-agent rate up to polylogarithmic factors in the gap-independent case (Kao et al., 2021), while HC-UCB is presented as near-optimal because its upper bound is paired with a minimax lower bound scaling as $d \in [1,5]$1 (Baheri, 2024).

5. Applications and empirical instantiations

Education is a prominent application because curricula are naturally hierarchical. The tutoring system in (Castleman et al., 2024) organizes content into sections, concepts, and problems, and evaluates random question sequencing, hierarchical MAB without difficulty, and hierarchical MAB with difficulty using simulated groups of 500 students. The paper reports that the random baseline initially looks better, likely because randomization exposes students to more varied concepts early, but after enough questions both hierarchical MAB variants outperform random sequencing in asymptotic mastery. It also reports that the difficulty-adaptive hierarchical MAB slightly outperforms the difficulty-agnostic one.

Online advertising provided an early empirical demonstration of hierarchical Bayesian bandits. In (Cherkassky et al., 2013), a dynamic SMC bandit is evaluated on 4 advertisements with click/no-click rewards, geographic location and time features, and an unbiased offline evaluator. The reported average reward per iteration is $d \in [1,5]$2 for dynamic SMC, compared with $d \in [1,5]$3 for random, an improvement of about $d \in [1,5]$4 over random with $d \in [1,5]$5. The paper interprets this as evidence that ad click probabilities are nonstationary.

Extreme-arm recommendation and classification motivate arm hierarchies for computational reduction. "Top-$d \in [1,5]$6 eXtreme Contextual Bandits with Arm Hierarchy" (Sen et al., 2021) uses a balanced $d \in [1,5]$7-ary tree and beam search to reduce a problem with $d \in [1,5]$8 arms to $d \in [1,5]$9 effective arms per context. On the amazon-3m dataset, with about 3 million arms, the hierarchy-based inference takes 7.9 ms per context for beam size $$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$0, which is roughly a 100× improvement over evaluating all arms naively.

Communication and control systems supply further hierarchical settings. In multi-LEO resource allocation, mmRAL models each satellite as a macro-agent and power, beam, and channel allocation as micro-agents, using only throughput feedback rather than CSI (Shen et al., 2023). The paper reports that $$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$1 converges best and that throughput reaches about 7.9 Gbps. In autonomous driving, BiM-ACPPO uses a bilevel curriculum bandit in which the first curriculum level is the number of surrounding vehicles and the second is maneuver type; the paper reports the highest success rate in all test settings and a 97% success rate in a few-shot overtaking scenario (Peng et al., 6 Feb 2025).

Recent systems work has also adapted hierarchical bandits to LLM-guided optimization. "KernelBand" (Ran et al., 24 Nov 2025) treats kernel selection and optimization-strategy selection as a hierarchical MAB, clusters kernels by runtime behavior with $$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$2 clusters, and combines UCB with hardware profiling priors. On A100 with DeepSeek-V3.2, the reported best-of-iteration speedup reaches 22.30× at iteration 9, compared to GEAK’s 3.22×.

6. Conceptual issues, limitations, and directions

A common misconception is that any structured action set can be flattened without loss. Several papers dispute this directly. The tutoring paper states that a single MAB over all concept-question pairs would be insufficient because the feasible question set depends on the chosen concept and the concepts themselves have prerequisite structure (Castleman et al., 2024). The nested-bandit literature similarly argues that flat Hedge- or EXP3-style exploration can spend excessive amounts of time exploring irrelevant alternatives with similar, suboptimal costs, producing the online-learning analogue of the red bus / blue bus paradox (Martin et al., 2022).

Another misconception is that hierarchical similarity is automatically exploitable under ordinary bandit feedback. "Leveraging Similarities in Multi-Armed Bandits" proves an impossibility result showing that standard one-point bandit feedback cannot generally exploit range or tree-induced similarity, even under very strong similarity constraints (Eldowa et al., 22 Jun 2026). This sharply separates hierarchy in the model from hierarchy that is algorithmically usable.

Strategic behavior creates a further complication. In arm-registration settings, standard UCB1 is replication-prone and can suffer linear regret in time $$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$3, because exploration is allocated to registered arms rather than to agents (Shin et al., 2021). Hierarchical selection at the agent level can therefore serve not only statistical efficiency but also mechanism-design objectives.

Empirical limitations are likewise domain-specific. The intelligent tutoring results are based on simulation, not a real student trial, and depend on Bayesian Knowledge Tracing, ASSISTments-derived mappings, binary correctness, fixed difficulty scores, and one-time correctness removal (Castleman et al., 2024). KernelBand’s regret guarantee includes approximation terms $$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$4, showing explicitly that clustering and compatibility-model error can degrade performance (Ran et al., 24 Nov 2025). In multi-bandit transfer, empirical Bayes approaches rely on estimating covariance structure across bandit instances rather than treating it as known (Jiang et al., 30 Oct 2025), while metadata-based multi-task bandits become closer to meta-TS and individual-TS when metadata is less informative (Wan et al., 2021).

Current directions therefore follow directly from the existing limitations. The tutoring work suggests real-world student trials, online self-updating of problem difficulty, exploration of different history-window sizes $$\boldsymbol{\theta} = (\boldsymbol{\beta}, \boldsymbol{\tau}, \boldsymbol{\phi}),$$5, and adaptive material redirects for underperforming students (Castleman et al., 2024). More broadly, the recent literature suggests that future progress in hierarchical multi-armed bandits will depend less on the generic presence of a hierarchy than on matching the hierarchy to an appropriate inferential mechanism, feedback model, and decision interface.