Hierarchical-Constrained Bandits

• Hierarchical-constrained bandits are a framework that organizes sequential decisions across multiple levels, each with its own constraints.
• They employ techniques like coarse-to-fine exploration and UCB-based methods to balance reward maximization and constraint adherence.
• The model is applied in areas such as recommendation systems, resource allocation, and control systems, offering provable regret bounds and scalable performance.

A hierarchical-constrained bandit problem is a sequential decision-making framework in which action selection and constraint satisfaction operate over multiple hierarchically structured levels. These problems generalize classical bandits by unfolding actions or policies through a hierarchy (e.g., categories, subcategories, specific options), imposing constraints at multiple levels (e.g., per-category, global resource, safety thresholds), and requiring efficient exploration and exploitation strategies that honor this structure. Hierarchical-constrained bandits naturally arise in applications such as recommendation systems, resource-limited adaptive sampling, personalized interventions, configuration optimization, and online control in complex systems.

1. Hierarchical Modeling in Bandits

Hierarchical-constrained bandits extend standard MABs and contextual bandits by arranging actions—or policy decisions—through multiple nested levels, each potentially equipped with its own set of constraints. In the canonical hierarchical-constrained bandit model (Baheri, 22 Oct 2024), at each decision epoch t, an agent observes context $x_t$ and selects a tuple of actions $\mathbf{a}_t = (a_t^{(1)}, ..., a_t^{(H)})$ where $H$ denotes the number of hierarchy levels. Each decision $a_t^{(h)}$ is subordinate to the higher-level choices $(a_t^{(1)}, ..., a_t^{(h-1)})$ and may influence both the immediate reward and the costs or constraints at its own level.

Formally, at each $t$:

• A context $x_t$ is observed.
• Actions are chosen at each hierarchical level: $a_t^{(1)}, a_t^{(2)}, ..., a_t^{(H)}$.
• The agent receives a reward $r_t$ and possibly multiple cost and feedback signals $\{c_t^{(h)}\}_{h=1}^H$.
• Each $c_t^{(h)}$ must satisfy a level-specific constraint, e.g., $x_t^\top \theta_c^{(h)} \leq \tau^{(h)}$.

Constraints may encode resource limits, safety requirements, budget consumptions, or feasibility regions and, crucially, these may be distinct at each level of the hierarchy, inducing complex dependencies between action selection and the constraint satisfaction process.

2. Key Algorithmic Structures

Algorithms for hierarchical-constrained bandit problems incorporate at least two intertwined ingredients:

• Hierarchically structured exploration and exploitation, such as coarse-to-fine selection (Yue et al., 2012) or tree-based recursive partitioning (Neyshabouri et al., 2016, Hong et al., 2022, Pasteris et al., 2023).
• Multi-level constraint management and estimation, commonly achieved via upper confidence bounding of both rewards and costs (Baheri, 22 Oct 2024), resource allocation schemes (Yang et al., 2020), or virtual queue-based Lyapunov methods (Cayci et al., 2021).

A central paradigm is to propagate uncertainty estimation, optimism, or pessimism (in UCB or Thompson Sampling, for example) throughout the hierarchy:

Algorithmic Layer	Example Approach	Mechanism
Global (coarse)	Subspace or cluster selection	Low-dimensional projection, clustering, tree node
Local (fine/detail)	Action within region/cluster	Arm selection, local index or UCB, intra-group TS
Constraint handling	Layer-wise UCB, Lyapunov control	Per-level confidence intervals, virtual queues

Concretely, the Hierarchical Constrained UCB (HC-UCB) (Baheri, 22 Oct 2024) estimates parameters and confidence radii for both the reward and each cost at every level. At round $t$, the agent solves at each level $h$ (setting $V_t$ the regularized Gram matrix):

$$\hat{\theta}_{r, t} = \arg\min_{\theta} \left[ \lambda \|\theta\|^2 + \sum_{s=1}^{t-1}(r_s - x_s^\top \theta)^2 \right]$$

and, analogously for each constraint parameter $\hat{\theta}_{c, t}^{(h)}$. Confidence ellipsoids are established via self-normalized martingale inequalities:

$$\|\hat{\theta}_{r, t} - \theta_r\|_{V_t} \leq \beta_t(\delta), \quad \|\hat{\theta}_{c, t}^{(h)} - \theta_c^{(h)}\|_{V_t} \leq \beta_t(\delta)$$

Action selection is performed by maximizing the reward UCB subject to all level-wise cost LCBs fitting under their thresholds.

3. Regret Analysis and Theoretical Guarantees

A defining feature of hierarchical-constrained bandit algorithms is their ability to guarantee sublinear cumulative regret while maintaining high-probability satisfaction of multi-level constraints. For HC-UCB (Baheri, 22 Oct 2024), the cumulative regret $R_T$ over $T$ rounds is, with high probability:

$$R_T = O\left( \sqrt{dT\log(\lambda + T/d)} + d\sqrt{T}\log(1/\delta) \right)$$

where $d$ is the dimensionality of the context space and $\lambda$ is the regularization coefficient. Constraint satisfaction at each level $h = 1,...,H$ is enforced such that, with probability at least $1-\delta$, the lower confidence bound for the expected cost at level $h$ is always below the threshold $\tau^{(h)}$.

A minimax lower bound derived in (Baheri, 22 Oct 2024) demonstrates that this rate is nearly optimal, as any (possibly adaptive) algorithm incurs regret at least:

$\Omega(\sqrt{d H T})$

where $H$ is the number of hierarchical levels, indicating an unavoidable scaling with the number of levels in worst-case scenarios.

4. Connections to Other Hierarchical Bandit Models

The hierarchical-constrained bandit framework unifies and generalizes a spectrum of prior hierarchical bandit models:

• Coarse-to-Fine Bandits: The CoFineUCB (Yue et al., 2012) algorithm employs prior structural knowledge to explore first a low-dimensional subspace before escalating to the full space, resulting in regret bounds that interpolate between those for low- and high-dimensional settings.
• Hierarchical Mixture-of-Experts: Partition-based approaches (Neyshabouri et al., 2016) recursively decompose the context/action space (e.g., via trees) and combine local policies or experts using adaptive weighting, enabling scalable regret and computational guarantees.
• Layered UCB and Experts: In multi-layered experts settings (Guo et al., 2022), each expert at a given layer uses a UCB policy, with regret bounds carefully analyzed to avoid linear growth with the number of layers; instead, suitably well-tuned UCB parameters yield logarithmic or sublinear layer dependence.
• Resource-Constrained Hierarchies: HATCH (Yang et al., 2020) separates resource allocation and personalized recommendation, clustering user space to perform dynamic programming for budgeted decision-making with proper regret scaling.
• Lyapunov-based Hierarchical Methods: For constrained settings where resource or safety constraints may be “soft” or stochastic, virtual queue-based Lyapunov optimization (Cayci et al., 2021) achieves $O(\sqrt{K B \log B})$ regret for $K$ arms and budget $B$, with provable constraint satisfaction.

5. Practical Applications and Real-World Implications

Hierarchical-constrained bandit frameworks are especially suited to domains where decisions are structured, constraints are multi-level and context-dependent, and exploration costs or operational safety are paramount:

• Autonomous Systems: In multi-tiered control stacks (e.g., high-level routing vs. low-level actuation), constraints propagate and must be enforced at all abstraction layers.
• Cloud and Resource Allocation: Allocation strategies must satisfy both global (datacenter-level) and local (machine- or rack-specific) power, bandwidth, or workload constraints.
• Personalized Recommendation: Rapid user preference learning is expedited by coarse clustering or factorized models, but fine-grained recommendations must adapt to atypical or boundary cases while budget and fairness constraints are maintained.
• Conversational Recommendation: Structured queries about categories or attributes before recommending items leverage a hierarchical structure to compress regret and lower the user burden (Zuo et al., 2022).
• Online Configuration Management: The ABoB algorithm (Avin et al., 25 May 2025) leverages cluster-based action hierarchies to improve adversarial and stochastic regret in large configuration spaces, with pronounced computational benefits and reduced regret under smoothness assumptions.

By balancing optimism about reward (UCB-based or Bayesian) and conservativeness regarding constraints at each hierarchical level, these algorithms are robust to mis-specification, capable of respecting real-world operational boundaries, and, in practice, reduce exploration cost and improve learning rate over “flat” bandit methods.

6. Open Directions and Further Developments

Several ongoing research areas intersect with hierarchical-constrained bandit problems:

• Deeper Hierarchical Structures: Extension to arbitrary depth, non-tree DAGs, and more complex dependency structures between levels (e.g., cross-level constraints or feedback).
• Adaptive Partitioning: Online learning of partition/hierarchy structure itself from data, merging or splitting regions as new evidence accumulates (Neyshabouri et al., 2016, Pasteris et al., 2023).
• High-Dimensional Contexts: Combining hierarchical bandit structures with efficient nonparametric estimation in metric, manifold, or high-dimensional feature spaces (Pasteris et al., 2023).
• Multi-objective and Pareto Constraints: Joint identification of feasible (e.g., safe) and optimal actions under multi-objective constraints, seeking Pareto frontiers within feasibility sets (Kone et al., 9 Jun 2025).
• Adversarial Environments: Robustness to adversarial reward manipulation, nonstationarity, and strategic context changes (ABoB in (Avin et al., 25 May 2025)).

The rigorous analysis of regret and constraint satisfaction in these complex hierarchical settings is an area of active research, with new algorithmic paradigms emerging to address increasingly structured and constrained decision-making environments.