Learning in Markovian bandits with non-observable states and constrained decision epochs

Abstract: This paper studies the problem of regret minimization in Markovian bandits with \emph{non-observable states} and possibly \emph{constrained} decision epochs. The focus is restricted to a ``pure'' regret benchmark, that compares the performance of the learning algorithm to the best \emph{pure policy} which -- akin to optimal policies of stochastic bandits -- picks the optimal arm from start to finish without ever switching. We introduce a generalization of rested Markovian bandits, \emph{self-degrading Markovian bandits}, for which pure policies are always asymptotically optimal.We show that without prior knowledge on the underlying bandit, the regret of algorithms that switch arms rarely necessarily scales super-logarithmically for every bandit, i.e., as $ω(\log(T))$, where $T$ is the learning horizon. Despite the unreachability of the logarithmic regime, we design UCB-NOM, an optimistic algorithm inspired by UCB, of which the regret is nearly logarithmic. Lastly, we show that given prior knowledge on the Markovian bandit in the form of a bound on the bias functions of its arm, a proper instantiation of UCB-NOM achieves $O(\log(T))$ regret. We further show that this prior knowledge allows for a $O(\sqrt{T \log(T)})$ worst-case regret bound for UCB-NOM. Notably, our regret bounds do not depend on the number of states of the underlying Markov chains. Our findings suggest that the non-observability of states is a mild inconvenience in self-degrading Markovian bandits.

• The paper introduces self-degrading Markovian bandits where a pure policy is asymptotically optimal for regret minimization.
• It establishes super-logarithmic regret lower bounds for rarely-switching algorithms and proposes an optimistic UCB-inspired algorithm achieving nearly-logarithmic regret.
• The study demonstrates that limited prior structural knowledge dramatically improves learning rates, bridging gaps with classical observable bandit models.

Learning in Markovian Bandits with Non-Observable States and Constrained Decision Epochs

Problem Setting and Motivation

This work addresses the multi-armed bandit (MAB) problem with Markovian arms, where the underlying states of each arm evolve according to a Markov process, but crucially, these states are not observable to the learner. Additionally, the opportunity to make decisions may be subject to external constraints (“constrained decision epochs”). The focus is on regret minimization relative to a “pure” policy baseline: a strategy that selects a fixed arm at the outset and never switches.

Previous literature has considered both rested and restless Markovian bandits [gittins2011multi, niño_markovian_2023], and partially observable MDPs [krishnamurthy2016partially], but the interplay between non-observability, switching constraints, and regret minimization yields new challenges. This setting is motivated by various practical applications (e.g., opportunistic spectrum access, adaptive polling) where state information is unavailable and decision opportunities may be sporadically constrained.

Self-Degrading Markovian Bandits and Pure Policy Benchmarks

A central conceptual contribution of the paper is the introduction of self-degrading Markovian bandits: a subclass where the optimal pure strategy is always asymptotically optimal for regret minimization. In these bandits, any arm that is not activated experiences state transitions that cannot increase its expected reward—in effect, making commitment to a single arm both robust and theoretically optimal in the long term.

This observation justifies the use of pure policy regret, in contrast to more ambitious benchmarks. The work identifies the significance of this pure-policy regime by contrasting it with classical stochastic bandit settings, where adaptive switching amongst arms can guarantee logarithmic regret uniformly over instances.

Regret Lower Bounds under Non-Observability

A major theoretical result is the establishment of super-logarithmic lower bounds on regret for rarely-switching algorithms when state information is completely non-observable and no prior structural information about the Markov chains is available. Specifically, for any algorithm that switches arms infrequently, the regret cannot scale as $\mathcal{O}(\log(T))$, but is necessarily $\omega(\log(T))$, where $T$ is the time horizon. This holds robustly across all Markovian bandit instances with non-observable states and marks a strong divergence from the classic i.i.d. setting [lai1985asymptotically, anantharam_1987_asymptotically] and from results established in more tractable observable-state Markovian models [auer_logarithmic_2006]. The result highlights a strict information-limited barrier induced by non-observability.

Optimistic Algorithm and Regret Guarantees

Despite the impossibility of uniform logarithmic regret without prior knowledge, the authors introduce a UCB-inspired optimistic algorithm (\algname{}) capable of achieving nearly-logarithmic regret in this challenging setting. The distinctive innovation is that the regret guarantees are independent of the number of underlying Markov states—a sharply non-trivial fact given the latent state space can be arbitrarily large, and most existing analyses scale poorly with the state cardinality [wang2020restless, ortner_regret_2012].

Further, the paper establishes that, given prior knowledge in the form of an (assumed) upper bound on the bias span of each arm’s Markov chain, one can recover the canonical $\mathcal{O}(\log(T))$ instance-dependent regret rate, and a worst-case bound of $\mathcal{O}(\sqrt{T\log(T)})$. These claims are rigorously established and indicate that limited prior information dramatically improves achievable rates in this domain.

Implications and Future Directions

The theoretical results suggest that in self-degrading Markovian bandits, non-observability of the state is only a mild hindrance for regret minimization with pure policy benchmarks, provided appropriate algorithmic techniques are leveraged and/or mild structural knowledge is available. This stands in contrast to more general POMDP settings, where latent state complexity often fundamentally limits learning rates [jin2020sample, russo_achieving_2025].

Several future research directions are noted:

• Known state space: The case where the state space is known may introduce new identification and planning subtleties, particularly in the presence of large or structured latent state spaces.
• Clustered states and partial observability: Allowing for states organized into clusters, or structure in the reward observation model, may recover tractability in broader regimes and enable new algorithmic strategies [liu2022partially].
• Generalizations to restless settings: Allowing for both activated and non-activated arms to yield rewards further aligns with practical queueing and network models [papadimitriou1999complexity, tekin_online_2011], opening questions about trade-offs between observability, arm dynamics, and achievable regret.

Conclusion

This paper rigorously delineates the fundamental learning complexity in Markovian bandits with non-observable states and constrained decision epochs. By identifying self-degrading processes and the utility of pure policy regret, and by providing both lower bounds and nearly-matching achievable guarantees (independent of state space cardinality), it clarifies the information-theoretic and algorithmic possibilities of this regime. The findings chart a clear path for future work in partially observed, non-i.i.d. bandit learning and highlight regimes where classical regret-optimal strategies remain viable.