Self-Degrading Markovian Bandits

• Self-degrading Markovian bandits are models where each arm’s optimal long-run performance is achieved by committing exclusively, as any deviation degrades its reward.
• The framework extends classical rested bandits by incorporating hidden state dynamics and constrained decision epochs that highlight performance degradation when switching.
• Algorithmic advances, such as UCB-NOM, utilize confidence bounds tailored to Markovian dependencies to achieve nearly logarithmic regret without explicit state observability.

Searching arXiv for the most relevant papers on self-degrading Markovian bandits and closely related RMAB models. {"query":"self-degrading Markovian bandits latent Markovian environment MARBLE Whittle index hidden states arXiv","max_results":10} Self-degrading Markovian bandits are a generalization of rested Markovian bandits in which each arm is an ergodic Markov reward process with non-observable states and possibly constrained decision epochs, and in which pure policies are always asymptotically optimal (Hira et al., 25 Jun 2026). In this class, any deviation from the pure policy can only hurt the asymptotic reward of an arm: the best long-run way to use arm $a$ is to commit to it permanently, and mixing it with other arms cannot improve its asymptotic performance (Hira et al., 25 Jun 2026). The concept belongs to the broader landscape of Markovian restless bandits, hidden-state bandits, and latent-regime RMABs, where degradation may arise endogenously through the arm’s own state dynamics or exogenously through an unobserved environment (Amiri et al., 12 Nov 2025).

1. Formal model and defining property

In the formulation introduced for hidden-state learning with constrained decision epochs, there is a finite set of arms $\mathcal{A}=\{1,\dots,K\}$. Each arm $a$ has a state space $\mathcal{S}_a$, a transition kernel

$P_a(s' \mid s,u),$

where $u\in\{0,1\}$ is the activation indicator, and a reward function $r_a:\mathcal{S}_a\to[0,1]$. The learner does not observe the Markov states. At decision epoch $n$, the observation history is

$$\mathcal{F}_n \equiv \sigma\big((I_1,R_1),\dots,(I_n,R_n)\big),$$

where $I_n$ is the selected arm and $\mathcal{A}=\{1,\dots,K\}$0 the realized reward. The model also allows constrained decision epochs: if $\mathcal{A}=\{1,\dots,K\}$1 denotes the sequence of decision times, then at time $\mathcal{A}=\{1,\dots,K\}$2 the learner selects $\mathcal{A}=\{1,\dots,K\}$3, which is activated over $\mathcal{A}=\{1,\dots,K\}$4 (Hira et al., 25 Jun 2026).

The defining property of a self-degrading arm is that, for any policy $\mathcal{A}=\{1,\dots,K\}$5, the time-average reward obtained from arm $\mathcal{A}=\{1,\dots,K\}$6 whenever it is pulled is asymptotically bounded above by $\mathcal{A}=\{1,\dots,K\}$7, where $\mathcal{A}=\{1,\dots,K\}$8 is the asymptotic average reward achieved by the pure policy that always plays arm $\mathcal{A}=\{1,\dots,K\}$9. Equivalently, the arm is its own best companion: the best way to use arm $a$0 is to commit to it permanently; leaving it idle, switching away, or returning later cannot improve its long-run performance (Hira et al., 25 Jun 2026). This generalizes classical rested Markovian bandits, where inactive arms freeze, by allowing non-activation dynamics that can move the state distribution to worse regions.

A related hidden-state template appears in the binary-state availability-constrained model of “Multi-armed Bandits with Constrained Arms and Hidden States,” where each arm has hidden state $a$1, observed availability $a$2, and belief state $a$3 with $a$4. In that setting, the arm-level control problem is a belief-MDP, and the optimal policy is a threshold policy under the stated monotonicity conditions (Mehta et al., 2017). For self-degrading models, this binary hidden-state construction is a concrete special case in which “good” and “bad” quality levels are inferred through rewards rather than directly observed.

2. Pure policies, asymptotic optimality, and regret

For each arm $a$5, define the asymptotic average reward under the pure policy that always selects $a$6 as

$a$7

Let

$a$8

A central theorem for self-degrading Markovian bandits states that the optimal average reward over all policies equals $a$9, and it is attained by any pure policy that always pulls an arm maximizing $\mathcal{S}_a$0. No policy that mixes arms can achieve an asymptotic average reward strictly greater than $\mathcal{S}_a$1 (Hira et al., 25 Jun 2026).

This result is structurally important because the “pure” benchmark is not a relaxation in this subclass. The regret benchmark is

$\mathcal{S}_a$2

In ordinary restless bandits, benchmarking against the best static arm would generally be weaker than benchmarking against the optimal dynamic policy. In self-degrading Markovian bandits, by contrast, the pure benchmark coincides with the optimal average-reward benchmark because mixing cannot improve long-run performance (Hira et al., 25 Jun 2026).

The lower-bound theory is correspondingly distinctive. Without prior knowledge on the underlying bandit, the regret of algorithms that switch arms rarely necessarily scales super-logarithmically for every bandit, i.e.

$\mathcal{S}_a$3

where $\mathcal{S}_a$4 is the learning horizon. Thus strict logarithmic regret is unattainable for rarely switching algorithms unless additional structural knowledge is supplied (Hira et al., 25 Jun 2026). This separates self-degrading hidden-state learning from classical stochastic bandits, in which logarithmic regret is the canonical regime.

3. Hidden-state learning and regret guarantees

The main positive algorithmic result for the self-degrading subclass is UCB-NOM, an optimistic algorithm inspired by UCB and tailored to Markovian rewards, non-observable states, and constrained decision epochs (Hira et al., 25 Jun 2026). For each arm $\mathcal{S}_a$5, the algorithm tracks the empirical mean

$\mathcal{S}_a$6

and constructs an optimistic index of the form

$\mathcal{S}_a$7

where the confidence radius accounts for Markovian dependence through the bias function $\mathcal{S}_a$8, defined by

$\mathcal{S}_a$9

Its span is

$P_a(s' \mid s,u),$0

Without prior knowledge of the bias span, UCB-NOM achieves nearly logarithmic regret. With prior knowledge in the form of a bound $P_a(s' \mid s,u),$1, a proper instantiation of UCB-NOM achieves $P_a(s' \mid s,u),$2 instance-dependent regret and $P_a(s' \mid s,u),$3 worst-case regret (Hira et al., 25 Jun 2026). A notable feature of these bounds is that they do not depend on the number of states of the underlying Markov chains. The paper accordingly concludes that the non-observability of states is a mild inconvenience in self-degrading Markovian bandits (Hira et al., 25 Jun 2026).

The concentration tools underlying Markovian bandit learning predate this formulation. For finite-state rested Markovian arms, “A Hoeffding Inequality for Finite State Markov Chains and its Applications to Markovian Bandits” proves a Hoeffding inequality controlled by the maximum hitting time

$P_a(s' \mid s,u),$4

and uses it to analyze a Markovian $P_a(s' \mid s,u),$5-UCB algorithm with $P_a(s' \mid s,u),$6 regret, with constants scaling with $P_a(s' \mid s,u),$7 (Moulos, 2020). That analysis applies directly only when each arm can be represented as a finite, irreducible, time-homogeneous Markov chain, but it provides a methodological baseline for the hidden-state concentration arguments used in later work.

4. Restless formulations, threshold structure, and indexability

Self-degrading Markovian bandits sit naturally inside the restless bandit formalism. In the standard survey formulation, a rested bandit has passive kernel $P_a(s' \mid s,u),$8, whereas a restless bandit allows both active and passive dynamics

$P_a(s' \mid s,u),$9

under a resource constraint such as

$u\in\{0,1\}$0

This is precisely the modeling move needed for degradation with use, aging while idle, or repair under activation (Niño-Mora, 19 Jan 2026).

Whittle’s relaxation replaces the hard per-period activation constraint by an average constraint and decomposes the RMAB into single-arm subsidy problems. For a discounted single arm, the Bellman equation takes the form

$u\in\{0,1\}$1

An arm is indexable if the passive-optimal set

$u\in\{0,1\}$2

increases monotonically from the empty set to the full state space as the passivity subsidy $u\in\{0,1\}$3 increases. The Whittle index is the subsidy at which active and passive are both optimal (Niño-Mora, 19 Jan 2026).

For one-dimensional continuous-time Markovian bandits, “A unifying computations of Whittle’s Index for Markovian bandits” gives sufficient conditions for threshold optimality and a unifying analytical expression

$u\in\{0,1\}$4

which equals Whittle’s index when threshold policies are optimal, the arm is indexable, and $u\in\{0,1\}$5 is monotone in $u\in\{0,1\}$6 (Ayesta et al., 2019). The machine repairman model in that paper is an exact self-degrading instance: passive action deteriorates the machine to $u\in\{0,1\}$7 with rate $u\in\{0,1\}$8, while active action repairs it to $u\in\{0,1\}$9 with rate $r_a:\mathcal{S}_a\to[0,1]$0.

The review literature emphasizes that many degrading queueing, maintenance, and Age-of-Information models are indexable and admit threshold policies (Niño-Mora, 19 Jan 2026). Yet a common misconception is that this tractability extends to general restless Markov bandits. It does not: “Regret Bounds for Restless Markov Bandits” proves that index-based policies are necessarily suboptimal for the general restless problem, where all arms evolve independently of the learner’s actions and regret is measured against the best dynamic policy (Ortner et al., 2012). Self-degrading Markovian bandits are therefore best understood as a structured subclass in which strong results become possible, not as evidence that generic restless learning reduces to independent per-arm indices.

5. Latent regimes, MARBLE, and environment-averaged degradation

A more recent extension replaces stationary arm dynamics by a latent Markovian environment. MARBLE—Multi-Armed Restless Bandits in a Latent Markovian Environment—augments an RMAB with an unobserved environment state $r_a:\mathcal{S}_a\to[0,1]$1 evolving according to a Markov kernel

$r_a:\mathcal{S}_a\to[0,1]$2

with $r_a:\mathcal{S}_a\to[0,1]$3 and $r_a:\mathcal{S}_a\to[0,1]$4 both unobserved (Amiri et al., 12 Nov 2025). Conditioned on $r_a:\mathcal{S}_a\to[0,1]$5, arm $r_a:\mathcal{S}_a\to[0,1]$6 has transition kernel

$r_a:\mathcal{S}_a\to[0,1]$7

and reward

$r_a:\mathcal{S}_a\to[0,1]$8

From the controller’s perspective, the problem is partially observable, but the algorithm does not explicitly infer the latent environment.

The connection to self-degrading Markovian bandits is explicit. To encode degradation, one lets $r_a:\mathcal{S}_a\to[0,1]$9 represent degradation level, for example $n$0 “new” to $n$1 “fully degraded,” with

$n$2

when the arm is active, and

$n$3

when passive, together with rewards $n$4 decreasing in $n$5 (Amiri et al., 12 Nov 2025). In one latent regime an arm may have slow degradation and high reward; in another, fast degradation and low reward. This produces regime-driven nonstationarity.

MARBLE handles this through environment-averaged quantities

$n$6

where $n$7 is the stationary distribution of the latent environment. The resulting Bellman equation is that of a stationary single-arm RMAB with averaged dynamics. The paper introduces Markov-Averaged Indexability (MAI), which requires only that the environment-averaged single-arm problem be indexable. Under bounded rewards, step-size conditions, MAI, and access to calibrated simulators $n$8, synchronous Q-learning with Whittle Indices converges almost surely to the optimal environment-averaged Q-function and the corresponding Whittle indices (Amiri et al., 12 Nov 2025).

The practical testbed is a push-notification recommender system in which users are arms, engagement states are $n$9, the latent environment is $$\mathcal{F}_n \equiv \sigma\big((I_1,R_1),\dots,(I_n,R_n)\big),$$0, and the platform activates $$\mathcal{F}_n \equiv \sigma\big((I_1,R_1),\dots,(I_n,R_n)\big),$$1 of $$\mathcal{F}_n \equiv \sigma\big((I_1,R_1),\dots,(I_n,R_n)\big),$$2 users per step. In this calibrated digital twin, repeated activation can lower user engagement states, and the estimated indices converge close to the oracle Whittle policy (Amiri et al., 12 Nov 2025). This suggests that self-degrading dynamics need not be purely endogenous: latent exogenous regimes can be folded into the same formalism, provided the target of optimization is the environment-averaged dynamics rather than dynamic regret against fully time-varying latent regimes.

6. Scope, misconceptions, and open directions

Several distinctions delimit the theory. First, the exactness of the pure-policy benchmark is specific to the self-degrading subclass. In general restless bandits, the optimal dynamic policy can be substantially better than the best static arm, and the survey literature emphasizes that finding the optimal dynamic policy is PSPACE-hard (Niño-Mora, 19 Jan 2026). Second, hidden state is not the primary difficulty in the self-degrading subclass; rather, the main structural difficulty is that rarely-switching exploration is intrinsically insufficient for strict logarithmic regret without prior knowledge (Hira et al., 25 Jun 2026).

Third, indexability results are highly model-dependent. The survey identifies threshold policies, monotone state spaces, and Partial Conservation Laws as recurring sufficient structures, but it also stresses that explicit sufficient conditions for indexability are hard in general (Niño-Mora, 19 Jan 2026). The hidden-state threshold and indexability results in the binary availability-constrained model apply directly to rested settings and only partially to restless ones (Mehta et al., 2017). The general restless-learning literature further warns that per-arm index policies are not universally optimal (Ortner et al., 2012).

Current extensions are therefore shaped by assumptions. In MARBLE, the latent environment must be irreducible and aperiodic with unique stationary distribution, rewards must be bounded, MAI must hold, and the algorithm assumes access to a calibrated simulator $$\mathcal{F}_n \equiv \sigma\big((I_1,R_1),\dots,(I_n,R_n)\big),$$3 together with a tabular representation (Amiri et al., 12 Nov 2025). In UCB-NOM, stronger rates require prior knowledge in the form of a bias-span bound, while open directions include clustered or partial state information, more general degradation patterns, and settings where the state-space size is known and small (Hira et al., 25 Jun 2026). The RMAB survey highlights parallel open problems: easier and more general indexability conditions, scalable algorithms for heterogeneous degrading systems, asymptotic optimality beyond homogeneous arms, online learning with unknown degradation dynamics, and partially observable degradation processes (Niño-Mora, 19 Jan 2026).

Taken together, these results define self-degrading Markovian bandits as a sharply structured corner of the Markovian bandit landscape. Their hallmark is that suboptimal mixing degrades an arm’s own asymptotic value, which collapses the control benchmark to the best pure arm and makes hidden-state learning substantially more tractable than in generic restless settings (Hira et al., 25 Jun 2026). At the same time, the surrounding literature shows that once one reintroduces latent regimes, full restlessness, or general action-dependent dynamics, the theory rapidly reconnects with the harder questions of indexability, partial observability, and nonstationary reinforcement learning (Amiri et al., 12 Nov 2025).