---
title: Self-Degrading Markovian Bandits
url: https://www.emergentmind.com/topics/self-degrading-markovian-bandits
type: topic
---

# Self-Degrading Markovian Bandits

Searching arXiv for the most relevant papers on self-degrading Markovian bandits and closely related RMAB models.
{"query":"self-degrading Markovian bandits latent Markovian environment MARBLE Whittle index hidden states arXiv","max_results":10}
Self-degrading Markovian bandits are a generalization of rested Markovian bandits in which each arm is an ergodic Markov reward process with non-observable states and possibly constrained decision epochs, and in which pure policies are always asymptotically optimal [2606.27448]. In this class, any deviation from the pure policy can only hurt the asymptotic reward of an arm: the best long-run way to use arm \(a\) is to commit to it permanently, and mixing it with other arms cannot improve its asymptotic performance [2606.27448]. The concept belongs to the broader landscape of Markovian restless bandits, hidden-state bandits, and latent-regime RMABs, where degradation may arise endogenously through the arm’s own state dynamics or exogenously through an unobserved environment [2511.09324].

## 1. Formal model and defining property

In the formulation introduced for hidden-state learning with constrained decision epochs, there is a finite set of arms \(\mathcal{A}=\{1,\dots,K\}\). Each arm \(a\) has a state space \(\mathcal{S}_a\), a transition kernel
\[
P_a(s' \mid s,u),
\]
where \(u\in\{0,1\}\) is the activation indicator, and a reward function \(r_a:\mathcal{S}_a\to[0,1]\). The learner does not observe the Markov states. At decision epoch \(n\), the observation history is
\[
\mathcal{F}_n \equiv \sigma\big((I_1,R_1),\dots,(I_n,R_n)\big),
\]
where \(I_n\) is the selected arm and \(R_n\) the realized reward. The model also allows constrained decision epochs: if \(\{T_n\}\) denotes the sequence of decision times, then at time \(T_n\) the learner selects \(I_n\), which is activated over \([T_n,T_{n+1})\) [2606.27448].

The defining property of a self-degrading arm is that, for any policy \(\pi\), the time-average reward obtained from arm \(a\) whenever it is pulled is asymptotically bounded above by \(\mu_a\), where \(\mu_a\) is the asymptotic average reward achieved by the pure policy that always plays arm \(a\). Equivalently, the arm is its own best companion: the best way to use arm \(a\) is to commit to it permanently; leaving it idle, switching away, or returning later cannot improve its long-run performance [2606.27448]. This generalizes classical rested Markovian bandits, where inactive arms freeze, by allowing non-activation dynamics that can move the state distribution to worse regions.

A related hidden-state template appears in the binary-state availability-constrained model of “Multi-armed Bandits with Constrained Arms and Hidden States,” where each arm has hidden state \(X_n(t)\in\{0,1\}\), observed availability \(Y_n(t)\in\{0,1\}\), and belief state \(S_n(t)=(\pi_n(t),Y_n(t))\) with \(\pi_n(t)=\Pr(X_n(t)=0\mid H_t)\). In that setting, the arm-level control problem is a belief-MDP, and the optimal policy is a threshold policy under the stated monotonicity conditions [1710.07115]. For self-degrading models, this binary hidden-state construction is a concrete special case in which “good” and “bad” quality levels are inferred through rewards rather than directly observed.

## 2. Pure policies, asymptotic optimality, and regret

For each arm \(a\), define the asymptotic average reward under the pure policy that always selects \(a\) as
\[
\mu_a := \lim_{N\to\infty}\frac{1}{N}\mathbb{E}\Big[\sum_{n=1}^N R_n \,\Big|\, I_1=\dots=I_N=a\Big].
\]
Let
\[
a^* \in \arg\max_a \mu_a,\qquad \mu^* := \mu_{a^*}.
\]
A central theorem for self-degrading Markovian bandits states that the optimal average reward over all policies equals \(\mu^*\), and it is attained by any pure policy that always pulls an arm maximizing \(\mu_a\). No policy that mixes arms can achieve an asymptotic average reward strictly greater than \(\mu^*\) [2606.27448].

This result is structurally important because the “pure” benchmark is not a relaxation in this subclass. The regret benchmark is
\[
R_T := T\mu^* - \mathbb{E}\left[\sum_{t=1}^T R_t\right].
\]
In ordinary restless bandits, benchmarking against the best static arm would generally be weaker than benchmarking against the optimal dynamic policy. In self-degrading Markovian bandits, by contrast, the pure benchmark coincides with the optimal average-reward benchmark because mixing cannot improve long-run performance [2606.27448].

The lower-bound theory is correspondingly distinctive. Without prior knowledge on the underlying bandit, the regret of algorithms that switch arms rarely necessarily scales super-logarithmically for every bandit, i.e.
\[
\omega(\log(T)),
\]
where \(T\) is the learning horizon. Thus strict logarithmic regret is unattainable for rarely switching algorithms unless additional structural knowledge is supplied [2606.27448]. This separates self-degrading hidden-state learning from classical stochastic bandits, in which logarithmic regret is the canonical regime.

## 3. Hidden-state learning and regret guarantees

The main positive algorithmic result for the self-degrading subclass is UCB-NOM, an optimistic algorithm inspired by UCB and tailored to Markovian rewards, non-observable states, and constrained decision epochs [2606.27448]. For each arm \(a\), the algorithm tracks the empirical mean
\[
\hat{\mu}_a(t) := \frac{1}{N_a(t)} \sum_{n \le t: I_n = a} R_n,
\]
and constructs an optimistic index of the form
\[
\text{index}_a(t) := \hat{\mu}_a(t) + f_a(t),
\]
where the confidence radius accounts for Markovian dependence through the bias function \(h_a\), defined by
\[
\mu_a + h_a(s) = r_a(s) + \sum_{s'} P_a(s' \mid s,1)\, h_a(s').
\]
Its span is
\[
\mathrm{sp}(h_a) := \max_s h_a(s)-\min_s h_a(s).
\]

Without prior knowledge of the bias span, UCB-NOM achieves nearly logarithmic regret. With prior knowledge in the form of a bound \(\mathrm{sp}(h_a)\le H\), a proper instantiation of UCB-NOM achieves \(O(\log(T))\) instance-dependent regret and \(O(\sqrt{T\log(T)})\) worst-case regret [2606.27448]. A notable feature of these bounds is that they do not depend on the number of states of the underlying Markov chains. The paper accordingly concludes that the non-observability of states is a mild inconvenience in self-degrading Markovian bandits [2606.27448].

The concentration tools underlying Markovian bandit learning predate this formulation. For finite-state rested Markovian arms, “A Hoeffding Inequality for Finite State Markov Chains and its Applications to Markovian Bandits” proves a Hoeffding inequality controlled by the maximum hitting time
\[
\mathrm{HitT}(P)=\max_{x,y\in S}\mathbb{E}[T_y\mid X_1=x],
\]
and uses it to analyze a Markovian \(\beta\)-UCB algorithm with \(O(\log T)\) regret, with constants scaling with \((d-c)^2\max_a \mathrm{HitT}(P_{\theta_a})^2\) [2001.01199]. That analysis applies directly only when each arm can be represented as a finite, irreducible, time-homogeneous Markov chain, but it provides a methodological baseline for the hidden-state concentration arguments used in later work.

## 4. Restless formulations, threshold structure, and indexability

Self-degrading Markovian bandits sit naturally inside the restless bandit formalism. In the standard survey formulation, a rested bandit has passive kernel \(P_n^0(x,y)=\delta_{xy}\), whereas a restless bandit allows both active and passive dynamics
\[
P_n^a(x,y)=\Pr(X_{n,t+1}=y\mid X_{n,t}=x,A_{n,t}=a),\qquad a\in\{0,1\},
\]
under a resource constraint such as
\[
\sum_{n=1}^N A_{n,t}=M.
\]
This is precisely the modeling move needed for degradation with use, aging while idle, or repair under activation [2601.13045].

Whittle’s relaxation replaces the hard per-period activation constraint by an average constraint and decomposes the RMAB into single-arm subsidy problems. For a discounted single arm, the Bellman equation takes the form
\[
V_\lambda(x)=\max_{a\in\{0,1\}}\Big[r(x,a)+\lambda(1-a)+\beta\sum_{y\in S}P^a(x,y)V_\lambda(y)\Big].
\]
An arm is indexable if the passive-optimal set
\[
\mathcal{P}(\lambda)=\{x\in S:\text{some optimal policy is passive in state }x\}
\]
increases monotonically from the empty set to the full state space as the passivity subsidy \(\lambda\) increases. The Whittle index is the subsidy at which active and passive are both optimal [2601.13045].

For one-dimensional continuous-time Markovian bandits, “A unifying computations of Whittle’s Index for Markovian bandits” gives sufficient conditions for threshold optimality and a unifying analytical expression
\[
\tilde W_k(n):=
\frac{\mathbb{E}(T_k^n(N_k^n,S_k^n(N_k^n)))-\mathbb{E}(T_k^{n-1}(N_k^{n-1},S_k^{n-1}(N_k^{n-1})))}
{\mathbb{E}(f_k(N_k^n,S_k^n(N_k^n)))-\mathbb{E}(f_k(N_k^{n-1},S_k^{n-1}(N_k^{n-1})))},
\]
which equals Whittle’s index when threshold policies are optimal, the arm is indexable, and \(\tilde W_k(n)\) is monotone in \(n\) [1906.10946]. The machine repairman model in that paper is an exact self-degrading instance: passive action deteriorates the machine to \(n+1\) with rate \(\lambda_k(n)\), while active action repairs it to \(0\) with rate \(r_k(n)\).

The review literature emphasizes that many degrading queueing, maintenance, and Age-of-Information models are indexable and admit threshold policies [2601.13045]. Yet a common misconception is that this tractability extends to general restless Markov bandits. It does not: “Regret Bounds for Restless Markov Bandits” proves that index-based policies are necessarily suboptimal for the general restless problem, where all arms evolve independently of the learner’s actions and regret is measured against the best dynamic policy [1209.2693]. Self-degrading Markovian bandits are therefore best understood as a structured subclass in which strong results become possible, not as evidence that generic restless learning reduces to independent per-arm indices.

## 5. Latent regimes, MARBLE, and environment-averaged degradation

A more recent extension replaces stationary arm dynamics by a latent Markovian environment. MARBLE—Multi-Armed Restless Bandits in a Latent Markovian Environment—augments an RMAB with an unobserved environment state \(E_k\in\mathcal{E}\) evolving according to a Markov kernel
\[
H(e'\mid e)=\Pr\{E_{k+1}=e'\mid E_k=e\},
\]
with \(E_k\) and \(H\) both unobserved [2511.09324]. Conditioned on \(E_k=e\), arm \(i\) has transition kernel
\[
p_e^{\,i}(s'\mid s,a)
\]
and reward
\[
r_e^{\,i}(s,a).
\]
From the controller’s perspective, the problem is partially observable, but the algorithm does not explicitly infer the latent environment.

The connection to self-degrading Markovian bandits is explicit. To encode degradation, one lets \(\mathcal{S}\) represent degradation level, for example \(s=0\) “new” to \(s=K\) “fully degraded,” with
\[
p_e^{\,i}(s+1\mid s,1)\ \text{large},
\]
when the arm is active, and
\[
p_e^{\,i}(s+1\mid s,0)\ \text{small, or } p_e^{\,i}(s-1\mid s,0)\ \text{positive},
\]
when passive, together with rewards \(r_e^{\,i}(s,a)\) decreasing in \(s\) [2511.09324]. In one latent regime an arm may have slow degradation and high reward; in another, fast degradation and low reward. This produces regime-driven nonstationarity.

MARBLE handles this through environment-averaged quantities
\[
r_{\mathcal E}^{\,i}(s,a):=\sum_{e\in\mathcal E}\mu_{\mathcal E}(e)\,r_e^{\,i}(s,a),\qquad
p_{\mathcal E}^{\,i}(s'\mid s,a):=\sum_{e\in\mathcal E}\mu_{\mathcal E}(e)\,p_e^{\,i}(s'\mid s,a),
\]
where \(\mu_{\mathcal E}\) is the stationary distribution of the latent environment. The resulting Bellman equation is that of a stationary single-arm RMAB with averaged dynamics. The paper introduces Markov-Averaged Indexability (MAI), which requires only that the environment-averaged single-arm problem be indexable. Under bounded rewards, step-size conditions, MAI, and access to calibrated simulators \(\mathcal G_i(\theta)\), synchronous Q-learning with Whittle Indices converges almost surely to the optimal environment-averaged Q-function and the corresponding Whittle indices [2511.09324].

The practical testbed is a push-notification recommender system in which users are arms, engagement states are \(s\in\{1,2,3,4\}\), the latent environment is \(E_t\in\{E_1,E_2\}\), and the platform activates \(M=10\) of \(N=100\) users per step. In this calibrated digital twin, repeated activation can lower user engagement states, and the estimated indices converge close to the oracle Whittle policy [2511.09324]. This suggests that self-degrading dynamics need not be purely endogenous: latent exogenous regimes can be folded into the same formalism, provided the target of optimization is the environment-averaged dynamics rather than dynamic regret against fully time-varying latent regimes.

## 6. Scope, misconceptions, and open directions

Several distinctions delimit the theory. First, the exactness of the pure-policy benchmark is specific to the self-degrading subclass. In general restless bandits, the optimal dynamic policy can be substantially better than the best static arm, and the survey literature emphasizes that finding the optimal dynamic policy is PSPACE-hard [2601.13045]. Second, hidden state is not the primary difficulty in the self-degrading subclass; rather, the main structural difficulty is that rarely-switching exploration is intrinsically insufficient for strict logarithmic regret without prior knowledge [2606.27448].

Third, indexability results are highly model-dependent. The survey identifies threshold policies, monotone state spaces, and Partial Conservation Laws as recurring sufficient structures, but it also stresses that explicit sufficient conditions for indexability are hard in general [2601.13045]. The hidden-state threshold and indexability results in the binary availability-constrained model apply directly to rested settings and only partially to restless ones [1710.07115]. The general restless-learning literature further warns that per-arm index policies are not universally optimal [1209.2693].

Current extensions are therefore shaped by assumptions. In MARBLE, the latent environment must be irreducible and aperiodic with unique stationary distribution, rewards must be bounded, MAI must hold, and the algorithm assumes access to a calibrated simulator \(\mathcal G_i(\theta)\) together with a tabular representation [2511.09324]. In UCB-NOM, stronger rates require prior knowledge in the form of a bias-span bound, while open directions include clustered or partial state information, more general degradation patterns, and settings where the state-space size is known and small [2606.27448]. The RMAB survey highlights parallel open problems: easier and more general indexability conditions, scalable algorithms for heterogeneous degrading systems, asymptotic optimality beyond homogeneous arms, online learning with unknown degradation dynamics, and partially observable degradation processes [2601.13045].

Taken together, these results define self-degrading Markovian bandits as a sharply structured corner of the Markovian bandit landscape. Their hallmark is that suboptimal mixing degrades an arm’s own asymptotic value, which collapses the control benchmark to the best pure arm and makes hidden-state learning substantially more tractable than in generic restless settings [2606.27448]. At the same time, the surrounding literature shows that once one reintroduces latent regimes, full restlessness, or general action-dependent dynamics, the theory rapidly reconnects with the harder questions of indexability, partial observability, and nonstationary reinforcement learning [2511.09324].

Source: https://www.emergentmind.com/topics/self-degrading-markovian-bandits