---
title: 'Latent Bandits: Hidden-State Decision Models'
url: https://www.emergentmind.com/topics/latent-bandits
type: topic
---

# Latent Bandits: Hidden-State Decision Models

Searching arXiv for recent and foundational papers on latent bandits and related latent-state bandit formulations.
Latent bandits are sequential decision-making models in which rewards depend on an unobserved latent variable, and online learning is used to infer that variable while choosing actions. In the classical formulation, a latent bandit problem is one in which the learning agent knows the arm reward distributions conditioned on an unknown discrete latent state, and the primary goal is to identify the latent state, after which it can act optimally [2006.08714]. Subsequent work broadened this template to latent user classes for cold-start recommendation [1604.06743], latent Markov and hidden Markov dynamics [2012.00386; 2604.08149], latent heterogeneity in stochastic linear bandits [2502.00423], low-rank and clustered reward models [1609.01508; 2301.07040], and formulations that assume only preference orderings or partial orders rather than full reward distributions [2508.05367; 2605.07304]. This body of work treats latent bandits as a midpoint between online and offline learning, especially when rich offline models can be learned but per-instance online interaction is short [2006.08714].

## 1. Formal scope and canonical formulations

A common formalization uses a finite arm set, an observed context, and a latent state. In one widely used notation, at round \(t\) the learner observes \(X_t\in\mathcal X\), chooses \(A_t\in\mathcal A\), and receives \(R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)\), where \(S_t\) is unobserved [2207.03635]. The stationary case takes \(S_t\equiv S_1\); the non-stationary case places a Markov transition matrix on the latent state. The resulting posterior belief \(b_t(s)=P(S_t=s\mid H_t,X_t)\) is updated by
\[
b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .
\]
This posterior-filtering viewpoint is central in model-based latent-bandit algorithms [2207.03635].

The latent object is not always a discrete state in the same sense. In stochastic linear bandits with latent heterogeneity, each arrival has an unobserved subgroup \(g_i\in\{1,2\}\), drawn from a logistic gating model, and rewards are linear with subgroup-specific parameters [2502.00423]. In low-rank recommendation models, each user is represented by an unknown mixture over representative user classes, inducing a low-rank reward matrix [1609.01508]. In latent order and latent preference bandits, the latent state specifies only a preference ordering or a partial order over actions [2508.05367; 2605.07304]. A different line models rewards as deterministic functions of a common latent random variable \(X\), so that arm correlation is mediated entirely by that latent source [1808.05904].

At a more abstract level, latent bandits are not merely a family of examples. A de Finetti theorem for decision processes shows that every exchangeable and coherent stateless decision process is a latent bandit [2405.17324]. This result places latent-bandit models on a general probabilistic footing: latent structure is not only a modeling convenience for recommendation and personalization, but also a representation theorem for a broad class of stateless decision processes.

| Formulation | Latent object | Representative paper |
|---|---|---|
| Classical latent bandit | Unknown discrete latent state | [2006.08714] |
| Non-stationary latent bandit | Markov latent state \(S_t\) | [2012.00386] |
| Latent heterogeneous linear bandit | Unobserved subgroup \(g_i\) | [2502.00423] |
| Low-rank / cluster latent bandit | User mixture or user cluster | [1609.01508], [2301.07040] |
| Preference / order latent bandit | Ordering or partial order over actions | [2508.05367], [2605.07304] |
| Correlated latent-source bandit | Common latent random source \(X\) | [1808.05904] |

## 2. Posterior inference and classical decision rules

The foundational algorithmic template combines latent-state inference with optimism or posterior sampling. In the fixed-state model, the mUCB and mTS families maintain a posterior or a consistent set over latent states. For mUCB, the consistent set is
\[
C_t=\Bigl\{s\in\mathcal Z:\;G_t(s)\le \sigma\sqrt{6\,N_t(s)\ln T}\Bigr\},
\]
and the learner chooses the arm maximizing \(\mu(a,X_t,s)\) over \(s\in C_t\). For mTS, the learner samples a latent state \(B_t\) from the posterior and acts greedily for that state [2006.08714]. In the known-model case, mUCB satisfies
\[
R(T;s_*,\theta_*)\le 3Z+2\sigma\sqrt{6ZT\ln T},
\]
and the Bayes regret of mTS has the same bound. In the misspecified setting, the regret bound becomes
\[
R(T)\le T\delta+3Z+2T\varepsilon+2\sigma\sqrt{6ZT\ln T},
\]
showing explicitly how model uncertainty and approximation error enter the analysis [2006.08714]. A central implication is that when the number of latent states \(Z\) is much smaller than the number of arms \(K\), latent-state identification can replace the usual \(\sqrt{K}\)-scale dependence by a \(\sqrt{Z}\)-scale dependence [2006.08714].

Hong et al. extend this posterior-sampling view to non-stationary latent bandits with a Markov transition matrix over latent states [2012.00386]. Their model-based Thompson sampling algorithm maintains the exact filtering posterior \(P_t(s)=P[S_t=s|H_t]\), while the uncertain-model variant maintains a joint posterior over \((S_t,\theta,\phi)\). Because the exact joint update is generally intractable, the paper uses Sequential Monte Carlo with particles carrying latent paths, model parameters, and weights [2012.00386]. The same paper emphasizes a practical theme that recurs throughout the literature: offline-learned prototypes may be misspecified, but online posterior sampling can refine them over time [2012.00386].

In latent contextual bandits for personalized recommendations for new users, the latent variable is a user class \(c_u\in\{1,\dots,N\}\), rewards are linear within class, and the online system uses a two-phase design [1604.06743]. The first phase bootstraps data with LinUCB; the second re-estimates the latent mixture model, constructs one policy per latent class, and runs a meta contextual-bandit algorithm such as EXP4.P, EXP4, Epoch-Greedy, or Generalized Thompson Sampling over those policies [1604.06743]. The problem-independent regret guarantee is
\[
E[\mathrm{Reg}]=O(\sqrt{UTK\ln N}),
\]
under the setting described in the paper [1604.06743]. This formulation is explicitly tied to the cold-start problem and treats latent classes as reusable decision primitives rather than merely clustering artifacts [1604.06743].

A more explicitly Bayesian cold-start variant defines a posterior \(p_n(g)\) over user groups and scores candidate items by combining immediate reward with an approximation to expected future regret reduction [2305.18305]. That algorithm reports higher accuracy and lower regret empirically than decision-tree and cluster-based baselines on Netflix, Jester, and Goodreads10K, but the paper does not include formal regret-bound theorems [2305.18305]. This contrast is representative: some latent-bandit work prioritizes end-to-end Bayesian scoring and empirical recommendation performance, whereas other work emphasizes finite-time guarantees.

A recurring correction to the basic posterior-greedy template is that “choosing the highest reward arm according to the agent’s beliefs about the states” can be suboptimal when information-gathering arms are not themselves high-reward [2207.03635]. The AGEmTS algorithm therefore computes an information criterion based on average KL gap relative to average one-step regret, performs rollout comparisons between information-gathering and greedy strategies, and chooses the information arm if the projected long-run benefit exceeds a threshold [2207.03635]. This directly addresses a common failure mode of latent-bandit policies that treat latent inference as a passive by-product of reward maximization.

## 3. Non-stationary and dynamical latent environments

Non-stationarity in latent bandits has been modeled in several distinct ways. Hong et al. consider an unobserved latent state \(S_t\in\{1,\dots,M\}\) evolving as a Markov chain, with rewards \(R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)\) [2012.00386]. Their analysis derives a regret bound for sliding-window UCB on a path with \(L\) changes,
\[
R(n;\theta_*,s_{1:n})=O\bigl(n^{2/3}\sqrt{ML\log n}\bigr),
\]
and then transfers this to Thompson sampling through the Russo–Van Roy posterior/UCB reduction [2012.00386]. For fixed \(\theta_*,\phi_*\) and a latent path with change-rate \(p\), the Bayes regret becomes
\[
BR(n;\theta_*,\phi_*)=O\bigl(n^{2/3}\sqrt{M(1+pn)\log n}\bigr).
\]
The same work explicitly frames the problem as fast personalization to non-stationary users by combining offline and online learning [2012.00386].

A separate line studies finite-armed contextual bandits driven by a hidden Markov chain. The direct approach of “Staged LinUCB on estimated beliefs” uses contexts alone to estimate HMM beliefs \(\hat b_t\approx P(h_t\mid x_{1:t})\), then performs ridge regression and UCB on the stacked feature \(\hat b_t\otimes \phi(a,x_t)\) [2604.08149]. Under spectral identifiability, mixing, and belief-estimation assumptions, the paper proves a high-probability regret bound
\[
R_T=\widetilde O(T^{7/8}),
\]
with \(\lambda=T^{3/4}\) and stage length \(\ell=\lceil T^{3/4}\rceil\) [2604.08149]. It contrasts this with the “belief-linear” simplification associated with Nelson et al. (2022), for which \(\widetilde O(T^{1/2})\) is attainable when beliefs are known and \(\widetilde O(T^{3/4})\) with online belief estimation [2604.08149]. The paper’s point is not only algorithmic but also model-theoretic: it treats rewards as directly state-dependent rather than merely linear in posterior state probabilities [2604.08149].

Latent auto-regressive dynamics provide another non-stationary mechanism. In the latent AR bandit model, a one-dimensional latent state \(z_t\) follows an \(AR(k)\) process, and arm rewards are linear in \(z_t\) [2402.03110]. The key reduction rewrites rewards using past realized rewards and residuals, turning the problem into a linear bandit with corrupted contexts. A LinUCB-style algorithm on these corrupted contexts achieves
\[
\mathrm{Reg}(T)=\widetilde O(k\sqrt T),
\]
under AR stability, boundedness, and a control condition on cumulative context corruption [2402.03110]. This model is notable იმიტომ that it does not impose a sub-linear non-stationarity budget; instead it exploits a parameterized latent dynamics model [2402.03110].

State-model-free approaches avoid explicit latent-state inference altogether. “Adaptive Exploration for Latent-State Bandits” introduces LC-UCB based on lagged contexts \(\phi_t=(a_{t-1},r_{t-1})\), as well as RP-UCB and SP-UCB, which deliberately probe multiple arms to form state fingerprints [2602.05139]. In a two-state setting, fixed-interval probing every \(\tau\) rounds yields
\[
\frac{1}{T}E[R_T]\le \frac{\Delta_{\rm probe}}{\tau}+\frac{\Delta_{\max}q\tau}{2}+\Delta_{\max}\varepsilon_{\rm fp}+O\!\left(\frac1T\right),
\]
which makes the exploration–staleness trade-off explicit [2602.05139]. The adaptive variants trigger probing by residual, uncertainty, and hazard gates rather than a fixed schedule [2602.05139].

In restless bandits, MARBLE augments the RMAB model with a latent Markov environment \(Z_k\), averages transitions and rewards over the stationary law \(\mu_Z\), and introduces Markov-Averaged Indexability [2511.09324]. Under MAI and two-timescale stochastic approximation, synchronous Q-learning with Whittle Indices converges almost surely to the optimal averaged Q-function and averaged Whittle indices [2511.09324]. The paper does not provide a finite-time regret bound and explicitly leaves sample complexity and regret as future work [2511.09324]. That limitation marks an important boundary in current latent-bandit theory: asymptotic convergence is available in some dynamical latent settings even when finite-time regret is not.

## 4. Linear, contextual, and identifiable latent representations

Latent structure is especially consequential in linear and contextual bandits. In stochastic linear bandits with latent heterogeneity, each arrival \(i\) has an unobserved subgroup \(g_i\in\{1,2\}\), with logistic gating
\[
\Pr(g_i=1\mid z_i)=p(z_i^\top\gamma^*),\qquad p(u)=\frac{1}{1+e^{-u}},
\]
and subgroup-specific linear rewards \(y_{i,k}\mid g_i=g=x_{i,k}^\top\beta_g^*+\epsilon_{i,k}\) [2502.00423]. The proposed phased EM-based algorithm alternates between regularized EM updates on past data and greedy action selection based on the predicted subgroup \(\widehat g_i\) [2502.00423]. The paper distinguishes two regret notions. The “strong regret” against an oracle that knows each realized subgroup is necessarily non-sub-linear because the latent-group classification error \(R(\gamma^*)>0\) cannot vanish, and the upper bound includes a linear term proportional to \(R(\gamma^*)T\). By contrast, the “regular regret” against an oracle that knows only the deterministic components is
\[
\mathrm{Reg}(T)=O\Bigl(n_0+\|\beta_1^*-\beta_2^*\|_1\sqrt{s\log d\log n_0}\sqrt T\Bigr),
\]
with a matching minimax lower bound \(\Omega(\|\beta_1^*-\beta_2^*\|_1\sqrt{s\log d\,T})\) [2502.00423]. The same paper shows that a standard linear bandit fitting a single model incurs constant average regret that accumulates linearly over time under latent heterogeneity [2502.00423].

Offline data can also be used to learn a latent subspace before online interaction begins. In linear latent bandits, the unknown reward parameter is \(\beta_*=U_*\theta_*\) with \(U_*\in\mathbb R^{d_A\times d_K}\) and \(d_K\ll d_A\) [2405.17324]. SOLD learns the subspace from short offline trajectories, while LOCAL-UCB and ProBALL-UCB exploit it online [2405.17324]. LOCAL-UCB enjoys
\[
\mathrm{Reg}_T=\widetilde O\Bigl(\min\{d_A\sqrt T,\; d_K\sqrt T(1+\sqrt{d_AT/(d_KN)})\}\Bigr),
\]
so the effective dimension decreases as the offline dataset size \(N\) grows [2405.17324]. This line treats latent bandits not only as a hidden-state problem but also as an offline–online dimension-reduction problem.

Identifiability results push the latent approach further. In identifiable latent bandits, each instance \(q\) has an unobserved latent vector \(Z_q\), contexts are generated by a smooth invertible mixing function \(g\), and rewards are linear in \(Z_q\) through arm-specific parameters \(\theta_a\) [2407.16239]. Training a representation \(h(x)\) to classify which patient generated a context via multinomial logistic regression recovers
\[
h(x)=A\,g^{-1}(x)+d
\]
for an invertible matrix \(A\) and vector \(d\), under the stated assumptions [2407.16239]. After fitting reward parameters in this latent space, the online policy simply averages inferred latent vectors and acts greedily. The paper shows that this greedy policy has constant expected regret,
\[
E[R_T]=O(1),
\]
because the latent estimate converges at rate \(O(1/\sqrt t)\) and the best arm is identified after \(O(1)\) additional pulls [2407.16239]. This is one of the strongest available latent-bandit statements: when a sufficient representation is identifiable from observational data, online exploration can collapse to a short identification phase.

Transfer under latent covariate shift adds a causal layer. In latent contextual bandits with hidden context \(z\), observed proxy \(w\), and a target domain with shifted \(P_Z\), naive transfer of \(E[Y\mid w,x]\) can cause negative transfer [2502.20153]. The transportability analysis yields
\[
p^*(y\mid w,do(x))=\sum_z p(y\mid z,x)\,p^*(z\mid w),
\]
and the paper uses this formula directly in a binary case and approximately via a variational autoencoder in the high-dimensional proxy case [2502.20153]. No closed-form regret bound is provided, but the empirical claim is that naive transfer can lead to \(O(T)\) regret, whereas transportability-guided transfer avoids that failure mode [2502.20153].

## 5. Low-rank, clustered, and collaborative latent bandits

Recommendation-style latent bandits often exploit shared structure across many users. In low-rank bandits with latent mixtures, user \(b\) has an unknown mixture vector \(v_b\in\Delta_C\) over latent classes, items have unknown class-reward vectors, and the expected reward matrix factorizes as \(R=UV^\top\) with rank at most \(C\) [1609.01508]. The algorithm alternates between “ESTIMATE” mini-sessions, which collect random samples and recover item-side features via the Robust Tensor Power Method of Anandkumar et al. (2014b), and “OFUL” mini-sessions, which learn user-side mixture weights using the OFUL algorithm of Abbasi-Yadkori et al. (2011) [1609.01508]. The expected regret after \(T\) user interactions is
\[
\widetilde O(C\sqrt{BT}),
\]
where \(B\) is the number of users [1609.01508]. A key technical contribution is a robustness property of OFUL for approximately linear reward models [1609.01508].

Cluster-structured latent bandits replace mixtures by discrete user clusters. Pal et al. consider \(N\) users partitioned into \(C\) unknown clusters, all users in a cluster sharing the same mean-reward vector across \(M\) arms [2301.07040]. Their LATTICE algorithm operates in \(O(\log T)\) phases, repeatedly collecting partial observations on user-group by active-arm submatrices, calling an offline low-rank matrix completion oracle, clustering estimated rows, and eliminating clearly suboptimal arms [2301.07040]. Under the cluster-structure assumptions, LATTICE achieves the minimax optimal regret
\[
\widetilde O(\sqrt{(M+N)T}),
\]
when the number of clusters is \(\widetilde O(1)\), and it requires only \(O(\log T)\) calls to the offline matrix completion oracle [2301.07040]. The paper emphasizes that this is the first algorithm with such a strong regret bound in that model [2301.07040].

Blocked collaborative bandits impose an additional budget constraint: each user–arm pair can be sampled at most \(\mathsf B\) times. B-LATTICE extends the matrix-completion and latent-clustering strategy to this blocked setting [2311.03376]. Under \(\mathsf B=\Theta(\log \mathsf T)\), the per-user regret is
\[
\widetilde O\bigl(\sqrt{\mathsf T(1+\mathsf N\mathsf M^{-1})}\bigr),
\]
which the paper describes as the first sub-linear regret bound for the blocked collaborative-bandit problem and one that matches the minimax regret bound when \(\mathsf B=\mathsf T\) [2311.03376]. The algorithm explicitly separates “exploit” rounds, which use high-confidence golden items, from “explore” rounds, which sample entries for matrix completion [2311.03376].

These collaborative models illustrate a distinctive role of latent bandits in modern recommendation theory. The latent variable need not be inferred only within one trajectory; it can be amortized across many users, learned by matrix completion or tensor decomposition, and then exploited online with a much smaller effective exploration burden. This suggests why the cold-start problem, personalized recommendation, and online collaborative filtering are recurring application domains throughout the literature [1604.06743; 2301.07040].

## 6. Relaxed priors, misspecification, and recurrent failure modes

A persistent theme is that classical latent-bandit assumptions may be too strong. “Latent Preference Bandits” assumes not a full reward model \(\{p(R_a\mid Z)\}\), but only that each latent state \(z\) induces an ordering \(O_z\) over actions [2508.05367]. The algorithm lpbTS samples a latent state from an approximate posterior, re-estimates state-wise mean vectors by weighted isotonic regression over the constraint set \(H_z\), and updates the posterior by constrained likelihood [2508.05367]. The paper gives an asymptotic lower bound adapted from Agrawal and Teneketzis (1989) and Maillard and Mannor (2014), along with a worst-case upper bound
\[
R(T)=O\bigl(\sqrt{\min(k,m)\,T}\bigr).
\]
Its empirical claim is that preference-only modeling is competitive with latent bandits that have full knowledge of the reward distribution when that model is well specified, and superior when absolute reward scales differ across instances with the same latent state [2508.05367].

Latent Order Bandits generalize this idea from total orders to partial orders \(O_s\subset A\times A\) [2605.07304]. The lobUCB algorithm projects empirical means onto the feasible cones \(H_s\), merges arms pooled by violated order constraints, and then computes UCB indices using pooled counts [2605.07304]. For total orders with unique optimal arms, the paper gives both instance-dependent and instance-independent regret bounds, including
\[
\mathrm{Reg}(T)=O\bigl(k\sqrt{mT\ln T}\bigr)
\]
in the instance-independent case [2605.07304]. The associated posterior-sampling variant lobTS uses likelihood upper bounds over the union of feasible cones where a given arm is optimal [2605.07304]. These formulations address a concrete limitation of full-prior latent bandits: a small number of latent states may capture relative preferences but not common reward scales.

A related but structurally different relaxation appears in transfer learning with latent continuity. There the latent quantity is a shared Lipschitz constant \(L\) across tasks, rather than a latent state per trajectory [2102.02472]. The estimator \(\hat L_\beta\) uses the \(\beta\)-th largest empirical Lipschitz estimate across prior tasks, plus a safety margin \(\varepsilon_\beta\), and the plug-in algorithm \(\pi(\hat L_\beta)\) then runs a Lipschitz-aware bandit algorithm on the new task [2102.02472]. The paper’s main qualitative message is explicit: underestimating \(L\) is catastrophic and can cause linear regret, whereas overestimating \(L\) only increases the regret constant \(C(\mu,L')\) [2102.02472]. This work broadens the latent-bandit idea from latent reward states to latent structural parameters shared across tasks.

Several recurrent failure modes are now well documented. First, always choosing the arm with highest expected reward under the current belief can leave the learner permanently uncertain because information-gathering arms may never be selected [2207.03635]. Second, in stochastic linear bandits with latent heterogeneity, collapsing all observations into a single linear model leads to bias and linear cumulative regret [2502.00423]. Third, under covariate shift in latent contextual bandits, naively transferring all knowledge from the source domain can lead to negative transfer and \(O(T)\) regret [2502.20153]. Fourth, some dynamical latent-bandit frameworks, such as MARBLE, presently offer almost-sure convergence without finite-time regret guarantees [2511.09324].

These limitations are not peripheral. They define much of the current research agenda. A plausible implication is that future latent-bandit methods will continue to move away from fully specified latent-state reward models toward weaker but more stable forms of transferable structure: identifiable representations, low-dimensional subspaces, order constraints, causal transport formulas, and adaptive information-gathering rules.

Source: https://www.emergentmind.com/topics/latent-bandits