Latent Bandits: Hidden-State Decision Models

• Latent bandits are sequential decision-making models where rewards depend on unobserved states, requiring online latent inference.
• They encompass various formulations such as discrete latent states, Markov dynamics, preference orderings, and low-rank collaborative structures.
• They combine statistical inference with exploration strategies like UCB and Thompson Sampling to balance reward maximization with latent state identification.

Searching arXiv for recent and foundational papers on latent bandits and related latent-state bandit formulations. Latent bandits are sequential decision-making models in which rewards depend on an unobserved latent variable, and online learning is used to infer that variable while choosing actions. In the classical formulation, a latent bandit problem is one in which the learning agent knows the arm reward distributions conditioned on an unknown discrete latent state, and the primary goal is to identify the latent state, after which it can act optimally (Hong et al., 2020). Subsequent work broadened this template to latent user classes for cold-start recommendation (Zhou et al., 2016), latent Markov and hidden Markov dynamics (Hong et al., 2020, Li et al., 9 Apr 2026), latent heterogeneity in stochastic linear bandits (Chen et al., 1 Feb 2025), low-rank and clustered reward models (Gopalan et al., 2016, Pal et al., 2023), and formulations that assume only preference orderings or partial orders rather than full reward distributions (Mwai et al., 7 Aug 2025, Carlsson et al., 8 May 2026). This body of work treats latent bandits as a midpoint between online and offline learning, especially when rich offline models can be learned but per-instance online interaction is short (Hong et al., 2020).

1. Formal scope and canonical formulations

A common formalization uses a finite arm set, an observed context, and a latent state. In one widely used notation, at round $t$ the learner observes $X_t\in\mathcal X$, chooses $A_t\in\mathcal A$, and receives $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$, where $S_t$ is unobserved (Galozy et al., 2022). The stationary case takes $S_t\equiv S_1$; the non-stationary case places a Markov transition matrix on the latent state. The resulting posterior belief $b_t(s)=P(S_t=s\mid H_t,X_t)$ is updated by

$$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$

This posterior-filtering viewpoint is central in model-based latent-bandit algorithms (Galozy et al., 2022).

The latent object is not always a discrete state in the same sense. In stochastic linear bandits with latent heterogeneity, each arrival has an unobserved subgroup $g_i\in\{1,2\}$, drawn from a logistic gating model, and rewards are linear with subgroup-specific parameters (Chen et al., 1 Feb 2025). In low-rank recommendation models, each user is represented by an unknown mixture over representative user classes, inducing a low-rank reward matrix (Gopalan et al., 2016). In latent order and latent preference bandits, the latent state specifies only a preference ordering or a partial order over actions (Mwai et al., 7 Aug 2025, Carlsson et al., 8 May 2026). A different line models rewards as deterministic functions of a common latent random variable $X$, so that arm correlation is mediated entirely by that latent source (Gupta et al., 2018).

At a more abstract level, latent bandits are not merely a family of examples. A de Finetti theorem for decision processes shows that every exchangeable and coherent stateless decision process is a latent bandit (Kausik et al., 2024). This result places latent-bandit models on a general probabilistic footing: latent structure is not only a modeling convenience for recommendation and personalization, but also a representation theorem for a broad class of stateless decision processes.

Formulation	Latent object	Representative paper
Classical latent bandit	Unknown discrete latent state	(Hong et al., 2020)
Non-stationary latent bandit	Markov latent state $X_t\in\mathcal X$0	(Hong et al., 2020)
Latent heterogeneous linear bandit	Unobserved subgroup $X_t\in\mathcal X$1	(Chen et al., 1 Feb 2025)
Low-rank / cluster latent bandit	User mixture or user cluster	(Gopalan et al., 2016, Pal et al., 2023)
Preference / order latent bandit	Ordering or partial order over actions	(Mwai et al., 7 Aug 2025, Carlsson et al., 8 May 2026)
Correlated latent-source bandit	Common latent random source $X_t\in\mathcal X$2	(Gupta et al., 2018)

2. Posterior inference and classical decision rules

The foundational algorithmic template combines latent-state inference with optimism or posterior sampling. In the fixed-state model, the mUCB and mTS families maintain a posterior or a consistent set over latent states. For mUCB, the consistent set is

$X_t\in\mathcal X$3

and the learner chooses the arm maximizing $X_t\in\mathcal X$4 over $X_t\in\mathcal X$5. For mTS, the learner samples a latent state $X_t\in\mathcal X$6 from the posterior and acts greedily for that state (Hong et al., 2020). In the known-model case, mUCB satisfies

$X_t\in\mathcal X$7

and the Bayes regret of mTS has the same bound. In the misspecified setting, the regret bound becomes

$X_t\in\mathcal X$8

showing explicitly how model uncertainty and approximation error enter the analysis (Hong et al., 2020). A central implication is that when the number of latent states $X_t\in\mathcal X$9 is much smaller than the number of arms $A_t\in\mathcal A$0, latent-state identification can replace the usual $A_t\in\mathcal A$1-scale dependence by a $A_t\in\mathcal A$2-scale dependence (Hong et al., 2020).

Hong et al. extend this posterior-sampling view to non-stationary latent bandits with a Markov transition matrix over latent states (Hong et al., 2020). Their model-based Thompson sampling algorithm maintains the exact filtering posterior $A_t\in\mathcal A$3, while the uncertain-model variant maintains a joint posterior over $A_t\in\mathcal A$4. Because the exact joint update is generally intractable, the paper uses Sequential Monte Carlo with particles carrying latent paths, model parameters, and weights (Hong et al., 2020). The same paper emphasizes a practical theme that recurs throughout the literature: offline-learned prototypes may be misspecified, but online posterior sampling can refine them over time (Hong et al., 2020).

In latent contextual bandits for personalized recommendations for new users, the latent variable is a user class $A_t\in\mathcal A$5, rewards are linear within class, and the online system uses a two-phase design (Zhou et al., 2016). The first phase bootstraps data with LinUCB; the second re-estimates the latent mixture model, constructs one policy per latent class, and runs a meta contextual-bandit algorithm such as EXP4.P, EXP4, Epoch-Greedy, or Generalized Thompson Sampling over those policies (Zhou et al., 2016). The problem-independent regret guarantee is

$A_t\in\mathcal A$6

under the setting described in the paper (Zhou et al., 2016). This formulation is explicitly tied to the cold-start problem and treats latent classes as reusable decision primitives rather than merely clustering artifacts (Zhou et al., 2016).

A more explicitly Bayesian cold-start variant defines a posterior $A_t\in\mathcal A$7 over user groups and scores candidate items by combining immediate reward with an approximation to expected future regret reduction (Young et al., 2023). That algorithm reports higher accuracy and lower regret empirically than decision-tree and cluster-based baselines on Netflix, Jester, and Goodreads10K, but the paper does not include formal regret-bound theorems (Young et al., 2023). This contrast is representative: some latent-bandit work prioritizes end-to-end Bayesian scoring and empirical recommendation performance, whereas other work emphasizes finite-time guarantees.

A recurring correction to the basic posterior-greedy template is that “choosing the highest reward arm according to the agent’s beliefs about the states” can be suboptimal when information-gathering arms are not themselves high-reward (Galozy et al., 2022). The AGEmTS algorithm therefore computes an information criterion based on average KL gap relative to average one-step regret, performs rollout comparisons between information-gathering and greedy strategies, and chooses the information arm if the projected long-run benefit exceeds a threshold (Galozy et al., 2022). This directly addresses a common failure mode of latent-bandit policies that treat latent inference as a passive by-product of reward maximization.

3. Non-stationary and dynamical latent environments

Non-stationarity in latent bandits has been modeled in several distinct ways. Hong et al. consider an unobserved latent state $A_t\in\mathcal A$8 evolving as a Markov chain, with rewards $A_t\in\mathcal A$9 (Hong et al., 2020). Their analysis derives a regret bound for sliding-window UCB on a path with $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$0 changes,

$R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$1

and then transfers this to Thompson sampling through the Russo–Van Roy posterior/UCB reduction (Hong et al., 2020). For fixed $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$2 and a latent path with change-rate $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$3, the Bayes regret becomes

$R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$4

The same work explicitly frames the problem as fast personalization to non-stationary users by combining offline and online learning (Hong et al., 2020).

A separate line studies finite-armed contextual bandits driven by a hidden Markov chain. The direct approach of “Staged LinUCB on estimated beliefs” uses contexts alone to estimate HMM beliefs $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$5, then performs ridge regression and UCB on the stacked feature $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$6 (Li et al., 9 Apr 2026). Under spectral identifiability, mixing, and belief-estimation assumptions, the paper proves a high-probability regret bound

$R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$7

with $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$8 and stage length $R_t\sim P(R\mid A_t,X_t,S_t;\theta_*)$9 (Li et al., 9 Apr 2026). It contrasts this with the “belief-linear” simplification associated with Nelson et al. (2022), for which $S_t$0 is attainable when beliefs are known and $S_t$1 with online belief estimation (Li et al., 9 Apr 2026). The paper’s point is not only algorithmic but also model-theoretic: it treats rewards as directly state-dependent rather than merely linear in posterior state probabilities (Li et al., 9 Apr 2026).

Latent auto-regressive dynamics provide another non-stationary mechanism. In the latent AR bandit model, a one-dimensional latent state $S_t$2 follows an $S_t$3 process, and arm rewards are linear in $S_t$4 (Trella et al., 2024). The key reduction rewrites rewards using past realized rewards and residuals, turning the problem into a linear bandit with corrupted contexts. A LinUCB-style algorithm on these corrupted contexts achieves

$S_t$5

under AR stability, boundedness, and a control condition on cumulative context corruption (Trella et al., 2024). This model is notable იმიტომ that it does not impose a sub-linear non-stationarity budget; instead it exploits a parameterized latent dynamics model (Trella et al., 2024).

State-model-free approaches avoid explicit latent-state inference altogether. “Adaptive Exploration for Latent-State Bandits” introduces LC-UCB based on lagged contexts $S_t$6, as well as RP-UCB and SP-UCB, which deliberately probe multiple arms to form state fingerprints (Jin et al., 4 Feb 2026). In a two-state setting, fixed-interval probing every $S_t$7 rounds yields

$S_t$8

which makes the exploration–staleness trade-off explicit (Jin et al., 4 Feb 2026). The adaptive variants trigger probing by residual, uncertainty, and hazard gates rather than a fixed schedule (Jin et al., 4 Feb 2026).

In restless bandits, MARBLE augments the RMAB model with a latent Markov environment $S_t$9, averages transitions and rewards over the stationary law $S_t\equiv S_1$0, and introduces Markov-Averaged Indexability (Amiri et al., 12 Nov 2025). Under MAI and two-timescale stochastic approximation, synchronous Q-learning with Whittle Indices converges almost surely to the optimal averaged Q-function and averaged Whittle indices (Amiri et al., 12 Nov 2025). The paper does not provide a finite-time regret bound and explicitly leaves sample complexity and regret as future work (Amiri et al., 12 Nov 2025). That limitation marks an important boundary in current latent-bandit theory: asymptotic convergence is available in some dynamical latent settings even when finite-time regret is not.

4. Linear, contextual, and identifiable latent representations

Latent structure is especially consequential in linear and contextual bandits. In stochastic linear bandits with latent heterogeneity, each arrival $S_t\equiv S_1$1 has an unobserved subgroup $S_t\equiv S_1$2, with logistic gating

$S_t\equiv S_1$3

and subgroup-specific linear rewards $S_t\equiv S_1$4 (Chen et al., 1 Feb 2025). The proposed phased EM-based algorithm alternates between regularized EM updates on past data and greedy action selection based on the predicted subgroup $S_t\equiv S_1$5 (Chen et al., 1 Feb 2025). The paper distinguishes two regret notions. The “strong regret” against an oracle that knows each realized subgroup is necessarily non-sub-linear because the latent-group classification error $S_t\equiv S_1$6 cannot vanish, and the upper bound includes a linear term proportional to $S_t\equiv S_1$7. By contrast, the “regular regret” against an oracle that knows only the deterministic components is

$S_t\equiv S_1$8

with a matching minimax lower bound $S_t\equiv S_1$9 (Chen et al., 1 Feb 2025). The same paper shows that a standard linear bandit fitting a single model incurs constant average regret that accumulates linearly over time under latent heterogeneity (Chen et al., 1 Feb 2025).

Offline data can also be used to learn a latent subspace before online interaction begins. In linear latent bandits, the unknown reward parameter is $b_t(s)=P(S_t=s\mid H_t,X_t)$0 with $b_t(s)=P(S_t=s\mid H_t,X_t)$1 and $b_t(s)=P(S_t=s\mid H_t,X_t)$2 (Kausik et al., 2024). SOLD learns the subspace from short offline trajectories, while LOCAL-UCB and ProBALL-UCB exploit it online (Kausik et al., 2024). LOCAL-UCB enjoys

$b_t(s)=P(S_t=s\mid H_t,X_t)$3

so the effective dimension decreases as the offline dataset size $b_t(s)=P(S_t=s\mid H_t,X_t)$4 grows (Kausik et al., 2024). This line treats latent bandits not only as a hidden-state problem but also as an offline–online dimension-reduction problem.

Identifiability results push the latent approach further. In identifiable latent bandits, each instance $b_t(s)=P(S_t=s\mid H_t,X_t)$5 has an unobserved latent vector $b_t(s)=P(S_t=s\mid H_t,X_t)$6, contexts are generated by a smooth invertible mixing function $b_t(s)=P(S_t=s\mid H_t,X_t)$7, and rewards are linear in $b_t(s)=P(S_t=s\mid H_t,X_t)$8 through arm-specific parameters $b_t(s)=P(S_t=s\mid H_t,X_t)$9 (Balcıoğlu et al., 2024). Training a representation $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$0 to classify which patient generated a context via multinomial logistic regression recovers

$$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$1

for an invertible matrix $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$2 and vector $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$3, under the stated assumptions (Balcıoğlu et al., 2024). After fitting reward parameters in this latent space, the online policy simply averages inferred latent vectors and acts greedily. The paper shows that this greedy policy has constant expected regret,

$$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$4

because the latent estimate converges at rate $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$5 and the best arm is identified after $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$6 additional pulls (Balcıoğlu et al., 2024). This is one of the strongest available latent-bandit statements: when a sufficient representation is identifiable from observational data, online exploration can collapse to a short identification phase.

Transfer under latent covariate shift adds a causal layer. In latent contextual bandits with hidden context $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$7, observed proxy $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$8, and a target domain with shifted $$b_{t+1}(s') \propto \sum_{s\in\mathcal S} b_t(s)\,P(s'|s;\phi_*)\,P(R_t|A_t,X_t,s;\theta_*) .$$9, naive transfer of $g_i\in\{1,2\}$0 can cause negative transfer (Deng et al., 27 Feb 2025). The transportability analysis yields

$g_i\in\{1,2\}$1

and the paper uses this formula directly in a binary case and approximately via a variational autoencoder in the high-dimensional proxy case (Deng et al., 27 Feb 2025). No closed-form regret bound is provided, but the empirical claim is that naive transfer can lead to $g_i\in\{1,2\}$2 regret, whereas transportability-guided transfer avoids that failure mode (Deng et al., 27 Feb 2025).

5. Low-rank, clustered, and collaborative latent bandits

Recommendation-style latent bandits often exploit shared structure across many users. In low-rank bandits with latent mixtures, user $g_i\in\{1,2\}$3 has an unknown mixture vector $g_i\in\{1,2\}$4 over latent classes, items have unknown class-reward vectors, and the expected reward matrix factorizes as $g_i\in\{1,2\}$5 with rank at most $g_i\in\{1,2\}$6 (Gopalan et al., 2016). The algorithm alternates between “ESTIMATE” mini-sessions, which collect random samples and recover item-side features via the Robust Tensor Power Method of Anandkumar et al. (2014b), and “OFUL” mini-sessions, which learn user-side mixture weights using the OFUL algorithm of Abbasi-Yadkori et al. (2011) (Gopalan et al., 2016). The expected regret after $g_i\in\{1,2\}$7 user interactions is

$g_i\in\{1,2\}$8

where $g_i\in\{1,2\}$9 is the number of users (Gopalan et al., 2016). A key technical contribution is a robustness property of OFUL for approximately linear reward models (Gopalan et al., 2016).

Cluster-structured latent bandits replace mixtures by discrete user clusters. Pal et al. consider $X$0 users partitioned into $X$1 unknown clusters, all users in a cluster sharing the same mean-reward vector across $X$2 arms (Pal et al., 2023). Their LATTICE algorithm operates in $X$3 phases, repeatedly collecting partial observations on user-group by active-arm submatrices, calling an offline low-rank matrix completion oracle, clustering estimated rows, and eliminating clearly suboptimal arms (Pal et al., 2023). Under the cluster-structure assumptions, LATTICE achieves the minimax optimal regret

$X$4

when the number of clusters is $X$5, and it requires only $X$6 calls to the offline matrix completion oracle (Pal et al., 2023). The paper emphasizes that this is the first algorithm with such a strong regret bound in that model (Pal et al., 2023).

Blocked collaborative bandits impose an additional budget constraint: each user–arm pair can be sampled at most $X$7 times. B-LATTICE extends the matrix-completion and latent-clustering strategy to this blocked setting (Pal et al., 2023). Under $X$8, the per-user regret is

$X$9

which the paper describes as the first sub-linear regret bound for the blocked collaborative-bandit problem and one that matches the minimax regret bound when $X_t\in\mathcal X$00 (Pal et al., 2023). The algorithm explicitly separates “exploit” rounds, which use high-confidence golden items, from “explore” rounds, which sample entries for matrix completion (Pal et al., 2023).

These collaborative models illustrate a distinctive role of latent bandits in modern recommendation theory. The latent variable need not be inferred only within one trajectory; it can be amortized across many users, learned by matrix completion or tensor decomposition, and then exploited online with a much smaller effective exploration burden. This suggests why the cold-start problem, personalized recommendation, and online collaborative filtering are recurring application domains throughout the literature (Zhou et al., 2016, Pal et al., 2023).

6. Relaxed priors, misspecification, and recurrent failure modes

A persistent theme is that classical latent-bandit assumptions may be too strong. “Latent Preference Bandits” assumes not a full reward model $X_t\in\mathcal X$01, but only that each latent state $X_t\in\mathcal X$02 induces an ordering $X_t\in\mathcal X$03 over actions (Mwai et al., 7 Aug 2025). The algorithm lpbTS samples a latent state from an approximate posterior, re-estimates state-wise mean vectors by weighted isotonic regression over the constraint set $X_t\in\mathcal X$04, and updates the posterior by constrained likelihood (Mwai et al., 7 Aug 2025). The paper gives an asymptotic lower bound adapted from Agrawal and Teneketzis (1989) and Maillard and Mannor (2014), along with a worst-case upper bound

$X_t\in\mathcal X$05

Its empirical claim is that preference-only modeling is competitive with latent bandits that have full knowledge of the reward distribution when that model is well specified, and superior when absolute reward scales differ across instances with the same latent state (Mwai et al., 7 Aug 2025).

Latent Order Bandits generalize this idea from total orders to partial orders $X_t\in\mathcal X$06 (Carlsson et al., 8 May 2026). The lobUCB algorithm projects empirical means onto the feasible cones $X_t\in\mathcal X$07, merges arms pooled by violated order constraints, and then computes UCB indices using pooled counts (Carlsson et al., 8 May 2026). For total orders with unique optimal arms, the paper gives both instance-dependent and instance-independent regret bounds, including

$X_t\in\mathcal X$08

in the instance-independent case (Carlsson et al., 8 May 2026). The associated posterior-sampling variant lobTS uses likelihood upper bounds over the union of feasible cones where a given arm is optimal (Carlsson et al., 8 May 2026). These formulations address a concrete limitation of full-prior latent bandits: a small number of latent states may capture relative preferences but not common reward scales.

A related but structurally different relaxation appears in transfer learning with latent continuity. There the latent quantity is a shared Lipschitz constant $X_t\in\mathcal X$09 across tasks, rather than a latent state per trajectory (Park et al., 2021). The estimator $X_t\in\mathcal X$10 uses the $X_t\in\mathcal X$11-th largest empirical Lipschitz estimate across prior tasks, plus a safety margin $X_t\in\mathcal X$12, and the plug-in algorithm $X_t\in\mathcal X$13 then runs a Lipschitz-aware bandit algorithm on the new task (Park et al., 2021). The paper’s main qualitative message is explicit: underestimating $X_t\in\mathcal X$14 is catastrophic and can cause linear regret, whereas overestimating $X_t\in\mathcal X$15 only increases the regret constant $X_t\in\mathcal X$16 (Park et al., 2021). This work broadens the latent-bandit idea from latent reward states to latent structural parameters shared across tasks.

Several recurrent failure modes are now well documented. First, always choosing the arm with highest expected reward under the current belief can leave the learner permanently uncertain because information-gathering arms may never be selected (Galozy et al., 2022). Second, in stochastic linear bandits with latent heterogeneity, collapsing all observations into a single linear model leads to bias and linear cumulative regret (Chen et al., 1 Feb 2025). Third, under covariate shift in latent contextual bandits, naively transferring all knowledge from the source domain can lead to negative transfer and $X_t\in\mathcal X$17 regret (Deng et al., 27 Feb 2025). Fourth, some dynamical latent-bandit frameworks, such as MARBLE, presently offer almost-sure convergence without finite-time regret guarantees (Amiri et al., 12 Nov 2025).

These limitations are not peripheral. They define much of the current research agenda. A plausible implication is that future latent-bandit methods will continue to move away from fully specified latent-state reward models toward weaker but more stable forms of transferable structure: identifiable representations, low-dimensional subspaces, order constraints, causal transport formulas, and adaptive information-gathering rules.