Kalman-UCB in Linear Bandits

• Kalman-UCB is an optimism-based method that uses Kalman filtering to predict latent state dynamics in linear bandits.
• It computes an upper confidence bonus from the predictive variance of the latent state projected along actions to drive exploration in nonstationary, partially observed settings.
• The method balances immediate reward prediction with long-term state estimation, outperforming traditional bandit algorithms when system parameters are reliably known.

Searching arXiv for the specified paper and closely related bandit/state-space work. Kalman Filter Upper Confidence Bound (Kalman-UCB) is an optimism-based action-selection method for a linear bandit in which the unknown linear parameter is not static but is instead the hidden state of a known Linear Gaussian Dynamical System (LGDS). In the formulation introduced in "A Control Theory inspired Exploration Method for a Linear Bandit driven by a Linear Gaussian Dynamical System" (Gornet et al., 1 Oct 2025), the learner observes only the scalar reward associated with the chosen action, while the latent state evolves over time according to linear Gaussian dynamics. Kalman-UCB combines one-step Kalman state prediction with an upper-confidence-style exploration bonus derived from the predictive uncertainty of the latent state projected onto an action direction. The method is presented alongside IDEA as one of two algorithms for balancing exploration and exploitation in this nonstationary, partially observed bandit setting (Gornet et al., 1 Oct 2025).

1. Problem formulation

The model begins from a standard linear bandit reward equation,

$X_t = \left\langle a_t, z \right\rangle + \eta_t,$

and replaces the fixed unknown parameter $z$ by a time-varying latent state $z_t$ generated by a known LGDS: $$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$ Here $a_t \in \mathcal A \subseteq \mathbb R^d$ is the chosen action, $X_t\in\mathbb R$ is the observed reward, $\Gamma\in\mathbb R^{d\times d}$ is the known state transition matrix, and the noises satisfy

$$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$

with $Q \succeq \mathbf 0$ and $\sigma > 0$ (Gornet et al., 1 Oct 2025).

The action set is constrained by

$z$0

so actions lie on the unit sphere, or on a subset of it. The horizon $z$1 is assumed known. The paper distinguishes between known quantities—$z$2, $z$3, $z$4, $z$5, $z$6, and $z$7—and unknown online quantities, notably the latent state $z$8, the rewards of unplayed actions, and the oracle-optimal action $z$9, which depends on $z_t$0 (Gornet et al., 1 Oct 2025).

The learner’s objective is to maximize cumulative reward

$z_t$1

equivalently expressed through the optimization problem

$z_t$2

Regret is defined as

$z_t$3

where

$z_t$4

For the continuous unit-sphere case, this oracle action is the action aligned with $z_t$5 (Gornet et al., 1 Oct 2025).

2. Motivation and structural departures from standard UCB

The principal departure from ordinary linear or arm-wise UCB is that the reward process is nonstationary and coupled across actions through a common latent state. The paper identifies three reasons standard UCB-style methods are insufficient in this setting: the rewards are nonstationary because $z_t$6 evolves; rewards across actions are coupled through the same latent state; and observing one action gives information about the hidden state, which can improve predictions for many other actions (Gornet et al., 1 Oct 2025).

This structural coupling makes Kalman filtering natural, because the environment is exactly a linear Gaussian state-space model and the Kalman filter is the optimal one-step state predictor in mean-squared error. The predicted expected reward for action $z_t$7, conditioned on the history up to time $z_t$8, is

$z_t$9

where $$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$0 is the one-step predictor produced by the filter (Gornet et al., 1 Oct 2025).

The paper also emphasizes that dynamic programming is intractable because action selection and state estimation are coupled: the chosen action affects both immediate reward and future estimation quality through the covariance update. Its Theorem 1 states that the separation principle fails in the bandit setting with partial observation. This motivates approximate optimism-based methods such as Kalman-UCB (Gornet et al., 1 Oct 2025).

The observation equation may be rewritten as

$$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$1

which makes explicit that each action determines the measurement direction. This measurement geometry connects the bandit problem to observability. The paper defines the observability Gramian as

$$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$2

and says the system is observable from round $$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$3 to $$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$4 if this Gramian is positive definite (Gornet et al., 1 Oct 2025).

3. Algorithmic definition of Kalman-UCB

Kalman-UCB is introduced as an optimism-based policy of the form

$$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$5

with the specific Kalman-UCB action-selection rule

$$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$6

This is Algorithm 1 in the paper (Gornet et al., 1 Oct 2025).

The exploration bonus is derived from the confidence statement

$$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$7

with probability at least $$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$8, $$\begin{cases} z_{t+1} = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\ X_t = \left\langle a_t, z_t\right\rangle + \eta_t . \end{cases}$$9. The corresponding confidence-parameterized score is

$a_t \in \mathcal A \subseteq \mathbb R^d$0

The paper then removes the term $a_t \in \mathcal A \subseteq \mathbb R^d$1 because it is independent of $a_t \in \mathcal A \subseteq \mathbb R^d$2, and Algorithm 1 uses the simplified score with exploration coefficient effectively equal to $a_t \in \mathcal A \subseteq \mathbb R^d$3 (Gornet et al., 1 Oct 2025).

The uncertainty quantity used by Kalman-UCB is the predictive variance of the latent-state contribution to reward,

$a_t \in \mathcal A \subseteq \mathbb R^d$4

If measurement noise is included, the predictive reward variance is

$a_t \in \mathcal A \subseteq \mathbb R^d$5

but action selection depends only on

$a_t \in \mathcal A \subseteq \mathbb R^d$6

because $a_t \in \mathcal A \subseteq \mathbb R^d$7 is action-independent. Accordingly, the confidence term is based on predictive state uncertainty projected along action direction $a_t \in \mathcal A \subseteq \mathbb R^d$8, rather than on posterior variance after update (Gornet et al., 1 Oct 2025).

Operationally, the procedure is: predict the hidden state $a_t \in \mathcal A \subseteq \mathbb R^d$9 using the Kalman filter to obtain $X_t\in\mathbb R$0 and $X_t\in\mathbb R$1; for every candidate action compute the predicted reward $X_t\in\mathbb R$2 and bonus $X_t\in\mathbb R$3; choose the action with maximum optimistic score; observe only the reward for that action; and update the hidden-state posterior with the scalar observation (Gornet et al., 1 Oct 2025).

4. Kalman filter recursion and state-estimation role

Kalman-UCB uses the Kalman filter in one-step predictor form. The paper gives the recursion

$X_t\in\mathbb R$4

with

$X_t\in\mathbb R$5

The paper also presents Assumption 2: the matrix pair $X_t\in\mathbb R$6 is controllable, and states that this is necessary for existence of the Kalman filter (Gornet et al., 1 Oct 2025).

The filter quantities are defined in the standard way. The filtered estimate is

$X_t\in\mathbb R$7

the one-step predictor is $X_t\in\mathbb R$8, the prediction error is

$X_t\in\mathbb R$9

and $\Gamma\in\mathbb R^{d\times d}$0 is the covariance of $\Gamma\in\mathbb R^{d\times d}$1. The paper further states that $\Gamma\in\mathbb R^{d\times d}$2 converges if $\Gamma\in\mathbb R^{d\times d}$3 is detectable and $\Gamma\in\mathbb R^{d\times d}$4 is controllable (Gornet et al., 1 Oct 2025).

Reward prediction follows directly from the latent-state estimate: $\Gamma\in\mathbb R^{d\times d}$5 The predictive variance of the latent component is

$\Gamma\in\mathbb R^{d\times d}$6

while the variance of the observed reward is

$\Gamma\in\mathbb R^{d\times d}$7

This is the direct statistical source of the Kalman-UCB exploration bonus (Gornet et al., 1 Oct 2025).

A typographical issue is explicitly identifiable in the formulas as printed: $\Gamma\in\mathbb R^{d\times d}$8 uses denominator $\Gamma\in\mathbb R^{d\times d}$9, while $$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$0 is written with $$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$1. This suggests a typographical inconsistency, since the confidence derivation and the $$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$2 expression both use $$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$3 (Gornet et al., 1 Oct 2025).

5. Theoretical properties and regret status

The theoretical treatment does not include a standalone regret theorem specifically for Kalman-UCB of the type common in stationary bandit analysis. Instead, the paper provides a general regret upper bound for any optimism-based method, a Kalman-UCB-specific lemma on periodic exploration, confidence-bound motivation from predictive Gaussian uncertainty, and lower bounds showing that regret is inherently linear in the LGDS environment (Gornet et al., 1 Oct 2025).

For any policy of the form

$$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$4

Theorem 4 gives

$$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$5

For Kalman-UCB,

$$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$6

or in the confidence-parameterized variant,

$$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$7

The paper notes that because $$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$8, the upper bound increases at least linearly (Gornet et al., 1 Oct 2025).

The lower-bound analysis is environment-level rather than algorithm-specific. For the continuous unit-sphere action set, the paper proves

$$\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),$$9

where

$Q \succeq \mathbf 0$0

In the stable steady-state case, it gives

$Q \succeq \mathbf 0$1

These results are used to justify why logarithmic regret is not the target in this setting (Gornet et al., 1 Oct 2025).

The most specific theoretical statement about Kalman-UCB is Lemma 3. Let $Q \succeq \mathbf 0$2 solve the algebraic Riccati equation

$Q \succeq \mathbf 0$3

If for every action $Q \succeq \mathbf 0$4 there exists another action $Q \succeq \mathbf 0$5 such that

$Q \succeq \mathbf 0$6

then every action $Q \succeq \mathbf 0$7 will be sampled periodically. The paper interprets this as an implicit periodic exploration schedule induced by the covariance dynamics under repeated measurements with one action (Gornet et al., 1 Oct 2025).

6. Relation to IDEA, observability, and performance-prediction metric

Kalman-UCB is presented together with Information filter Directed Exploration Action-selection (IDEA). The two methods differ in what they regard as the relevant exploration signal. Kalman-UCB selects actions by

$Q \succeq \mathbf 0$8

so it explores based on reward prediction uncertainty for that action. IDEA instead uses

$Q \succeq \mathbf 0$9

and therefore values how much an action reduces future state-prediction error through Kalman feedback (Gornet et al., 1 Oct 2025).

The paper states that Kalman-UCB explores actions with the highest reward prediction error, and that this can be advantageous if the LGDS lacks an observable action $\sigma > 0$0. IDEA is said to be beneficial if there exists an action that minimizes reward prediction error for all other actions, namely a single especially informative action. Observability is thus central to IDEA and only indirect for Kalman-UCB: the Kalman-UCB bonus is local, $\sigma > 0$1, while IDEA’s exploration term reflects covariance reduction after measurement (Gornet et al., 1 Oct 2025).

To predict which method will perform better, the paper develops a metric based on a Wasserstein-type distance between the optimism-based action-selection distribution and the oracle distribution. It defines

$\sigma > 0$2

and

$\sigma > 0$3

The metric is

$\sigma > 0$4

and performance is summarized by the interval

$\sigma > 0$5

where $\sigma > 0$6 is the steady-state covariance solving $\sigma > 0$7. The paper interprets $\sigma > 0$8 as quantifying how much the optimism term distorts action selection away from the oracle, with smaller $\sigma > 0$9 indicating less harmful perturbation (Gornet et al., 1 Oct 2025).

7. Empirical evaluation, applications, and limitations

The experiments compare Kalman-UCB with IDEA, KODE, UCB, Rexp3, OFUL, Random Agent, and a benchmark labeled Kalman Oracle Action-selection. The environment generation uses $z$00, $z$01 random LGDS instances per distribution, 10 interactions per algorithm per LGDS, horizon $z$02, and an initial state prepared by running the LGDS for $z$03 iterations. Random system entries are drawn independently from Gaussian, Uniform, Exponential, Cauchy, or Bernoulli distributions; if $z$04, then the state transition matrix is normalized to spectral radius $z$05 by

$z$06

Noise is constructed as

$z$07

with $z$08 and $z$09 (Gornet et al., 1 Oct 2025).

The principal reported metric is fractional difference of regret relative to Kalman Oracle Action-selection, where lower is better and zero would match oracle performance. The table entries for Kalman-UCB are:

Method	Gaussian	Cauchy	Uniform
Kalman UCB	1.52 (0.95)	2.40 (12.44)	0.90 (0.45)

Method	Bernoulli	Exponential
Kalman UCB	0.44 (0.22)	0.57 (0.26)

For IDEA, the corresponding values are 1.37 (0.86), 1.82 (8.25), 0.84 (0.43), 0.11 (0.08), and 0.08 (0.07) (Gornet et al., 1 Oct 2025).

The paper interprets these results as showing that Kalman-UCB is significantly better than conventional bandit baselines such as UCB, OFUL, Rexp3, and Random Agent, while IDEA is usually better than Kalman-UCB, especially for Bernoulli- and Exponential-generated systems. For Gaussian, Cauchy, and Uniform distributions, Kalman-UCB and IDEA are often close. The paper attributes the superiority of Kalman-UCB and IDEA over classic bandits to the use of the Kalman-filter state predictor $z$10, which has much lower error than arm-wise statistics (Gornet et al., 1 Oct 2025).

A direct Kalman-UCB versus IDEA scatter plot shows that the two methods are often comparable for Gaussian, Cauchy, and Uniform systems, whereas IDEA consistently outperforms Kalman-UCB for Bernoulli and Exponential systems. This is reported to match the comparison metric based on $z$11 (Gornet et al., 1 Oct 2025).

The robustness analysis perturbs the model used by the algorithms through $z$12, the actions $z$13, and $z$14. For perturbing $z$15, the reported median performance decrease is $z$16 for KODE, $z$17 for IDEA, and $z$18 for Kalman-UCB. For perturbing actions, the reported decrease is $z$19 for KODE, $z$20 for IDEA, and $z$21 for Kalman-UCB. The paper interprets this as indicating that Kalman-UCB is less robust than KODE and IDEA to $z$22-misspecification, but relatively less sensitive to action perturbation than IDEA and KODE (Gornet et al., 1 Oct 2025).

The paper also motivates the framework through hyperparameter optimization. In that interpretation, each action corresponds to a hyperparameter configuration, the hidden state $z$23 captures latent training dynamics or performance factors over training time, and the reward

$z$24

is the observed performance of the chosen configuration at round $z$25. The paper states that large action spaces are a major problem in hyperparameter optimization because they hinder methods inspired by optimism in the face of uncertainty, which must explore each action to lower reward prediction uncertainty. This is one reason IDEA may be preferable there (Gornet et al., 1 Oct 2025).

Kalman-UCB’s main limitations in the paper are explicit. It requires known system parameters $z$26; it can be less effective than IDEA when one action strongly improves observability of the whole state; large action spaces can disadvantage optimism-based methods because they tend to reduce uncertainty action by action; and performance degrades under system misspecification (Gornet et al., 1 Oct 2025). A plausible implication is that Kalman-UCB is most natural when the LGDS specification is reliable and when per-action predictive uncertainty is a more useful exploration signal than global state observability.