---
title: Kalman-UCB in Linear Bandits
url: https://www.emergentmind.com/topics/kalman-filter-upper-confidence-bound-kalman-ucb
type: topic
---

# Kalman-UCB in Linear Bandits

Searching arXiv for the specified paper and closely related bandit/state-space work.
Kalman Filter Upper Confidence Bound (Kalman-UCB) is an optimism-based action-selection method for a linear bandit in which the unknown linear parameter is not static but is instead the hidden state of a known Linear Gaussian Dynamical System (LGDS). In the formulation introduced in "A Control Theory inspired Exploration Method for a Linear Bandit driven by a Linear Gaussian Dynamical System" [2510.01364], the learner observes only the scalar reward associated with the chosen action, while the latent state evolves over time according to linear Gaussian dynamics. Kalman-UCB combines one-step Kalman state prediction with an upper-confidence-style exploration bonus derived from the predictive uncertainty of the latent state projected onto an action direction. The method is presented alongside IDEA as one of two algorithms for balancing exploration and exploitation in this nonstationary, partially observed bandit setting [2510.01364].

## 1. Problem formulation

The model begins from a standard linear bandit reward equation,
\[
X_t = \left\langle a_t, z \right\rangle + \eta_t,
\]
and replaces the fixed unknown parameter \(z\) by a time-varying latent state \(z_t\) generated by a known LGDS:
\[
\begin{cases}
z_{t+1}  = \Gamma z_t + \xi_t, \quad z_0 \sim \mathcal N(\mathbf 0,\Sigma_0)\\
X_t = \left\langle a_t, z_t\right\rangle + \eta_t .
\end{cases}
\]
Here \(a_t \in \mathcal A \subseteq \mathbb R^d\) is the chosen action, \(X_t\in\mathbb R\) is the observed reward, \(\Gamma\in\mathbb R^{d\times d}\) is the known state transition matrix, and the noises satisfy
\[
\xi_t \sim \mathcal N(\mathbf 0,Q), \qquad \eta_t \sim \mathcal N(0,\sigma^2),
\]
with \(Q \succeq \mathbf 0\) and \(\sigma > 0\) [2510.01364].

The action set is constrained by
\[
\mathcal{A} \subseteq \mathbb{S}^{d-1} \triangleq \left\{a_t \in \mathbb{R}^d \mid \left\Vert a_t\right\Vert_2 = 1\right\},
\]
so actions lie on the unit sphere, or on a subset of it. The horizon \(n\) is assumed known. The paper distinguishes between known quantities—\(\Gamma\), \(Q\), \(\sigma^2\), \(\mathcal A\), \(\Sigma_0\), and \(n\)—and unknown online quantities, notably the latent state \(z_t\), the rewards of unplayed actions, and the oracle-optimal action \(a_t^*\), which depends on \(z_t\) [2510.01364].

The learner’s objective is to maximize cumulative reward
\[
S_n = \sum_{t=1}^n X_t,
\]
equivalently expressed through the optimization problem
\[
\begin{array}{cc}
\underset{a_1,\dots,a_n \in \mathcal{A}}{\max} & \sum_{t=1}^n \left\langle a_t, z_t \right\rangle\\
\mbox{ s.t. } & \begin{cases}
z_{t+1}  = \Gamma z_t + \xi_t, ~z_0 \sim \mathcal{N}\left(0,\Sigma_0\right)\\
X_t  = \left\langle a_t, z_t\right\rangle + \eta_t
\end{cases}
\end{array}.
\]
Regret is defined as
\[
R_n \triangleq \sum_{t=1}^n \mathbb{E}\left[X_t^* - X_t\right],
\]
where
\[
a_t^* \triangleq \underset{a \in \mathcal{A}}{\arg\max} ~ \left\langle a, z_t \right\rangle .
\]
For the continuous unit-sphere case, this oracle action is the action aligned with \(z_t\) [2510.01364].

## 2. Motivation and structural departures from standard UCB

The principal departure from ordinary linear or arm-wise UCB is that the reward process is nonstationary and coupled across actions through a common latent state. The paper identifies three reasons standard UCB-style methods are insufficient in this setting: the rewards are nonstationary because \(z_t\) evolves; rewards across actions are coupled through the same latent state; and observing one action gives information about the hidden state, which can improve predictions for many other actions [2510.01364].

This structural coupling makes Kalman filtering natural, because the environment is exactly a linear Gaussian state-space model and the Kalman filter is the optimal one-step state predictor in mean-squared error. The predicted expected reward for action \(a\), conditioned on the history up to time \(t-1\), is
\[
\mathbb E[X_t \mid \mathcal F_{t-1}, a_t=a] = \left\langle a,\hat z_{t|t-1}\right\rangle,
\]
where \(\hat z_{t|t-1}\) is the one-step predictor produced by the filter [2510.01364].

The paper also emphasizes that dynamic programming is intractable because action selection and state estimation are coupled: the chosen action affects both immediate reward and future estimation quality through the covariance update. Its Theorem 1 states that the separation principle fails in the bandit setting with partial observation. This motivates approximate optimism-based methods such as Kalman-UCB [2510.01364].

The observation equation may be rewritten as
\[
X_t = a_t^\top z_t + \eta_t,
\]
which makes explicit that each action determines the measurement direction. This measurement geometry connects the bandit problem to observability. The paper defines the observability Gramian as
\[
\mathcal{O}\left(\Gamma,t_0,t_1\right) \triangleq \sum_{\tau=t_0}^{t_1} \left(\Gamma^\top\right)^\tau a_\tau a_\tau^\top \Gamma^\tau \in \mathbb{R}^{d \times d},
\]
and says the system is observable from round \(t_0\) to \(t_1\) if this Gramian is positive definite [2510.01364].

## 3. Algorithmic definition of Kalman-UCB

Kalman-UCB is introduced as an optimism-based policy of the form
\[
a_t = \underset{a \in \mathcal{A}}{\arg\max} ~ \left\langle a, \hat{z}_{t|t-1}\right\rangle + u_t\left(a\mid P_{t|t-1}\right),
\]
with the specific Kalman-UCB action-selection rule
\[
a_t = \underset{a \in \mathcal{A}}{\arg\max} \left\langle a, \hat{z}_{t|t-1} \right\rangle + \sqrt{a^\top P_{t|t-1}a}.
\]
This is Algorithm 1 in the paper [2510.01364].

The exploration bonus is derived from the confidence statement
\[
\left\vert X_t-\left\langle a, \hat{z}_{t|t-1}\right\rangle\right\vert \leq \sqrt{\left(a^\top P_{t|t-1} a+\sigma^2\right)\log\left(1/\delta\right)}
\]
with probability at least \(1-\delta\), \(\delta\in(0,1)\). The corresponding confidence-parameterized score is
\[
a_{t+1} = \underset{a \in \mathcal{A}}{\arg\max} \left\langle a, \hat{z}_{t|t-1}\right\rangle \ + \sqrt{\left(a^\top P_{t|t-1} a\right)\log\left(1/\delta\right)}.
\]
The paper then removes the term \(\sigma^2\log(1/\delta)\) because it is independent of \(a\), and Algorithm 1 uses the simplified score with exploration coefficient effectively equal to \(1\) [2510.01364].

The uncertainty quantity used by Kalman-UCB is the predictive variance of the latent-state contribution to reward,
\[
a^\top P_{t|t-1} a.
\]
If measurement noise is included, the predictive reward variance is
\[
a^\top P_{t|t-1} a + \sigma^2,
\]
but action selection depends only on
\[
\sqrt{a^\top P_{t|t-1} a},
\]
because \(\sigma^2\) is action-independent. Accordingly, the confidence term is based on predictive state uncertainty projected along action direction \(a\), rather than on posterior variance after update [2510.01364].

Operationally, the procedure is: predict the hidden state \(z_t\) using the Kalman filter to obtain \(\hat z_{t|t-1}\) and \(P_{t|t-1}\); for every candidate action compute the predicted reward \(\langle a,\hat z_{t|t-1}\rangle\) and bonus \(\sqrt{a^\top P_{t|t-1}a}\); choose the action with maximum optimistic score; observe only the reward for that action; and update the hidden-state posterior with the scalar observation [2510.01364].

## 4. Kalman filter recursion and state-estimation role

Kalman-UCB uses the Kalman filter in one-step predictor form. The paper gives the recursion
\[
\begin{cases}
\hat{z}_{t+1|t} & = \Gamma \hat{z}_{t|t} + \Gamma  K_t \left(X_t - \left\langle a_t, \hat{z}_{t|t-1}\right\rangle\right)\\
P_{t+1|t} & = g\left(P_{t|t-1},a_t\right)\\
K_t & = P_{t|t-1} a_t\left(a_t^\top P_{t|t-1} a_t + \sigma^2\right)^{-1}
\end{cases},
\]
with
\[
g\left(P_{t|t-1},a_t\right) \triangleq \Gamma P_{t|t-1} \Gamma^\top + Q \ - \Gamma P_{t|t-1} a_t \left(a_t^\top P_{t|t-1} a_t + \sigma\right)^{-1} a_t^\top P_{t|t-1} \Gamma^\top .
\]
The paper also presents Assumption 2: the matrix pair \((\Gamma,Q^{1/2})\) is controllable, and states that this is necessary for existence of the Kalman filter [2510.01364].

The filter quantities are defined in the standard way. The filtered estimate is
\[
\hat z_{t|t}\triangleq \mathbb E[z_t\mid \mathcal F_t],
\]
the one-step predictor is \(\hat z_{t|t-1}\), the prediction error is
\[
e_{t|t-1}\triangleq z_t-\hat z_{t|t-1},
\]
and \(P_{t|t-1}\) is the covariance of \(e_{t|t-1}\). The paper further states that \(P_{t|t-1}\) converges if \((\Gamma,a_t^\top)\) is detectable and \((\Gamma,Q^{1/2})\) is controllable [2510.01364].

Reward prediction follows directly from the latent-state estimate:
\[
\hat r_t(a) = \left\langle a,\hat z_{t|t-1}\right\rangle.
\]
The predictive variance of the latent component is
\[
\mathrm{Var}\!\left(a^\top z_t \mid \mathcal F_{t-1}\right) = a^\top P_{t|t-1}a,
\]
while the variance of the observed reward is
\[
\mathrm{Var}\!\left(X_t \mid \mathcal F_{t-1},a_t=a\right) = a^\top P_{t|t-1}a + \sigma^2.
\]
This is the direct statistical source of the Kalman-UCB exploration bonus [2510.01364].

A typographical issue is explicitly identifiable in the formulas as printed: \(K_t\) uses denominator \(a_t^\top P_{t|t-1} a_t + \sigma^2\), while \(g(\cdot)\) is written with \(a_t^\top P_{t|t-1} a_t + \sigma\). This suggests a typographical inconsistency, since the confidence derivation and the \(K_t\) expression both use \(+\sigma^2\) [2510.01364].

## 5. Theoretical properties and regret status

The theoretical treatment does not include a standalone regret theorem specifically for Kalman-UCB of the type common in stationary bandit analysis. Instead, the paper provides a general regret upper bound for any optimism-based method, a Kalman-UCB-specific lemma on periodic exploration, confidence-bound motivation from predictive Gaussian uncertainty, and lower bounds showing that regret is inherently linear in the LGDS environment [2510.01364].

For any policy of the form
\[
a_t = \arg\max_{a\in\mathcal A}\ \langle a,\hat z_{t|t-1}\rangle + u_t(a\mid P_{t|t-1}),
\]
Theorem 4 gives
\[
R_n \leq \sum_{t = 1}^n u_t\left(a_t\mid P_{t|t-1}\right) - u_t\left(a_t^*\mid P_{t|t-1}\right) \ + 2 \left\Vert e_{t|t-1}\right\Vert_2 .
\]
For Kalman-UCB,
\[
u_t(a\mid P_{t|t-1}) = \sqrt{a^\top P_{t|t-1}a},
\]
or in the confidence-parameterized variant,
\[
u_t(a\mid P_{t|t-1}) = \sqrt{(a^\top P_{t|t-1}a)\log(1/\delta)}.
\]
The paper notes that because \(\|e_{t|t-1}\|_2\ge 0\), the upper bound increases at least linearly [2510.01364].

The lower-bound analysis is environment-level rather than algorithm-specific. For the continuous unit-sphere action set, the paper proves
\[
R_n \geq \sum_{t=1}^n \mathbb{E}\left[\sqrt{\nu_t^\top Z_t \nu_t}\right] - \mathbb{E}\left[\sqrt{\hat{\nu}_t^\top \left(Z_t - P'\right)\hat{\nu}_t}\right],
\]
where
\[
Z_t \triangleq \mathbb{E}[z_t z_t^\top].
\]
In the stable steady-state case, it gives
\[
R_n \geq \sum_{t=1}^n \mathbb{E}\left[\sqrt{\nu_t^\top Z \nu_t}\right] - \mathbb{E}\left[\sqrt{\hat{\nu}_t^\top \left(Z - P\right)\hat{\nu}_t}\right].
\]
These results are used to justify why logarithmic regret is not the target in this setting [2510.01364].

The most specific theoretical statement about Kalman-UCB is Lemma 3. Let \(P_a\) solve the algebraic Riccati equation
\[
P_a = g(P_a,a).
\]
If for every action \(a\in\mathcal A\) there exists another action \(a'\in\mathcal A\) such that
\[
\sqrt{a^\top P_a a} \le \sqrt{(a')^\top P_a a'},
\]
then every action \(a\in\mathcal A\) will be sampled periodically. The paper interprets this as an implicit periodic exploration schedule induced by the covariance dynamics under repeated measurements with one action [2510.01364].

## 6. Relation to IDEA, observability, and performance-prediction metric

Kalman-UCB is presented together with Information filter Directed Exploration Action-selection (IDEA). The two methods differ in what they regard as the relevant exploration signal. Kalman-UCB selects actions by
\[
\left\langle a, \hat{z}_{t|t-1} \right\rangle + \sqrt{a^\top P_{t|t-1}a},
\]
so it explores based on reward prediction uncertainty for that action. IDEA instead uses
\[
\left\langle a, \hat{z}_{t|t-1}\right\rangle \ + \sqrt{\mbox{tr}\left(\frac{\Gamma P_{t|t-1} aa^\top P_{t|t-1} \Gamma^\top}{a^\top P_{t|t-1} a + \sigma^2}\right)},
\]
and therefore values how much an action reduces future state-prediction error through Kalman feedback [2510.01364].

The paper states that Kalman-UCB explores actions with the highest reward prediction error, and that this can be advantageous if the LGDS lacks an observable action \(a\in\mathcal A\). IDEA is said to be beneficial if there exists an action that minimizes reward prediction error for all other actions, namely a single especially informative action. Observability is thus central to IDEA and only indirect for Kalman-UCB: the Kalman-UCB bonus is local, \(a^\top P a\), while IDEA’s exploration term reflects covariance reduction after measurement [2510.01364].

To predict which method will perform better, the paper develops a metric based on a Wasserstein-type distance between the optimism-based action-selection distribution and the oracle distribution. It defines
\[
\mu_i\left(P\right) \triangleq \begin{pmatrix} u_t\left(a_i \mid P \right) - u_t \left(a_1 \mid P\right) \\
\vdots \\
u_t\left(a_i \mid P \right) - u_t \left(a_{k-1} \mid P\right) \\
\mathbf{0}_{k-1} \end{pmatrix}
\]
and
\[
\hat{\Sigma}_{i,j}\left(P\right) \triangleq \begin{pmatrix}
A_i \left(Z - P\right) A_i^\top & A_i\left(Z - P\right) A_j^\top \\
A_j \left(Z - P\right) A_i^\top & A_j Z A_j^\top
\end{pmatrix}.
\]
The metric is
\[
\phi\left(i,j \mid P\right) = \left\Vert \mu_i \right\Vert_2 + \mbox{tr}\left(\Sigma_{i,j} +\hat{\Sigma}_{i,j}\left(P\right) \right) \ - 2\mbox{tr}\left(\left(\hat{\Sigma}_{i,j}\left(P\right)^{1/2} \Sigma_{i,j}\hat{\Sigma}_{i,j}\left(P\right)^{1/2}\right)^{1/2} \right),
\]
and performance is summarized by the interval
\[
\begin{pmatrix}
\min_{i \neq j, a \in \mathcal{A}} \phi\left(i,j \mid P_a\right),\ 
\max_{i \neq j ,a \in \mathcal{A}} \phi\left(i,j \mid P_a\right)
\end{pmatrix},
\]
where \(P_a\) is the steady-state covariance solving \(P_a=g(P_a,a)\). The paper interprets \(\phi(i,j\mid P)\) as quantifying how much the optimism term distorts action selection away from the oracle, with smaller \(\phi\) indicating less harmful perturbation [2510.01364].

## 7. Empirical evaluation, applications, and limitations

The experiments compare Kalman-UCB with IDEA, KODE, UCB, Rexp3, OFUL, Random Agent, and a benchmark labeled Kalman Oracle Action-selection. The environment generation uses \(k=d=10\), \(10^3\) random LGDS instances per distribution, 10 interactions per algorithm per LGDS, horizon \(n=10^3\), and an initial state prepared by running the LGDS for \(10^4\) iterations. Random system entries are drawn independently from Gaussian, Uniform, Exponential, Cauchy, or Bernoulli distributions; if \(T\sim p\), then the state transition matrix is normalized to spectral radius \(0.9\) by
\[
\Gamma = \left(0.9/\rho(T)\right)T.
\]
Noise is constructed as
\[
Q = RR^\top,\qquad \sigma^2=r^2,
\]
with \(R\sim p\) and \(r\sim p\) [2510.01364].

The principal reported metric is fractional difference of regret relative to Kalman Oracle Action-selection, where lower is better and zero would match oracle performance. The table entries for Kalman-UCB are:

| Method | Gaussian | Cauchy | Uniform |
|---|---:|---:|---:|
| Kalman UCB | 1.52 (0.95) | 2.40 (12.44) | 0.90 (0.45) |

| Method | Bernoulli | Exponential |
|---|---:|---:|
| Kalman UCB | 0.44 (0.22) | 0.57 (0.26) |

For IDEA, the corresponding values are 1.37 (0.86), 1.82 (8.25), 0.84 (0.43), 0.11 (0.08), and 0.08 (0.07) [2510.01364].

The paper interprets these results as showing that Kalman-UCB is significantly better than conventional bandit baselines such as UCB, OFUL, Rexp3, and Random Agent, while IDEA is usually better than Kalman-UCB, especially for Bernoulli- and Exponential-generated systems. For Gaussian, Cauchy, and Uniform distributions, Kalman-UCB and IDEA are often close. The paper attributes the superiority of Kalman-UCB and IDEA over classic bandits to the use of the Kalman-filter state predictor \(\langle a,\hat z_{t|t-1}\rangle\), which has much lower error than arm-wise statistics [2510.01364].

A direct Kalman-UCB versus IDEA scatter plot shows that the two methods are often comparable for Gaussian, Cauchy, and Uniform systems, whereas IDEA consistently outperforms Kalman-UCB for Bernoulli and Exponential systems. This is reported to match the comparison metric based on \(\phi\) [2510.01364].

The robustness analysis perturbs the model used by the algorithms through \(\Gamma\), the actions \(a\in\mathcal A\), and \(Q\). For perturbing \(\Gamma\), the reported median performance decrease is \(9\%\) for KODE, \(18\%\) for IDEA, and \(23\%\) for Kalman-UCB. For perturbing actions, the reported decrease is \(47\%\) for KODE, \(48\%\) for IDEA, and \(40\%\) for Kalman-UCB. The paper interprets this as indicating that Kalman-UCB is less robust than KODE and IDEA to \(\Gamma\)-misspecification, but relatively less sensitive to action perturbation than IDEA and KODE [2510.01364].

The paper also motivates the framework through hyperparameter optimization. In that interpretation, each action corresponds to a hyperparameter configuration, the hidden state \(z_t\) captures latent training dynamics or performance factors over training time, and the reward
\[
X_t=\langle a_t,z_t\rangle+\eta_t
\]
is the observed performance of the chosen configuration at round \(t\). The paper states that large action spaces are a major problem in hyperparameter optimization because they hinder methods inspired by optimism in the face of uncertainty, which must explore each action to lower reward prediction uncertainty. This is one reason IDEA may be preferable there [2510.01364].

Kalman-UCB’s main limitations in the paper are explicit. It requires known system parameters \((\Gamma,Q,\sigma,\mathcal A)\); it can be less effective than IDEA when one action strongly improves observability of the whole state; large action spaces can disadvantage optimism-based methods because they tend to reduce uncertainty action by action; and performance degrades under system misspecification [2510.01364]. A plausible implication is that Kalman-UCB is most natural when the LGDS specification is reliable and when per-action predictive uncertainty is a more useful exploration signal than global state observability.

Source: https://www.emergentmind.com/topics/kalman-filter-upper-confidence-bound-kalman-ucb