---
title: 'Matrix Contextual Bandits: Graph & Low-Rank Models'
url: https://www.emergentmind.com/topics/matrix-contextual-bandit-cb
type: topic
---

# Matrix Contextual Bandits: Graph & Low-Rank Models

Matrix contextual bandit (CB) denotes an online decision-making framework in which contextual information is represented as a matrix and the reward model is governed by an unknown low-rank parameter matrix. It is described as an extension of the well-known multi-armed bandit and is motivated by sequential decision-making scenarios in which low-rank structure is intrinsic, including recommendation-style settings with user and item features [2507.17528]. In the formulation with graph information, matrix CB incorporates not only low-rank structure but also user–user and item–item similarity graphs through Laplacian regularization, while in the inference-oriented formulation it also supports valid online statistical inference under adaptive data collection by combining low-rank estimation with online debiasing [2212.11385].

## 1. Problem formulation and reward models

In the recommendation-style formulation, at each round \(t=1,2,\dots,T\), the learner observes a user feature vector \(p_t\in\mathbb{R}^{d_1}\) and an item feature vector \(q_t\in\mathbb{R}^{d_2}\), and combines them into an action matrix
\[
X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.
\]
The learner then chooses one of \(n\) possible user–item pairs and receives a noisy reward \(y_t\). The expected reward is modeled by a generalized linear model,
\[
\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),
\]
where \(\Theta^*\in\mathbb{R}^{d_1\times d_2}\) is an unknown low-rank matrix with rank \(r\ll \min\{d_1,d_2\}\), \(\langle A,B\rangle=\operatorname{tr}(A^\top B)\) is the Frobenius inner product, and \(\mu\) is a known inverse-link function such as logistic, Poisson, or the identity [2507.17528].

The objective in this setting is cumulative regret minimization. With
\[
X^*=\arg\max_X \mu(\langle X,\Theta^*\rangle),
\]
the cumulative regret is defined as
\[
R_T=\sum_{t=1}^T\Bigl[\mu\bigl(\langle X^*,\Theta^*\rangle\bigr)-\mu\bigl(\langle X_t,\Theta^*\rangle\bigr)\Bigr].
\]
This formulation makes explicit that action quality is determined through a matrix inner product against the latent low-rank parameter [2507.17528].

A related formulation studies online decision-making with matrix context in a binary-action setting. At each round \(t=1,\dots,n\), an agent observes a collection of \(K\) arms, with \(K=2\) and \(i\in\{0,1\}\), where arm \(i\) is associated with a matrix context \(X_{t,i}\in\mathbb{R}^{p\times q}\). Upon choosing arm \(i\), the observed reward satisfies
\[
y_t=r_{t,i}=\langle \Theta^*,X_{t,i}\rangle+\epsilon_t,
\]
with \(\operatorname{rank}(\Theta^*)\le r\). In the binary-arm setting, one may think \(X_{t,0}=X_{t,1}=X_t\) [2212.11385].

These two formulations share the same structural core: matrix-valued context, a low-rank latent parameter, and sequential arm selection. A plausible implication is that “matrix contextual bandit” is best viewed not as a single algorithm, but as a class of structured contextual bandit problems in which bilinear or matrix-structured covariates permit low-dimensional statistical regularization.

## 2. Low-rank structure and graph information

A central premise of matrix CB is that the unknown parameter matrix is low rank. In the generalized low-rank formulation, this is encoded by assuming \(\Theta^*\in\mathbb{R}^{d_1\times d_2}\) has rank \(r\ll \min\{d_1,d_2\}\), and estimation uses the nuclear norm
\[
\|\Theta\|_*=\sum_i \sigma_i(\Theta),
\]
which promotes \(\operatorname{rank}(\Theta)\le r\) [2507.17528].

The graph-augmented model adds side information through a user–user graph \(G_u\) with Laplacian \(L_u\in\mathbb{R}^{d_1\times d_1}\) and an item–item graph \(G_v\) with Laplacian \(L_v\in\mathbb{R}^{d_2\times d_2}\). Roughly speaking, if two users \(i,j\) are connected in \(G_u\), their corresponding rows of \(\Theta^*\) should vary smoothly; likewise for connected items in \(G_v\) [2507.17528].

To exploit both sources of structure, the offline estimator is defined by the convex program
\[
\widehat\Theta
=\arg\min_{\Theta\in\mathbb{R}^{d_1\times d_2}}
\frac1n\sum_{t=1}^n\bigl(y_t-\langle X_t,\Theta\rangle\bigr)^2
+\lambda_* \|\Theta\|_*
+\lambda_u \operatorname{tr}(\Theta^\top L_u\Theta)
+\lambda_v \operatorname{tr}(\Theta L_v\Theta^\top).
\]
The two Laplacian terms have explicit interpretations:
\[
\operatorname{tr}(\Theta^\top L_u\Theta)
\]
penalizes changes across the user graph, and
\[
\operatorname{tr}(\Theta L_v\Theta^\top)
\]
penalizes changes across the item graph. Each Laplacian term enforces smoothness: if users \(i\) and \(j\) are connected, then row \(i\) and row \(j\) of \(\Theta\) are encouraged to be similar; likewise for columns and \(L_v\) [2507.17528].

This combination of nuclear norm regularization and matrix Laplacian regularization is the defining structural feature of the graph-informed variant. The paper’s abstract states that existing matrix CB methods fail to explore such graph information, and thereby making them difficult to generate effective decision-making policies; the proposed framework is intended to fill this void by integrating low-rank structure and graph information in a unified manner [2507.17528].

## 3. Algorithmic frameworks

The graph-informed method is organized as a two-phase algorithm called Graph-UCB. In Phase I, a small batch of \(n\) exploratory samples \(\{(X_t,y_t)\}_{t=1}^n\) is collected, the convex regularized estimator is solved to obtain \(\widehat\Theta\), and the top-\(r\) SVD of \(\widehat\Theta\) is used to form low-dimensional subspaces. The problem is then projected into that subspace to reduce dimension [2507.17528].

In Phase II, the procedure applies an online generalized linear UCB rule. For each candidate action \(X\), the score is
\[
\mathrm{UCB}(X)=\mu\bigl(\langle X,\widehat\Theta\rangle\bigr)+\alpha \sqrt{\operatorname{Var}(X)},
\]
where \(\alpha>0\) is a tuning constant and \(\operatorname{Var}(X)\approx X^\top V^{-1}X\) is computed under a regularized design matrix. The stated design matrix includes a diagonal penalizer, the cumulative action outer products, and an action-graph regularizer:
\[
V
=
\Lambda
+\sum_{\tau=1}^{t-1}\operatorname{vect}(X_\tau)\operatorname{vect}(X_\tau)^\top
+a_\mu \alpha\,\widetilde X^\top L\,\widetilde X.
\]
Here \(\widetilde X\) stacks all \(\operatorname{vect}(X)\) of the \(n\) possible arms and \(L\) is the Laplacian over the action graph; in practice one encloses both user and item Laplacians via Kronecker sums. At time \(t\), the algorithm chooses the arm maximizing \(\mathrm{UCB}(X)\), observes reward, and updates \(V\) and the GLM-MLE \(\widehat\theta_t\) [2507.17528].

A distinct algorithmic line studies fully online estimation and inference rather than a two-phase UCB procedure. In that setting, the low-rank parameter is factorized as \(M=UV^\top\) with \(U\in\mathbb{R}^{p\times r}\) and \(V\in\mathbb{R}^{q\times r}\), and the estimator is updated by stochastic gradient descent with inverse-probability weighting. After choosing \(a_t\) and observing \(y_t\), the stochastic gradient is
\[
g_t
=
\left(\frac{\mathbf{1}\{a_t=i\}}{\pi_t}\right)
\begin{bmatrix}
(\langle U_{t-1}V_{t-1}^\top,X_t\rangle-y_t)\,X_tV_{t-1} \\
(\langle U_{t-1}V_{t-1}^\top,X_t\rangle-y_t)\,X_t^\top U_{t-1}
\end{bmatrix},
\]
followed by a re-normalization step via an \(r\times r\) SVD so that \(\|U_t\|\approx \|V_t\|\), and
\[
(U_t,V_t)\leftarrow (U_{t-1},V_{t-1})-\eta_t g_t,\qquad M_t=U_tV_t^\top,
\]
with decaying step size \(\eta_t=c\cdot t^{-\alpha}\), \(\alpha\in(0.5,1)\) [2212.11385].

Arm selection in that framework uses \(\epsilon\)-greedy,
\[
\pi_t
=
(1-\epsilon)\mathbf{1}\{\langle M_{1,t-1}-M_{0,t-1},X_t\rangle>0\}
+\epsilon/2,
\qquad a_t\sim\mathrm{Bern}(\pi_t).
\]
For inference, each arm maintains two sequences: \(M_{i,t}\), which is low-rank and biased for action selection, and \(\widehat M_{i,t}\), which is unbiased and not low-rank for inference. The online debiasing step is
\[
\widetilde M_{1,t}
=
M_{1,t-1}
+
\left(\frac{\mathbf{1}\{a_t=1\}}{\pi_t}\right)
(y_t-\langle M_{1,t-1},X_t\rangle)X_t,
\]
\[
\widehat M_{1,t}
=
\frac{t-1}{t}\widehat M_{1,t-1}
+
\frac1t \widetilde M_{1,t},
\]
with the analogous update for arm \(0\). After \(n\) rounds, low-rankness is reintroduced by the rank-\(r\) projection
\[
\widehat M_{1,n}^{(\mathrm{proj})}
=
\arg\min_{\operatorname{rank}\le r}\|\widehat M_{1,n}-M\|_F,
\]
and for any test matrix \(T\in\mathbb{R}^{p\times q}\),
\[
\widehat m_T^{(1)}=\langle \widehat M_{1,n}^{(\mathrm{proj})},T\rangle,\qquad
m_T^{(1)}=\langle M_1,T\rangle.
\]
The paper characterizes this as an online doubly-debiased estimator that simultaneously handles low-rank bias and adaptive sampling bias [2212.11385].

## 4. Regret analysis and statistical inference

Under mild regularity conditions—sub-Gaussian noise, bounded \(\|\Theta^*\|_F\le 1\), and a link function with bounded derivative \(c_\mu\le \mu^\prime\le k_\mu\)—Graph-UCB achieves, with high probability,
\[
R_T=\widetilde O\!\bigl(\zeta \sqrt{d_1d_2rT}\bigr),
\]
where \(r=\operatorname{rank}(\Theta^*)\), \(d_1,d_2\) are feature dimensions, and \(\zeta\in(0,1]\) is a factor that shrinks as graph information becomes richer; the more edges in \(G_u,G_v\), the smaller \(\zeta\). The notation \(\widetilde O(\cdot)\) hides logarithmic factors [2507.17528].

The same source gives explicit comparisons to two baseline structural assumptions. Standard low-rank CB without graph achieves \(\widetilde O(\sqrt{d_1d_2rT})\), so Graph-UCB gains the factor \(\zeta<1\). Graph-only CB without low rank scales like \(\widetilde O(d\sqrt{T})\) for \(d=\max\{d_1,d_2\}\), which is worse if \(r\ll d\) [2507.17528]. This comparison formalizes the claim that the method benefits from exploiting both structural sources simultaneously.

The inference-oriented formulation emphasizes a different theoretical target. Its low-rank SGD estimator satisfies a non-asymptotic convergence statement: under Assumptions 1–2, a good initialization, and \(\mathrm{SNR}\ge C\), choosing \(\eta_t=c\max\{t,t^*\}^{-\alpha}\), \(\alpha\in(0.5,1)\), yields for any \(1\le t\le n\), with probability at least \(1-4n/d^\gamma\),
\[
\|M_{i,t}-M_i\|_F
\le
C\cdot \gamma\cdot \sigma_i \cdot \sqrt{d\,r\,\log^2 d\,/\,t^\alpha}.
\]
The paper also states that the low-rank SGD part achieves \(O(n^{\alpha/2})\) estimation error, \(\alpha>0.5\), sufficient to drive \(O(\sqrt n)\) regret in the \(\epsilon\)-greedy policy, described as standard linear bandit rates up to log-factors [2212.11385].

Its principal inferential result is asymptotic normality for the doubly-debiased estimator. Let \(m_T^{(i)}=\langle M_i,T\rangle\) and \(\widehat m_T^{(i)}\) be the projected online estimator. Under Assumptions 1–4, as \(n,d\to\infty\),
\[
\sqrt n\,(\widehat m_T^{(i)}-m_T^{(i)})\to_D N(0,\sigma_i^2 S_i^2),
\]
with
\[
S_i^2=
\int
\frac{
\bigl\langle
U_{i,\perp}U_{i,\perp}^\top X V_iV_i^\top
+
U_iU_i^\top X V_{i,\perp}V_{i,\perp}^\top,
T
\bigr\rangle^2
}{
(1-\epsilon)\mathbf{1}\{\langle M_i-M_{1-i},X\rangle>0\}+\epsilon/2
}\,dP_X.
\]
For the contrast between arms,
\[
\sqrt n\Bigl[(\widehat m_T^{(1)}-\widehat m_T^{(0)})-(m_T^{(1)}-m_T^{(0)})\Bigr]
\to
N(0,\sigma_1^2S_1^2+\sigma_0^2S_0^2).
\]
The framework also includes consistent online estimators \(\hat\sigma_i^2\) and \(\hat S_i^2\), and
\[
\frac{\widehat m_T^{(i)}-m_T^{(i)}}{\hat\sigma_i \hat S_i/\sqrt n}\to_D N(0,1),
\]
so that a \(1-\alpha\) confidence interval is
\[
\bigl[\widehat m_T^{(i)}\pm z_{\alpha/2}\hat\sigma_i \hat S_i/\sqrt n\bigr].
\]
These results apply to both parameter inference and optimal policy value inference [2212.11385].

## 5. Empirical evidence and application domains

The graph-informed study reports both synthetic and real-world experiments. In synthetic data, varying the Erdős–Rényi probability \(p\) or the Barabási–Albert parameter \(m\) shows that regret falls as the graph becomes denser, confirming the \(\zeta\)-dependence. The reported comparisons state that Graph-UCB outperforms low-rank-only CB, graph-only CB, and classical GLM-UCB with vectorization and no structure [2507.17528].

On real data, the reported tasks are CCLE cancer cell-line drug sensitivity with linear rewards, MovieLens with binary click/no-click, and KDD Cup CTR data with Poisson counts. In all cases, Graph-UCB achieves the lowest cumulative regret and the fastest hit rate growth, which is presented as evidence that it learns the optimal arms more quickly by exploiting both low rank and graph smoothness [2507.17528].

The inference-focused study frames its applications more broadly as online decision-making problems with matrix context in fields ranging from healthcare to autonomous systems. Its stated contribution is not a new regret benchmark on those domains, but uncertainty quantification under adaptive sampling. It provides valid entry-level confidence intervals for \(\langle \Theta^*,T\rangle\) in the online setting, corrects both low-rank bias and adaptive sampling bias, and enables hypothesis tests such as
\[
H_0:\Theta^*(j,k)=0
\quad\text{or}\quad
H_0:\Theta_1^*(j,k)-\Theta_0^*(j,k)=0
\]
[2212.11385].

A plausible implication is that empirical evaluation in matrix CB now extends along two axes rather than one: reward maximization and uncertainty quantification. The first paper emphasizes cumulative regret and hit-rate behavior; the second emphasizes valid confidence intervals and asymptotic normality under adaptive data collection.

## 6. Relation to adjacent approaches and recurrent misconceptions

One recurrent misconception is that low-rank structure alone exhausts the exploitable structure in matrix contextual bandits. The graph-informed formulation explicitly rejects this view by positing user–user and item–item graphs whose connectivity captures similarity relations not represented by low-rankness alone. The stated comparison shows that standard low-rank CB without graph achieves \(\widetilde O(\sqrt{d_1d_2rT})\), whereas the graph-informed method gains a multiplicative factor \(\zeta<1\) in the regret bound [2507.17528].

A second misconception is the converse: that graph regularization alone is sufficient. The same comparison states that graph-only CB without low rank scales like \(\widetilde O(d\sqrt T)\) for \(d=\max\{d_1,d_2\}\), which is worse if \(r\ll d\). This supports the interpretation that low-rankness and graph smoothness are complementary rather than interchangeable structural assumptions [2507.17528].

A third misconception is that matrix contextual bandit research is concerned only with reward maximization and not with statistical inference. The inference-oriented work identifies this gap directly: existing online decision algorithms mainly focus on reward maximization, while less attention has been devoted to statistical inference. It argues that standard low-rank estimators are biased and cannot be obtained in a sequential manner, while existing inference approaches in sequential decision-making fail to account for low-rankness and are also biased. Its proposed online debiasing procedure is intended to simultaneously handle both sources of bias [2212.11385].

These clarifications indicate two distinct but compatible trajectories within the topic. One trajectory develops more expressive decision policies by enriching the reward model with graph information; the other develops inferential machinery for adaptively collected matrix-context data. This suggests that matrix contextual bandit research is increasingly defined by joint treatment of structure, adaptivity, and uncertainty rather than by arm selection alone.

Source: https://www.emergentmind.com/topics/matrix-contextual-bandit-cb