Matrix Contextual Bandits: Graph & Low-Rank Models

• Matrix contextual bandits are an online decision-making framework where matrix-valued contexts and an unknown low-rank parameter drive reward modeling.
• They integrate generalized linear models with nuclear norm and Laplacian regularization to exploit both low-rank structure and user/item similarity graphs.
• Algorithmic frameworks like Graph-UCB and online SGD debiasing balance exploration with valid inference, optimizing cumulative regret and uncertainty quantification.

Matrix contextual bandit (CB) denotes an online decision-making framework in which contextual information is represented as a matrix and the reward model is governed by an unknown low-rank parameter matrix. It is described as an extension of the well-known multi-armed bandit and is motivated by sequential decision-making scenarios in which low-rank structure is intrinsic, including recommendation-style settings with user and item features (Wang et al., 23 Jul 2025). In the formulation with graph information, matrix CB incorporates not only low-rank structure but also user–user and item–item similarity graphs through Laplacian regularization, while in the inference-oriented formulation it also supports valid online statistical inference under adaptive data collection by combining low-rank estimation with online debiasing (Han et al., 2022).

1. Problem formulation and reward models

In the recommendation-style formulation, at each round $t=1,2,\dots,T$, the learner observes a user feature vector $p_t\in\mathbb{R}^{d_1}$ and an item feature vector $q_t\in\mathbb{R}^{d_2}$, and combines them into an action matrix

$X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$

The learner then chooses one of $n$ possible user–item pairs and receives a noisy reward $y_t$. The expected reward is modeled by a generalized linear model,

$$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$

where $\Theta^*\in\mathbb{R}^{d_1\times d_2}$ is an unknown low-rank matrix with rank $r\ll \min\{d_1,d_2\}$, $\langle A,B\rangle=\operatorname{tr}(A^\top B)$ is the Frobenius inner product, and $p_t\in\mathbb{R}^{d_1}$0 is a known inverse-link function such as logistic, Poisson, or the identity (Wang et al., 23 Jul 2025).

The objective in this setting is cumulative regret minimization. With

$p_t\in\mathbb{R}^{d_1}$1

the cumulative regret is defined as

$p_t\in\mathbb{R}^{d_1}$2

This formulation makes explicit that action quality is determined through a matrix inner product against the latent low-rank parameter (Wang et al., 23 Jul 2025).

A related formulation studies online decision-making with matrix context in a binary-action setting. At each round $p_t\in\mathbb{R}^{d_1}$3, an agent observes a collection of $p_t\in\mathbb{R}^{d_1}$4 arms, with $p_t\in\mathbb{R}^{d_1}$5 and $p_t\in\mathbb{R}^{d_1}$6, where arm $p_t\in\mathbb{R}^{d_1}$7 is associated with a matrix context $p_t\in\mathbb{R}^{d_1}$8. Upon choosing arm $p_t\in\mathbb{R}^{d_1}$9, the observed reward satisfies

$q_t\in\mathbb{R}^{d_2}$0

with $q_t\in\mathbb{R}^{d_2}$1. In the binary-arm setting, one may think $q_t\in\mathbb{R}^{d_2}$2 (Han et al., 2022).

These two formulations share the same structural core: matrix-valued context, a low-rank latent parameter, and sequential arm selection. A plausible implication is that “matrix contextual bandit” is best viewed not as a single algorithm, but as a class of structured contextual bandit problems in which bilinear or matrix-structured covariates permit low-dimensional statistical regularization.

2. Low-rank structure and graph information

A central premise of matrix CB is that the unknown parameter matrix is low rank. In the generalized low-rank formulation, this is encoded by assuming $q_t\in\mathbb{R}^{d_2}$3 has rank $q_t\in\mathbb{R}^{d_2}$4, and estimation uses the nuclear norm

$q_t\in\mathbb{R}^{d_2}$5

which promotes $q_t\in\mathbb{R}^{d_2}$6 (Wang et al., 23 Jul 2025).

The graph-augmented model adds side information through a user–user graph $q_t\in\mathbb{R}^{d_2}$7 with Laplacian $q_t\in\mathbb{R}^{d_2}$8 and an item–item graph $q_t\in\mathbb{R}^{d_2}$9 with Laplacian $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$0. Roughly speaking, if two users $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$1 are connected in $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$2, their corresponding rows of $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$3 should vary smoothly; likewise for connected items in $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$4 (Wang et al., 23 Jul 2025).

To exploit both sources of structure, the offline estimator is defined by the convex program

$X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$5

The two Laplacian terms have explicit interpretations: $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$6 penalizes changes across the user graph, and

$X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$7

penalizes changes across the item graph. Each Laplacian term enforces smoothness: if users $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$8 and $X_t = p_t q_t^\top \in \mathbb{R}^{d_1\times d_2}.$9 are connected, then row $n$0 and row $n$1 of $n$2 are encouraged to be similar; likewise for columns and $n$3 (Wang et al., 23 Jul 2025).

This combination of nuclear norm regularization and matrix Laplacian regularization is the defining structural feature of the graph-informed variant. The paper’s abstract states that existing matrix CB methods fail to explore such graph information, and thereby making them difficult to generate effective decision-making policies; the proposed framework is intended to fill this void by integrating low-rank structure and graph information in a unified manner (Wang et al., 23 Jul 2025).

3. Algorithmic frameworks

The graph-informed method is organized as a two-phase algorithm called Graph-UCB. In Phase I, a small batch of $n$4 exploratory samples $n$5 is collected, the convex regularized estimator is solved to obtain $n$6, and the top-$n$7 SVD of $n$8 is used to form low-dimensional subspaces. The problem is then projected into that subspace to reduce dimension (Wang et al., 23 Jul 2025).

In Phase II, the procedure applies an online generalized linear UCB rule. For each candidate action $n$9, the score is

$y_t$0

where $y_t$1 is a tuning constant and $y_t$2 is computed under a regularized design matrix. The stated design matrix includes a diagonal penalizer, the cumulative action outer products, and an action-graph regularizer: $y_t$3 Here $y_t$4 stacks all $y_t$5 of the $y_t$6 possible arms and $y_t$7 is the Laplacian over the action graph; in practice one encloses both user and item Laplacians via Kronecker sums. At time $y_t$8, the algorithm chooses the arm maximizing $y_t$9, observes reward, and updates $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$0 and the GLM-MLE $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$1 (Wang et al., 23 Jul 2025).

A distinct algorithmic line studies fully online estimation and inference rather than a two-phase UCB procedure. In that setting, the low-rank parameter is factorized as $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$2 with $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$3 and $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$4, and the estimator is updated by stochastic gradient descent with inverse-probability weighting. After choosing $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$5 and observing $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$6, the stochastic gradient is

$$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$7

followed by a re-normalization step via an $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$8 SVD so that $$\mathbb{E}[y_t\mid X_t] = \mu\bigl(\langle X_t,\Theta^*\rangle\bigr),$$9, and

$\Theta^*\in\mathbb{R}^{d_1\times d_2}$0

with decaying step size $\Theta^*\in\mathbb{R}^{d_1\times d_2}$1, $\Theta^*\in\mathbb{R}^{d_1\times d_2}$2 (Han et al., 2022).

Arm selection in that framework uses $\Theta^*\in\mathbb{R}^{d_1\times d_2}$3-greedy,

$\Theta^*\in\mathbb{R}^{d_1\times d_2}$4

For inference, each arm maintains two sequences: $\Theta^*\in\mathbb{R}^{d_1\times d_2}$5, which is low-rank and biased for action selection, and $\Theta^*\in\mathbb{R}^{d_1\times d_2}$6, which is unbiased and not low-rank for inference. The online debiasing step is

$\Theta^*\in\mathbb{R}^{d_1\times d_2}$7

$\Theta^*\in\mathbb{R}^{d_1\times d_2}$8

with the analogous update for arm $\Theta^*\in\mathbb{R}^{d_1\times d_2}$9. After $r\ll \min\{d_1,d_2\}$0 rounds, low-rankness is reintroduced by the rank-$r\ll \min\{d_1,d_2\}$1 projection

$r\ll \min\{d_1,d_2\}$2

and for any test matrix $r\ll \min\{d_1,d_2\}$3,

$r\ll \min\{d_1,d_2\}$4

The paper characterizes this as an online doubly-debiased estimator that simultaneously handles low-rank bias and adaptive sampling bias (Han et al., 2022).

4. Regret analysis and statistical inference

Under mild regularity conditions—sub-Gaussian noise, bounded $r\ll \min\{d_1,d_2\}$5, and a link function with bounded derivative $r\ll \min\{d_1,d_2\}$6—Graph-UCB achieves, with high probability,

$r\ll \min\{d_1,d_2\}$7

where $r\ll \min\{d_1,d_2\}$8, $r\ll \min\{d_1,d_2\}$9 are feature dimensions, and $\langle A,B\rangle=\operatorname{tr}(A^\top B)$0 is a factor that shrinks as graph information becomes richer; the more edges in $\langle A,B\rangle=\operatorname{tr}(A^\top B)$1, the smaller $\langle A,B\rangle=\operatorname{tr}(A^\top B)$2. The notation $\langle A,B\rangle=\operatorname{tr}(A^\top B)$3 hides logarithmic factors (Wang et al., 23 Jul 2025).

The same source gives explicit comparisons to two baseline structural assumptions. Standard low-rank CB without graph achieves $\langle A,B\rangle=\operatorname{tr}(A^\top B)$4, so Graph-UCB gains the factor $\langle A,B\rangle=\operatorname{tr}(A^\top B)$5. Graph-only CB without low rank scales like $\langle A,B\rangle=\operatorname{tr}(A^\top B)$6 for $\langle A,B\rangle=\operatorname{tr}(A^\top B)$7, which is worse if $\langle A,B\rangle=\operatorname{tr}(A^\top B)$8 (Wang et al., 23 Jul 2025). This comparison formalizes the claim that the method benefits from exploiting both structural sources simultaneously.

The inference-oriented formulation emphasizes a different theoretical target. Its low-rank SGD estimator satisfies a non-asymptotic convergence statement: under Assumptions 1–2, a good initialization, and $\langle A,B\rangle=\operatorname{tr}(A^\top B)$9, choosing $p_t\in\mathbb{R}^{d_1}$00, $p_t\in\mathbb{R}^{d_1}$01, yields for any $p_t\in\mathbb{R}^{d_1}$02, with probability at least $p_t\in\mathbb{R}^{d_1}$03,

$p_t\in\mathbb{R}^{d_1}$04

The paper also states that the low-rank SGD part achieves $p_t\in\mathbb{R}^{d_1}$05 estimation error, $p_t\in\mathbb{R}^{d_1}$06, sufficient to drive $p_t\in\mathbb{R}^{d_1}$07 regret in the $p_t\in\mathbb{R}^{d_1}$08-greedy policy, described as standard linear bandit rates up to log-factors (Han et al., 2022).

Its principal inferential result is asymptotic normality for the doubly-debiased estimator. Let $p_t\in\mathbb{R}^{d_1}$09 and $p_t\in\mathbb{R}^{d_1}$10 be the projected online estimator. Under Assumptions 1–4, as $p_t\in\mathbb{R}^{d_1}$11,

$p_t\in\mathbb{R}^{d_1}$12

with

$p_t\in\mathbb{R}^{d_1}$13

For the contrast between arms,

$p_t\in\mathbb{R}^{d_1}$14

The framework also includes consistent online estimators $p_t\in\mathbb{R}^{d_1}$15 and $p_t\in\mathbb{R}^{d_1}$16, and

$p_t\in\mathbb{R}^{d_1}$17

so that a $p_t\in\mathbb{R}^{d_1}$18 confidence interval is

$p_t\in\mathbb{R}^{d_1}$19

These results apply to both parameter inference and optimal policy value inference (Han et al., 2022).

5. Empirical evidence and application domains

The graph-informed study reports both synthetic and real-world experiments. In synthetic data, varying the Erdős–Rényi probability $p_t\in\mathbb{R}^{d_1}$20 or the Barabási–Albert parameter $p_t\in\mathbb{R}^{d_1}$21 shows that regret falls as the graph becomes denser, confirming the $p_t\in\mathbb{R}^{d_1}$22-dependence. The reported comparisons state that Graph-UCB outperforms low-rank-only CB, graph-only CB, and classical GLM-UCB with vectorization and no structure (Wang et al., 23 Jul 2025).

On real data, the reported tasks are CCLE cancer cell-line drug sensitivity with linear rewards, MovieLens with binary click/no-click, and KDD Cup CTR data with Poisson counts. In all cases, Graph-UCB achieves the lowest cumulative regret and the fastest hit rate growth, which is presented as evidence that it learns the optimal arms more quickly by exploiting both low rank and graph smoothness (Wang et al., 23 Jul 2025).

The inference-focused study frames its applications more broadly as online decision-making problems with matrix context in fields ranging from healthcare to autonomous systems. Its stated contribution is not a new regret benchmark on those domains, but uncertainty quantification under adaptive sampling. It provides valid entry-level confidence intervals for $p_t\in\mathbb{R}^{d_1}$23 in the online setting, corrects both low-rank bias and adaptive sampling bias, and enables hypothesis tests such as

$p_t\in\mathbb{R}^{d_1}$24

(Han et al., 2022).

A plausible implication is that empirical evaluation in matrix CB now extends along two axes rather than one: reward maximization and uncertainty quantification. The first paper emphasizes cumulative regret and hit-rate behavior; the second emphasizes valid confidence intervals and asymptotic normality under adaptive data collection.

6. Relation to adjacent approaches and recurrent misconceptions

One recurrent misconception is that low-rank structure alone exhausts the exploitable structure in matrix contextual bandits. The graph-informed formulation explicitly rejects this view by positing user–user and item–item graphs whose connectivity captures similarity relations not represented by low-rankness alone. The stated comparison shows that standard low-rank CB without graph achieves $p_t\in\mathbb{R}^{d_1}$25, whereas the graph-informed method gains a multiplicative factor $p_t\in\mathbb{R}^{d_1}$26 in the regret bound (Wang et al., 23 Jul 2025).

A second misconception is the converse: that graph regularization alone is sufficient. The same comparison states that graph-only CB without low rank scales like $p_t\in\mathbb{R}^{d_1}$27 for $p_t\in\mathbb{R}^{d_1}$28, which is worse if $p_t\in\mathbb{R}^{d_1}$29. This supports the interpretation that low-rankness and graph smoothness are complementary rather than interchangeable structural assumptions (Wang et al., 23 Jul 2025).

A third misconception is that matrix contextual bandit research is concerned only with reward maximization and not with statistical inference. The inference-oriented work identifies this gap directly: existing online decision algorithms mainly focus on reward maximization, while less attention has been devoted to statistical inference. It argues that standard low-rank estimators are biased and cannot be obtained in a sequential manner, while existing inference approaches in sequential decision-making fail to account for low-rankness and are also biased. Its proposed online debiasing procedure is intended to simultaneously handle both sources of bias (Han et al., 2022).

These clarifications indicate two distinct but compatible trajectories within the topic. One trajectory develops more expressive decision policies by enriching the reward model with graph information; the other develops inferential machinery for adaptively collected matrix-context data. This suggests that matrix contextual bandit research is increasingly defined by joint treatment of structure, adaptivity, and uncertainty rather than by arm selection alone.