---
title: 'Net-SGD-UCB: Decentralized Contextual Bandit'
url: https://www.emergentmind.com/topics/net-sgd-ucb
type: topic
---

# Net-SGD-UCB: Decentralized Contextual Bandit

Searching arXiv for Net-SGD-UCB and closely related neural/UCB bandit references.
Net-SGD-UCB is a decentralized, network-aware contextual linear bandit algorithm for multi-agent settings in which reward structure contains both a shared component and node-specific heterogeneity. It is introduced together with NetLinUCB in "Decentralized Contextual Bandits with Network Adaptivity" [2508.13411]. The method is designed for synchronous learning over a network of \(N\) nodes that face the same arm set but observe locally varying contexts and rewards. Its defining combination is a shared/local parameter decomposition, adaptive cross-node information sharing through a dynamically updated weight matrix, online momentum SGD for parameter estimation, a diagonal gradient-based uncertainty matrix, and a UCB action rule. In the formulation of the paper, this construction aims to preserve the statistical benefit of sharing common structure while avoiding the matrix inversions and communication burden of full centralization, especially in high-dimensional, streaming, and high-variance regimes [2508.13411].

## 1. Problem formulation

The algorithm is posed in a decentralized contextual linear bandit with \(N\) nodes operating over \(T\) rounds. At each round \(t\), node \(i \in [N]\) observes a context vector \(\mathbf{x}_{i,t}\), chooses an arm \(a_{i,t} \in \mathcal{A}=\{a^{(1)},\dots,a^{(K)}\}\), receives the reward of the chosen arm, and updates its local state [2508.13411].

A central modeling device is the decomposition of context into common and node-specific coordinates:
\[
\mathbf{x}_{i,t} = [\mathbf{x}_{i,c,t},\ \mathbf{x}_{i,s,t}],
\qquad
\mathbf{x}_{i,c,t} \in \mathbb{R}^{d_c},
\qquad
\mathbf{x}_{i,s,t} \in \mathbb{R}^{d_{i,s}},
\qquad
d_i=d_c+d_{i,s}.
\]
For each arm \(a^{(k)}\), node \(i\) has parameter
\[
\theta_i^k = [\theta_c^k,\ \theta_{i,s}^k],
\]
where \(\theta_c^k \in \mathbb{R}^{d_c}\) is shared across all nodes and \(\theta_{i,s}^k \in \mathbb{R}^{d_{i,s}}\) is local [2508.13411].

The reward model is linear:
\[
r_{i,t}^k = \mathbf{x}_{i,t}^\top \theta_i^k + \epsilon_{i,t}^k,
\]
with \(\epsilon_{i,t}^k \sim \mathcal{N}(0,\sigma_k^2)\). The paper also states that, in the SGD-UCB analysis, the noise is treated as sub-Gaussian with variance \(\sigma^2\) [2508.13411]. The expected reward therefore satisfies
\[
\mathbb{E}[r_{i,t}^k \mid \mathbf{x}_{i,t}]
=
[\mathbf{x}_{i,c,t},\mathbf{x}_{i,s,t}]^\top [\theta_c^k,\theta_{i,s}^k].
\]

The network cumulative regret is defined by
\[
R(T) = \mathbb{E}\!\left[\sum_{t=1}^T \sum_{i=1}^N r_{i,t}^{a_{i,t}^*}\right]
-
\mathbb{E}\!\left[\sum_{t=1}^T \sum_{i=1}^N r_{i,t}^{a_{i,t}}\right],
\]
where
\[
a_{i,t}^* = \arg\max_{k\in\mathcal{A}} \mathbf{x}_{i,t}^\top \theta_i^k.
\]
The boundedness assumptions are
\[
\|\mathbf{x}_{i,t}\|_2 \le 1,
\qquad
\|\theta_i^k\|_2 \le 1,
\]
for all \(i,k,t\) [2508.13411].

This formulation sits between two extremes identified in the paper: disjoint local learners that relearn shared structure independently, and fully centralized shared learners that pool all information but pay higher computational and communication costs. Net-SGD-UCB is the first-order alternative within that design space [2508.13411].

## 2. Shared and local structure over an adaptive network

Net-SGD-UCB assumes an initially fully connected communication network, justified in the paper by \(d_c \ge 1\), so that every pair of nodes shares at least some common feature structure [2508.13411]. Effective communication, however, is not fixed; it is governed by an adaptive arm-specific weight matrix
\[
\Omega^k = (\omega_{ji}^k)_{j,i=1}^N,
\]
where \(\omega_{ji}^k\) measures the directional influence from node \(j\) to node \(i\) for arm \(k\) [2508.13411].

The matrix is initialized as
\[
\Omega^k = I.
\]
Its update uses two signals. The first is arm-selection similarity,
\[
\frac{n_i^k n_j^k}{\left(\sum_{q\in\mathcal{G}} n_q^k\right)^2},
\]
where \(n_i^k\) is the number of times node \(i\) selected arm \(k\). The second is common-context cosine similarity,
\[
\frac{\mathbf{x}_{i,c}^\top \mathbf{x}_{j,c}}
{\|\mathbf{x}_{i,c}\| \cdot \|\mathbf{x}_{j,c}\|}.
\]
The paper states that these are combined and normalized to form \(\Omega_{\text{new}}^k\), after which exponential smoothing is applied:
\[
\Omega_t^k = \rho \,\Omega_{t-1}^k + (1-\rho)\,\Omega_{\text{new}}^k,
\qquad
\rho \in [0,1).
\]
The exact closed-form combination rule before normalization is not specified beyond the pseudocode-level description [2508.13411].

The consequence of this structure is that only the common component is communicated and aggregated. After local updates, node \(i\) forms its shared estimate for arm \(k\) as
\[
\hat{\theta}_{i,c,t}^k = \sum_{j\in\mathcal{G}} \omega_{ji}^k \hat{\theta}_{j,c,t}^k,
\]
while the node-specific component remains local:
\[
\hat{\theta}_{i,s,t}^k = \hat{\theta}_{i,s,t}^k.
\]
The paper explicitly states that Net-SGD-UCB communicates only shared parameter estimates, not raw data and not full local statistics [2508.13411].

This decomposition operationalizes a hybrid statistical assumption: all nodes benefit from a common arm-dependent signal, but each node retains personalized coordinates. A plausible implication is that the method is intended for environments in which full pooling would oversmooth node-specific effects, while purely local learning would be redundant on the common subspace.

## 3. SGD-based estimation and uncertainty construction

The defining algorithmic departure from matrix-based UCB methods is that Net-SGD-UCB replaces ridge-regression estimation with online stochastic gradient descent with momentum [2508.13411]. For node \(i\), arm \(k\), and round \(t\), the paper uses the squared-error loss
\[
\mathcal{L}_{i,t}^k = \frac{1}{2}(r_{i,t}^k - \hat{r}_{i,t}^k)^2,
\]
with linear prediction
\[
\hat{r}_{i,t}^k = \mathbf{x}_{i,t}^\top \hat{\theta}_{i,t-1}^k.
\]
The gradient is
\[
\nabla_\theta \mathcal{L}_{i,t}^k
=
-\bigl(r_{i,t}^k - \mathbf{x}_{i,t}^\top \hat{\theta}_{i,t-1}^k\bigr)\mathbf{x}_{i,t}.
\]

The momentum update is
\[
\mathbf{v}_{i,t}^k = \mu \mathbf{v}_{i,t-1}^k + (1-\mu)\nabla_\theta \mathcal{L}_{i,t}^k,
\]
followed by the parameter update
\[
\hat{\theta}_{i,t}^k = \hat{\theta}_{i,t-1}^k - \eta^{\text{sgd}} \mathbf{v}_{i,t}^k,
\]
where \(\mu\) is the momentum parameter and \(\eta^{\text{sgd}}\) is the learning rate [2508.13411]. The update is online, one observation at a time; the paper does not mention minibatching.

Uncertainty is tracked through a gradient-based matrix \(G_t\). The paper first defines
\[
G_t = I + \sum_{\tau\in\Psi_t} \nabla_{\theta}\mathcal{L}_{\tau}\bigl(\nabla_{\theta}\mathcal{L}_{\tau}\bigr)^\top,
\]
but then states that only diagonal entries are kept to reduce complexity [2508.13411]. The online EMA update for coordinate \(h\) is
\[
(G_t^k)_{hh}
=
\gamma (G_{t-1}^k)_{hh}
+
(1-\gamma)\left(\frac{\partial \mathcal{L}_t^k}{\partial \theta_h}\right)^2,
\]
or, in algorithm form,
\[
G_{i,t}^k
\gets
\gamma G_{i,t-1}^k
+
(1-\gamma)\,\mathrm{diag}\bigl((\nabla_\theta\mathcal{L}_{i,t}^k)^{\odot 2}\bigr),
\]
with \(\gamma \in [0,1)\) [2508.13411].

The diagonal accumulator is partitioned into shared and local blocks. The shared block \(G_{j,c,t}^k\) participates in cross-node uncertainty aggregation, whereas the local block \(G_{i,s,t}^k\) remains node-specific. This is the uncertainty analogue of the shared/local parameter split.

## 4. UCB index and decision rule

For node \(i\), arm \(k\), and round \(t\), the Net-SGD-UCB index is
\[
\text{SGD-}\mathcal{U}_{i,t}^k
=
\mathbf{x}_{i,t}^\top [\hat{\theta}_{i,c}^{k}, \hat{\theta}_{i,s}^{k}]
+
\alpha^{\text{sgd}}
\sqrt{
\sum_{j=1}^{N}(\omega_{ji}^{k})^2\mathbf{x}_{i,c,t}^\top(G_{j,c,t}^{k})^{-1}\mathbf{x}_{i,c,t}
+
\mathbf{x}_{i,s,t}^\top(G_{i,s,t}^{k})^{-1}\mathbf{x}_{i,s,t}
}.
\]
The selected arm is
\[
a_{i,t} = \arg\max_{k\in\mathcal{A}} \text{SGD-}\mathcal{U}_{i,t}^k
\]
[2508.13411].

The first term is the predicted reward using the aggregated common estimate and the local estimate. The second is the optimism bonus. The common-component contribution uses squared weights \((\omega_{ji}^k)^2\), reflecting the variance-style aggregation stated in the paper, while the local-component contribution depends only on node \(i\)'s own uncertainty state [2508.13411].

This index mirrors the structure of NetLinUCB but substitutes gradient-based diagonal uncertainty for ridge covariance. NetLinUCB uses
\[
\text{RIDGE-}\mathcal{U}_{i,t}^k
=
\mathbf{x}_{i,t}^\top [\hat{\theta}_{i,c}^{k}, \hat{\theta}_{i,s}^{k}]
+
\alpha^{\text{ridge}}
\sqrt{
\sum_{j=1}^{N}(\omega_{ji}^{k})^2\mathbf{x}_{i,c,t}^\top(W_{j,c}^{k})^{-1}\mathbf{x}_{i,c,t}
+
\mathbf{x}_{i,s,t}^\top(W_{i,s}^{k})^{-1}\mathbf{x}_{i,s,t}
},
\]
whereas Net-SGD-UCB replaces \(W^{-1}\) by \(G^{-1}\) and closed-form ridge updates by first-order SGD [2508.13411].

A concise operational summary of one round is therefore: update \(\Omega^k\); aggregate the shared estimate; compute per-arm UCB scores; choose the maximizing arm; observe the reward; update gradient, momentum, parameter, and diagonal uncertainty accumulator. This yields a fully decentralized procedure in which nodes share only summaries of the common component [2508.13411].

## 5. Regret guarantees and scaling properties

The paper gives both single-node and network-level guarantees. At the level of uncertainty accumulation, Proposition 6 states
\[
\mathbf{x}_t^\top G^{-1} \mathbf{x}_t
\leq
\sum_{t=1}^T \frac{2d\log (\sigma^2 T)}{(1-\gamma)^2\sigma^2}
+
\frac{(1-\mu)\sigma^2\log T}{(1+\mu)(1-\gamma)},
\]
while also noting that the notation is imprecise because the left-hand side is per-time and the right-hand side is cumulative; the intended object in the proof and subsequent use is \(\sum_t \mathbf{x}_t^\top G_t^{-1}\mathbf{x}_t\) [2508.13411].

With probability at least \(1-\frac1T\), Proposition 7 gives the single-node regret bound
\[
R(T)
\leq
2\alpha^{\text{sgd}}
\left(
\frac{\sqrt{2dT \log (\sigma^2 T)}}{(1 - \gamma)\sigma}
+
\sigma \sqrt{\frac{(1 - \mu)T\log T}{(1 + \mu)(1 - \gamma)}}
\right),
\]
with \(\alpha^{\text{sgd}} \propto 1+\sigma^2\) [2508.13411]. The dependence is explicit in dimension \(d\), horizon \(T\), noise \(\sigma\), momentum \(\mu\), and EMA smoothing \(\gamma\).

Theorem 4 decomposes the network-wide regret into shared and node-specific terms. For the shared component,
\[
R_1(T) = O\left( \frac{\sqrt{ N d_c T \log (\sigma^2 T)}}{1 - \gamma} \right),
\]
and for the node-specific component,
\[
R_2(T)
=
O \left(
\sum_{i=1}^N
\frac{\sqrt{2d_{i,s}T \log (\sigma^2 T)}}{(1 - \gamma)\sigma}
+
\sigma N \sqrt{\frac{(1 - \mu)T\log T}{(1 + \mu)(1 - \gamma)}}
\right).
\]
The total regret is the sum of these two terms [2508.13411].

The scaling claim emphasized in the paper is that learning complexity associated with the shared structure is reduced from \(O(N)\) to sublinear \(O(\sqrt{N})\). Concretely, the paper contrasts a disjoint baseline with shared-component cost
\[
\tilde O(N d_c \sqrt{T})
\]
against Net-SGD-UCB's shared-component term
\[
\tilde O(\sqrt{N d_c T})
\]
[2508.13411]. This is the main statistical argument for adaptive information sharing without full centralization.

The computational and communication profile is likewise central to the method's identity:

| Quantity | NetLinUCB | Net-SGD-UCB |
|---|---:|---:|
| Computation | \(\mathcal{O}(NK(d_c+d_s)^3)\) | \(\mathcal{O}(NK(d_c+d_s))\) |
| Memory | \(\mathcal{O}(NK(d_c+d_s)^2)\) | \(\mathcal{O}(NK(d_c+d_s))\) |
| Communication | \(\mathcal{O}(N^2 K d_c)\) | \(\mathcal{O}(N^2 K d_c)\) |

These complexity statements explain why the paper presents Net-SGD-UCB as the scalable first-order option, especially when \(d_c+d_{i,s}\) is large or matrix inversion is undesirable [2508.13411].

## 6. Empirical behavior, interpretation, and relation to adjacent UCB methods

The empirical evaluation in [2508.13411] is conducted on synthetic networked pricing simulations. The main metrics are average per-round regret over time,
\[
\frac{R(t)}{t},
\]
and average per-round per-node regret,
\[
\frac{R(T)}{NT}.
\]
The baselines are Disjoint LinUCB, Shared LinUCB, NetLinUCB, and Net-SGD-UCB [2508.13411].

Several findings are reported. First, the per-round regret \(R(t)/t\) decreases toward zero, which the paper interprets as sublinear regret. Second, both network-aware methods, including Net-SGD-UCB, achieve per-node regret comparable to Shared LinUCB, supporting sublinear scaling in network size \(N\). Third, supplementary Figure 5(b) introduces a large contextual outlier, and Net-SGD-UCB remains robust due to adaptive variance tracking. Fourth, supplementary Figure 5(c) shows that when the number of actions increases, Net-SGD-UCB performs best. The paper repeatedly emphasizes that Net-SGD-UCB is stronger in noisier settings, whereas NetLinUCB is stronger when reward gaps are small and heterogeneity is fine-grained [2508.13411].

Supplementary Table 2 reports reduction in confidence radius relative to Disjoint LinUCB. For Net-SGD-UCB the reported reductions are \(21.72\%\) for \(3\times 4\), \(22.40\%\) for \(6\times 2\), and \(22.66\%\) for \(12\). For NetLinUCB the corresponding values are \(39.97\%\), \(84.75\%\), and \(94.82\%\) [2508.13411]. The paper interprets these numbers as showing that NetLinUCB benefits more from denser connectivity, while Net-SGD-UCB is more stable and consistent across connectivity levels.

Within the broader UCB literature, Net-SGD-UCB is structurally closer to first-order, linear, decentralized optimism than to overparameterized neural contextual bandits. "Neural Contextual Bandits with UCB-based Exploration" [1911.04462] uses a deep neural predictor and a confidence term built from dynamic gradient features \(\gb(\xb;\btheta_{t-1})^\top \Zb_{t-1}^{-1}\gb(\xb;\btheta_{t-1})/m\), whereas Net-SGD-UCB uses a contextual linear reward model with a diagonal uncertainty matrix \(G\) built from past squared gradients [1911.04462]. "Neural Bandit with Arm Group Graph" [2206.03644] extends neural-gradient UCB to graph-structured arm groups, but there the graph indexes correlations among arm groups, not communication among decentralized agents, and the reward model includes a GNN representation stage before the UCB score is formed [2206.03644]. In that sense, Net-SGD-UCB belongs to a different branch of the literature: networked contextual linear bandits with adaptive inter-agent sharing.

The paper also makes clear what Net-SGD-UCB does not eliminate. Communication still scales as
\[
\mathcal{O}(N^2 K d_c)
\]
per round; the exact rule that combines arm-count similarity and context similarity into \(\Omega_{\text{new}}^k\) is not fully specified beyond pseudocode; and several hyperparameters require tuning, namely \(\eta^{\text{sgd}}\), \(\mu\), \(\alpha^{\text{sgd}}\), \(\gamma\), and \(\rho\) [2508.13411]. The theoretical presentation is also described in the source material as informal in some places, especially around cumulative versus per-step notation. These caveats delimit the method's present scope while clarifying its role: a scalable decentralized UCB algorithm that preserves shared/local structure and replaces matrix-based confidence estimation with momentum-SGD updates and diagonal EMA uncertainty [2508.13411].

Source: https://www.emergentmind.com/topics/net-sgd-ucb