Practical and Optimal Algorithm for Linear Contextual Bandits with Rare Parameter Updates

Abstract: We study linear contextual bandits under rare parameter updates: the learner may incorporate reward feedback into its parameter estimate only at a small number of update times, while still observing contexts online and selecting actions sequentially. This viewpoint clarifies a practical distinction that is often blurred in the literature: many "strictly batched" methods additionally restrict within-interval context adaptivity, meaning that the action rule inside an interval cannot depend on the sequence of realized contexts/actions in that interval (beyond the current round's context). For linear contextual bandits, we propose two practical algorithms with only $O(\log\log T)$ parameter updates. Our first algorithm BLCE-G attains minimax-optimal regret (up to polylogarithmic factors in $T$) simultaneously in both the small-$K$ and large-$K$ regimes under a static schedule. Our second algorithm BLCE removes the near G-optimal design step -- a dominant computational bottleneck in prior strictly batched static-grid methods -- yet preserves minimax-optimal regret and achieves the lowest known runtime complexity among optimal algorithms. We further extend these rare-update and computational principles to generalized linear contextual bandits. Overall, our results yield statistically optimal algorithms under $O(\log\log T)$ parameter updates that are also computationally efficient in practice.

• The paper presents BLCE-G and BLCE, new algorithms that achieve minimax-optimal regret with O(loglog T) parameter updates in linear contextual bandit settings.
• It employs adaptive context exploitation and uncertainty maximization to reduce computational costs compared to full model retraining.
• Empirical experiments and theoretical analysis demonstrate that these methods outperform traditional batched approaches in both speed and regret performance.

Practical and Optimal Algorithm for Linear Contextual Bandits with Rare Parameter Updates

Problem Motivation and Setting

Linear contextual bandits serve as a canonical model for sequential decision making in the presence of context-dependent uncertainty, with applications in recommender systems, healthcare, and online advertising. Traditionally, most algorithms update model parameters at every round, but updating parameters in real-world deployments can be costly due to retraining, pipeline, privacy, and logging concerns. The paper addresses the rare parameter update regime: the learner observes contexts and chooses actions online, but incorporates reward feedback into parameter estimates only at a small number of update times. This regime removes the practicality bottleneck of frequent reward-driven retraining, which prior approaches address via strictly batched protocols, often at significant computational expense.

A major conceptual contribution is the separation of three settings:

1. Fully Sequential: Parameters updated and actions selected adaptively at every round.
2. Strictly Batched: Parameter updates and action rules fixed at batch boundaries; no within-interval adaptation.
3. Rare Parameter Updates: Parameters updated $\mathcal{O}(\log\log T)$ times, but reward-free statistics (Gram matrices, uncertainty estimates) updated and actions chosen sequentially within intervals.

Prior algorithms matched the statistical optimality of fully sequential approaches but often incurred excessive computational overhead due to preprocessing requirements such as G-optimal designs or context discretizations. The authors systematically address this bottleneck with new algorithms that maintain computational practicality while retaining optimal regret and batching properties.

Algorithms: BLCE-G and BLCE

Two primary algorithms are introduced:

• BLCE-G (Batched-Feedback Linear Contextual Bandit with Elimination and G-optimal design) leverages a blend of near G-optimal design and adaptive arm elimination. In each interval, an initial fraction of rounds use G-optimal designs for exploration; remaining phases balance uncertainty-driven exploration and exploitation based on previously computed estimators. Within-interval context adaptation substantially reduces the need for precomputed exploration plans, and updates to Gram matrices exploit lightweight Sherman-Morrison formulae for tractability.
• BLCE (Batched-Feedback Linear Contextual Bandit with Elimination) eliminates the expensive G-optimal design step entirely. All exploration is handled via uncertainty maximization and adaptive elimination. This approach significantly lowers total runtime complexity while, somewhat unexpectedly, retaining minimax-optimal regret in both the large- and small-arm regimes.

These algorithms function under a static grid schedule, i.e., update times are determined a priori, and maintain $\mathcal{O}(\log\log T)$ update complexity. Each algorithm incorporates reward-free within-interval adaptation, distinguishing them from strictly batched competitors which suffer from exponential or polynomial computational cost in dimensionality.

Theoretical Guarantees

The central theorems establish sharp upper bounds on cumulative regret:

• BLCE-G achieves, with $\mathcal{O}(\log\log T)$ parameter updates,

$$\mathcal{R}(T) = \mathcal{O}\big[\sqrt{d T}\left(\sqrt{\log(KT)} \wedge \sqrt{d+\log T}\right)\sqrt{\log d \log\log T}\big]$$

which matches the minimax lower bound $\Omega(\sqrt{dT\log K} \wedge d\sqrt{T})$ up to logarithmic factors in both the small-$K$ and large-$K$ regimes.

• BLCE preserves the same statistical rate, up to a $\sqrt{\log T}$ factor in the polylog term, yet attains runtime $\mathcal{O}(Kd^2 T \log\log T)$, substantially better than G-optimal and discretization-based competitors.

The algorithms generalize to generalized linear contextual bandits (GLCB) via BGLE, extending regret guarantees to link functions beyond the linear setup and—crucially—removing explicit dependence on the curvature parameter $\kappa$, which can otherwise be unbounded.

Empirical Evaluation

Extensive experiments demonstrate that both BLCE and BLCE-G consistently outpace prior baselines in realized regret and stability, across a wide spectrum of arm and dimension configurations.

Figure 1: Regret, zoomed-in regret, and update (batch) complexity over time for different values of $\mathcal{O}(\log\log T)$0 and $\mathcal{O}(\log\log T)$1.

Notably, BLCE achieves regret comparable to BLCE-G even without the computational burden of G-optimal design. BLCE-G and BLCE outperform both rare-switching and strictly batched approaches regardless of context distribution or batch structure. The computational advantage is especially stark; BLCE executes substantially faster than alternatives, while BLCE-G provides the strongest theoretical regret.

Additional experiments establish robustness to context distribution (normal, Student-t, beta, exponential, Laplace), demonstrate regret improvements with more greedy within-batch selection, and validate similar results for GLM reward structures.

Extensions and Design Principles

The extension to generalized linear contextual bandits is facilitated by the BGLE algorithm, which adapts the elimination schedule and exploration allocation while sidestepping heavy computation. BGLE renders the regret bound independent of $\mathcal{O}(\log\log T)$2, previously the limiting factor in minimax analysis for GLCB. The theoretical proofs provide explicit event-based argumentation, enabling the analysis to relax the traditional i.i.d. context assumption in favor of weaker batchwise conditional independence properties.

Figure 2: Regret and interval (batch) complexity over time for different values of $\mathcal{O}(\log\log T)$3 and $\mathcal{O}(\log\log T)$4 for generalized linear bandit scenarios.

The authors also empirically and theoretically investigate how the within-interval allocation rate $\mathcal{O}(\log\log T)$5 influences regret and adaptation, suggesting that smaller $\mathcal{O}(\log\log T)$6—i.e., more greedy within-interval selection—leads to consistently improved regret under stochastic contexts.

Implications and Future Directions

This work demonstrates, both theoretically and practically, that limited reward adaptivity does not necessitate computationally expensive pre-planned exploration; optimal regret is achievable with rare parameter updates and lightweight context adaptation. This finding more closely aligns practical deployment constraints with statistical optimality, opening the door to batch-constrained contextual bandit algorithms in resource-limited domains.

Theoretically, the framework enables regret-optimal algorithm design for rare-update settings, removing curvature dependence in GLM regret bounds and circumventing worst-case design computations. Practically, the results suggest that advanced contextual bandit algorithms can be safely deployed in settings where full retraining is infeasible or costly. Future avenues may explore tighter understanding of greedy selection's empirical benefits, extensions to nonlinear (e.g., neural) representations, and rare-update protocols for more structured bandit and RL environments.

Conclusion

The paper supplies the first minimax-optimal linear contextual bandit algorithms—BLCE-G and BLCE—with $\mathcal{O}(\log\log T)$7 reward-dependent updates and practical runtime, separates rare update and strict batch regimes, and extends these results to generalized linear settings without explicit dependence on worst-case curvature parameters. These insights have immediate theoretical, computational, and applied consequences for modern sequential decision making and adaptive data-driven systems.