Nonstationary Generalized Linear Bandits with Discounted Online Mirror Descent

Abstract: We study nonstationary generalized linear bandits (GLBs), where the expected reward is modeled through a nonlinear link function with an unknown time-varying parameter. This framework encompasses a broad class of reward models, including linear, Bernoulli, and binomial rewards. Existing approaches are predominantly based on maximum-likelihood estimation (MLE), using sliding-window, restart, or discounting mechanisms to handle nonstationarity. Although these methods achieve statistically efficient regret guarantees, they generally require revisiting past observations at every round, which leads to computation and memory costs that grow with time; moreover, several of them rely on a non-convex projection step. In this paper, we propose DOMD-GLB, a new algorithm for nonstationary GLBs that utilizes discounted online mirror descent (DOMD) for parameter estimation, thereby incurring only $O(1)$ computation and memory costs per round. We prove dynamic regret bounds of order $\tilde{O} \big(c_μ^{-1/2} d^{3/4} P_T^{1/4} T^{3/4}\big)$ in drifting environments and $\tilde{O}\big(c_μ^{-1/3} d^{2/3} ΓT^{1/3} T^{2/3}\big) $in piecewise-stationary environments, where $d$ denotes the feature dimension, $T$ the time horizon, $P_T$ the path length, $Γ_T$ the number of change points, and $cμ$ a curvature parameter associated with the link function, while substantially improving computational efficiency over prior work. To the best of our knowledge, this is the first algorithm for nonstationary GLBs with per-round computation and memory costs independent of time.

• The paper introduces a Discounted Online Mirror Descent method for nonstationary generalized linear bandits, adapting to drifting parameters with constant per-round cost.
• It leverages a surrogate quadratic loss and recursive updates to achieve order-optimal regret bounds in both drifting and piecewise-stationary environments.
• Empirical evaluations show that DOMD-GLB outperforms traditional MLE-based approaches in dynamic settings, offering scalability for real-time applications.

Nonstationary Generalized Linear Bandits via Discounted Online Mirror Descent

Problem Formulation and Motivation

The paper "Nonstationary Generalized Linear Bandits with Discounted Online Mirror Descent" (2605.25590) addresses sequential decision-making in environments where reward distributions are governed by nonlinear link functions with unknown, time-varying parameters. The generalized linear bandit (GLB) model subsumes a broad class of reward distributions, including Bernoulli and binomial models, unified under a link function $\mu$, with the reward expectation parametrized as $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$ at each round $t$.

Nonstationarity is formalized via two canonical metrics: path length $$P_T := \sum_{t=1}^{T-1} \|\theta_{t+1}^\star - \theta_t^\star\|_2$$ for drifting environments and number of change points $\Gamma_T := \sum_{t=1}^{T-1} \mathbbm{1}\{\theta_{t+1}^\star \neq \theta_t^\star\}$ for piecewise-stationary settings. This covers evolving scenarios such as changing user preferences or dynamic market behavior, with immediate relevance in contexts like recommendation and pricing.

Prior nonstationary GLB algorithms predominantly employ maximum likelihood estimation (MLE) with windowing, discounting, or restarts, incurring per-round computation and memory tied to the time step $t$ or window/restart length. Several also require non-convex projections, precluding polynomial-time guarantees.

Algorithmic Contribution: DOMD-GLB

The paper introduces Discounted Online Mirror Descent (DOMD) for parameter estimation in nonstationary GLBs. DOMD leverages the following innovations:

• Discounted Loss Geometry: Previous mirror descent analyses for stationary GLBs accumulate curvature over all history. The DOMD formulation downweights historical curvature, parameterized by discount factor $\gamma \in (0,1)$, ensuring adaptation to drift while retaining sufficient curvature for local stability.
• Surrogate Loss and Quadratic Approximation: The update at round $t$ is based on a second-order Taylor expansion $\widetilde{\ell}_t(\theta)$ of the negative log-likelihood loss, augmented with a proximal regularization in the matrix $A_t=\gamma H_t + (1-\gamma)\lambda I_d$, where $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$0 is the discounted accumulated curvature.
• Recursive Representation: DOMD updates depend only on the current sample, parameter estimate, and maintained curvature, yielding $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$1 per-round computation and memory cost, independent of $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$2.

This approach circumvents expensive revisits of historical data and avoids non-convex projections, marking a fundamental departure from the computational bottlenecks of prior windowed/discounted MLE-based strategies.

Theoretical Analysis

Confidence Bounds

A critical technical merit is the derivation of high-probability ellipsoidal confidence sets around the DOMD iterates, with radii incorporating both standard statistical terms and a nonstationarity variation kernel $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$3, which quantifies discounted comparator drift. The concentration argument replaces standard mix-loss martingale analysis (which fails under discounting) with a fixed-time exponential bound and union over $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$4.

Regret Guarantees

The regret bounds are established for both drifting and piecewise-stationary environments:

• Drifting: DOMD-GLB attains dynamic regret $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$5, matching the best known order among discounted MLE-based methods, while strictly improving computational efficiency—previous algorithms have per-round cost $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$6 or worse.
• Piecewise-Stationary: DOMD-GLB achieves $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$7, again matching order-optimal discounted methods modulo an additional $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$8 factor, with only $\mathbb{E}[r_t|x_t] = \mu(x_t^\top \theta_t^\star)$9 resource usage per round.

The paper explicitly clips the discount parameter $t$0 to feasible values, resulting in three-case regret bounds that rigorously hold across all parameter regimes, unlike earlier literature which often omits explicit feasibility checks.

Empirical Evaluation

The proposed algorithm's performance is assessed on synthetic Bernoulli GLB instances with dynamically evolving parameters. Against baselines including stationary OMD [zhang2025generalized], windowed/restarted MLE-based approaches [zhao2020simple, russac2020algorithms, wang2023revisiting], and adaptive restart methods [wei2021non], DOMD-GLB consistently exhibits lower cumulative dynamic regret and constant per-round runtime.

Figure 1: Cumulative dynamic regret and runtime for varying $t$1 across drifting and piecewise-stationary settings.

The empirical gap is especially pronounced in environments with high drift or multiple change points, where computational bottlenecks of competing strategies are severe. Adaptive restart algorithms such as MASTER [wei2021non] achieve theoretically optimal $t$2-dependence but perform poorly in practical regimes due to conservatively scaled confidence bonuses and nontrivial memory/computation.

Practical and Theoretical Implications

The methodology demonstrates that statistical efficiency in nonstationary parametric bandit learning can be decoupled from historical data revisit cost by integrating discounted geometry within online convex optimization and employing appropriately tailored confidence sets. The pattern of regret bounds reveals that online mirror descent, previously analyzed for stationary models, can—with suitable discounting—be extended to yield competitive statistical guarantees in drifting and piecewise-stationary regimes.

The additional $t$3 dependency in the piecewise-stationary regret reflects inherent trade-offs in retaining full computational online efficiency: $t$4 is a measure of link function nonlinearity, with previous polynomial-time approaches avoiding it at the expense of non-convexity or memory scaling. Refining this dependency while maintaining $t$5 cost per round remains nontrivial and open for further investigation.

Conclusion

The paper delivers the first online, computationally efficient algorithm for nonstationary GLBs with proven regret bounds and constant per-round operational cost. The DOMD framework achieves order-optimal dynamic regret up to link curvature dependencies and is empirically superior to all previous polynomial-time methods in both drifting and piecewise-stationary scenarios.

Constant per-round resource usage opens prospects for scalable bandit learning in genuinely nonstationary, high-dimensional, and long-horizon applications, suitable for real-time systems with strict latency constraints. Theoretical extension to more adaptive and change-sensitive mirror-descent algorithms, refined analysis of link function nonlinearity, and integration with higher-order parametric models constitute promising future directions.