Online Learning with Gradient-Variation Interval Regret

Abstract: This paper investigates non-stationary online learning using the metric of interval regret, which requires an online algorithm to perform well over every time interval. We propose the first online learning algorithm that achieves an interval regret bound scaling with gradient variation, a fundamental measure of the cumulative change in online function gradients, which relates to various problem-dependent quantities and is closely connected to stochastic optimization and other problems. Our method employs a simple and efficient two-layer online ensemble structure that achieves strong theoretical guarantees. Specifically, it enjoys a regret bound that simultaneously adapts to various problem-dependent quantities while also preserving the minimax-optimal rate in the worst case. Moreover, recognizing the challenge of hyperparameter tuning, we introduce a Lipschitz- and smoothness-agnostic variant that automatically adapts to these potentially unknown constants. This is primarily enabled by a novel Lipschitz-adaptive meta algorithm, which may be of independent interest. Beyond interval regret, our method also yields broader implications: it provides versatile bounds for interval dynamic regret, a stronger measure that competes with changing comparators over any interval, and yields the first piecewise characterization for stochastic extended adversarial optimization. Theoretical findings are validated by experiments.

• The paper introduces GAIR, the first online algorithm achieving interval regret bounds that scale with cumulative gradient variation.
• It derives rigorous regret guarantees that adapt to both minimax and small-loss regimes, unifying classical and dynamic performance measures.
• The study also develops GAIR-L, a Lipschitz-adaptive variant, validated empirically on non-stationary tasks like MNIST and synthetic regression.

Online Learning with Gradient-Variation Interval Regret

Introduction and Motivation

This paper introduces a new framework for non-stationary online learning by proposing gradient-variation interval regret as the primary performance measure. Traditional OCO analyses have focused on static regret or, more robustly, interval/adaptive regret to handle non-stationary environments. However, existing interval regret guarantees are either minimax (i.e., scale as $\mathcal{O}(\sqrt{|I|\log T})$ for interval $I$) or, at best, adapt to problem-dependent quantities such as small-loss, but do not directly leverage the more fundamental notion of cumulative gradient variation. The gradient-variation metric reflects local changes in the loss landscape, offering more fine-grained adaptivity in online learning. The paper's main contributions are introducing the first online learning algorithm, GAIR, that achieves interval regret guarantees scaling with gradient variation, and a Lipschitz/smoothness-adaptive variant, GAIR-L, suitable for less parameterized practical deployment.

Formalization and Key Technical Contributions

The central quantity, for any interval $I=[r,s]$, is gradient variation:

$V_{I} = \sum_{t = r+1}^{s} \sup_{\x \in \mathcal{X}} \|\nabla f_t(\x) - \nabla f_{t-1}(\x)\|^2,$

which captures the cumulative local dynamism of the gradient field.

Main technical contributions:

• First gradient-variation interval regret bound: The analysis overcomes instability due to dynamic ensemble management of base learners, pivotal in interval regret frameworks, by employing negative Bregman divergences from loss smoothness rather than conventional stability-based cancellations. This leads to regret of order $\tilde{\mathcal{O}}(\sqrt{V_I})$ (ignoring log factors) under standard smoothness and boundedness.
• Versatility: The derived regret bound simultaneously captures classical small-loss and minimax guarantees, as it bounds interval regret by the minimum of three quantities: gradient variation, small-loss, and minimax rates.
• Parameter adaptivity with GAIR-L: Lipschitz and smoothness constants are often unknown or highly non-stationary. GAIR-L uses a novel, adaptive meta-algorithm (LEO Adapt-ML-Prod) to online-estimate these constants and select the appropriate learning rate, ensuring robust performance without hyperparameter knowledge.
• Implications: The structure extends to interval dynamic regret and the Stochastically Extended Adversarial (SEA) model, yielding strong results in flexible adversarial environments as well as those interpolating between stochastic and adversarial processes.

Algorithmic Scheme

The proposed GAIR algorithm is built via a two-layer online ensemble:

• Base layer: Base learners per interval, using optimistic OMD with self-confident stepsizes, predict over their assigned local time ranges. The optimism leverages previous gradients for variance reduction.
• Meta layer: Aggregates active base learner predictions applying a Lipschitz-adaptive Prod-style weight update with optimism for non-stationary losses.
• Scheduling: Either problem-dependent (data-driven markers) or independent (dyadic intervals), the scheduling policy manages ensemble complexity efficiently, often to $\mathcal{O}(\log F_T)$ in favorable regimes.

The overall algorithm executes a single gradient query per round and maintains only a logarithmic number of base learners in practice.

Theoretical Results

The main result (Theorem 1) and its extensions rigorously upper bound the interval regret by

$$\mathcal{O}\left(\min\left\{ \sqrt{ \min\{V_I, F_I\} \log F_I \log F_T },\ \sqrt{|I| \log T} \right\}\right)$$

where $F_I$ is the local cumulative small-loss over $I$.

Additional Regret Guarantees

• Interval dynamic regret: The same machinery and analysis extend to handle more general, even non-stationary/comparator sequences with path length $P_I$, leading to regret

$I$0

• SEA model: For stochastic/adversarially drifting gradient sequences, the regret becomes

$I$1

where $I$2 encodes stochasticity and $I$3 encodes adversarial drift.

These results are all achieved without additional oracle or second-order complexity.

Analysis Overview

The technical novelty lies in the management of instability from dynamic ensemble activation, which disrupts classic stability-based analyses. By exploiting the intrinsic negative Bregman divergence of smooth functions rather than stability from the algorithm itself, the analysis decouples the regret incurred by the meta-algorithm, yielding the desired dependence on actual variation of gradients, not just their empirical upper bounds.

(Figure 1)

Figure 2: Performance comparison between GAIR-L and baselines shows robust adaptivity when the Lipschitz constant $I$4 is misspecified (left) and via ablation study (right).

Further, the analysis carefully tracks the impact of the schedule's threshold mechanisms on empirical gradient variation and demonstrates tight coverage of arbitrary intervals by sets of active expert intervals, ensuring optimal adaptation to both difficult and benign sub-regions in the sequence.

(Figure 3)

Figure 4: Cumulative loss (left) and classification accuracy (right) for GAIR-L and baselines on MNIST under a non-stationary regime, validating empirical superiority of gradient-variation adaptivity and Lipschitz-agnosticity.

Practical and Theoretical Implications

Practical:

• The method's parameter-free (or near parameter-free) regime ensures robust deployment in real-world, highly dynamic, or nonparametric OCO scenarios.
• Empirical results, especially on synthetic regression with increasing $I$5 and SEA-modeled non-stationary MNIST, demonstrate that GAIR-L does not degrade under misspecified environment parameters and adapts accuracy and loss to local difficulty.

Theoretical:

• Unifies and strengthens problem-dependent interval regret, minimizing pessimism in analyses, and extends the flexibility of OCO tools to richer, piecewise stationary models and comparator classes.
• The analysis method—canceling algorithmic instability with intrinsic function curvature—may inform the design of future interval-adaptive online methods in OCO and related domains, including online meta-learning.

Future Directions:

• Developing tighter decoupling between gradient variation and small-loss for further refinement of the regret expressions.
• Extending negative Bregman-based analysis to more general online settings such as zero-sum games and unconstrained/bandit optimization.

Conclusion

This work establishes the first interval regret bounds scaling with gradient variation, a fundamental step beyond existing worst-case or small-loss analyses in OCO. By leveraging smoothness-derived cancellations and new Lipschitz-adaptive meta algorithms, it achieves "best-of-all-worlds" guarantees under minimal parameter knowledge. Extensions to interval dynamic regret and flexible adversarial-stochastic models highlight the versatility and practical relevance of the approach.