Adaptive Online Optimization Algorithm

• Adaptive online optimization algorithms are sequential decision-making methods that adjust their update rules and parameters based on observed data streams.
• They leverage a Follow-The-Regularized-Leader framework with discounting and a two-stage update strategy for magnitude and direction adjustments.
• The algorithms ensure robust performance through instance-dependent guarantees, maintaining low regret and optimality gaps in adversarial, nonstationary settings.

An adaptive online optimization algorithm is a class of sequential decision-making methods in which the update rules and, crucially, their parameters or structural components are automatically adjusted in response to observed data streams, task non-stationarity, or the geometry of the problem. The primary objective is to maintain strong performance—quantified via regret, optimality gaps, or constraint violations—across a wide spectrum of environments without prior tuning or static assumptions. Such algorithms, including the instance developed for discounted online convex optimization with adversarially chosen loss sequences and nonstationary environments, fundamentally reshape the principle of regularization and learning rate selection, offering refined guarantees that are instance-dependent and robust to distributional evolution (Zhang et al., 5 Feb 2024).

1. Problem Formulation and Notation

In the generic Online Convex Optimization (OCO) setting, a learner makes predictions $x_t \in \mathcal{X} \subseteq \mathbb{R}^d$ at each round $t$, incurs loss $l_t(x_t)$, and observes a subgradient $g_t \in \partial l_t(x_t)$. The cumulative static regret with respect to a fixed comparator $u$ is

$\Reg_T(l_{1:T}, u) = \sum_{t=1}^T [l_t(x_t) - l_t(u)].$

In adversarial and nonstationary settings, a discounted regret framework discounts earlier losses via weights $\gamma_{s,t} = \prod_{i=s}^{t-1} \lambda_i$, with $\lambda_i \in (0,1]$, yielding

$\Reg_T^{\lambda_{1:T}}(l_{1:T}, u) = \sum_{t=1}^T \gamma_{t,T}[l_t(x_t) - l_t(u)].$

Important effective quantities are defined: $$H_T = \sum_{t=1}^T \gamma_{t,T}^2, \quad V_T = \sum_{t=1}^T \gamma_{t,T}^2 \|g_t\|^2, \quad G_T = \max_{t\leq T} \gamma_{t,T}\|g_t\|.$$ Here, $H_T$ represents the effective horizon—an analog to the window size for forgetting, $V_T$ the discounted aggregate gradient variance, and $G_T$ the maximal discounted gradient norm, each critical in adaptive analysis (Zhang et al., 5 Feb 2024).

2. Algorithmic Structure and Implementation

The core algorithm is a Follow-The-Regularized-Leader (FTRL) approach with explicit discounting and data-driven regularization: $$x_{t+1} = \arg\min_{x \in \mathcal{X}} \sum_{s=1}^t \gamma_{s,t} \langle g_s, x \rangle + R_t(x),$$ with $R_t$ a time- and data-dependent regularizer parameterized by observed gradients and discount factors. Practically, the solution leverages a polar decomposition: the magnitude $y_t \geq 0$ is updated via a 1D discounted FTRL (with a convex conjugate of a parameterized "erfi-potential"), and the direction $w_t \in \mathbb{R}^d$, $\|w_t\| \leq 1$, is updated via an AdaGrad-style routine on the unit ball. The full iterate is then $x_t = y_t w_t$.

Main Loop Pseudocode (Key Steps):

1. Query 1D magnitude learner for $y_t$.
2. Query AdaGrad-ball learner for direction $w_t$.
3. Play action $x_t = y_t w_t$.
4. Observe gradient $g_t$, discount $\lambda_t$.
5. Update gradient statistics and hints for subsequent subroutine invocations.
6. Proceed to next round.

The 1D FTRL subroutine employs an explicit update for