TEWA-SE: Tilted EWA with Sleeping Experts

• TEWA-SE is a framework that combines sleeping experts with tilted exponential weighting to optimize online learning in non-stationary and adversarial environments.
• It employs adaptive weight updates using context-specific learning rates to efficiently track the best expert on active rounds.
• Applications span bandit convex optimization, fairness-aware learning, and simulation-based decision-making, ensuring strong, practical regret guarantees.

The Tilted Exponentially Weighted Average with Sleeping Experts (TEWA-SE) is a class of algorithms designed to optimally combine online learning, non-stationary environments, and partial-information feedback through a principled synthesis of sleeping experts and tilted exponential weighting. TEWA-SE approaches provide a unifying methodology for tracking the best action, policy, or expert in changing, possibly adversarial settings, often under bandit feedback or subject to complex constraints. As such, the TEWA-SE framework has had substantial influence in areas including bandit convex optimization, fairness-aware online learning, simulation-based decision-making, online function estimation, and adaptive expert aggregation.

1. Core Concepts and Algorithmic Structure

TEWA-SE generalizes the classical exponentially weighted average (EWA) methodology to settings with “sleeping experts” and structural non-stationarity. The algorithm maintains a set of experts (policies, predictors, arms), some of which may be inactive (“asleep”) at a given round due to context, feasibility, or availability constraints.

At each round $t$, the TEWA-SE meta-predictor computes a convex combination of the recommendations $x_{e,t}$ from all currently awake experts $\mathcal{A}_t$, assigning weights $w_{e,t}$ based on cumulative loss or regret, and possibly on expert-specific learning rates $\eta_e$:

$$\hat{x}_t = \frac{\sum_{e \in \mathcal{A}_t} \eta_e \exp(-L_{e,t-1}) x_{e,t}}{\sum_{e \in \mathcal{A}_t} \eta_e \exp(-L_{e,t-1})}$$

where $L_{e,t-1}$ is the cumulative surrogate loss for expert $e$ through $t-1$. The “tilted” aspect refers to the use of non-uniform learning rates or prior weightings, tailored to expert type or time-scale.

Key properties:

• Sleeping experts: Only a subset of experts participate (“awake”) at each round, reflecting feasibility, context, or resource availability.
• Iterative weight update: Weights are updated using losses only when the expert is awake; sleeping experts sustain no additional loss.
• Adaptive coverage: Multiple instances (“experts”) with different learning rates and lifetimes are maintained to provide adaptivity across multiple non-stationarity scales.

2. Sleeping Experts Mechanism and Theoretical Principles

The sleeping experts framework arises in online learning where, in each round, only certain experts are available. Standard regret guarantees are generalized: for any expert $e$, the algorithm’s regret is measured on the rounds when $e$ is awake (i.e., $t: e \in \mathcal{A}_t$):

$$\operatorname{sleepingregret}(e) = \sum_{t: e \in \mathcal{A}_t} \left( \ell_t(\hat{x}_t) - \ell_t(x_{e,t}) \right)$$

TEWA-SE extends this to settings with constraints, partial feedback, or arbitrary non-stationarity, supporting per-group, per-interval, or per-arm guarantees as appropriate to the problem context.

Notably, TEWA-SE frameworks:

• Guarantee interval/segment-wise optimality (e.g., optimal tracking of the best expert for any subinterval).
• Enable fine-grained adaptivity (e.g., adaptivity to both short and long regime switches).
• Integrate "tilting" (learning rate or prior adaptation) for improved adaptation to varying expert quality or input difficulty.

3. Application Domains and Algorithm Instantiations

TEWA-SE frameworks underpin a broad spectrum of recent advances in online, sequential, and adaptive learning:

a) Bandit Convex Optimization (BCO) and Non-Stationary Regret

TEWA-SE achieves minimax-optimal regret in BCO with switching, path-length, and total-variation non-stationarity. For strongly convex losses, the regret with respect to the best $S$-switch or $\Delta$-variation sequence achieves the lower bound up to log factors:

$$\max_{\text{comp.}} R_T \lesssim d\sqrt{S T} + d^2 S, \qquad R_T \lesssim d^{2/3}\Delta^{1/3}T^{2/3}$$

TEWA-SE accomplishes this by using a geometric covering of intervals, running experts at all relevant time scales and learning rate grids, aggregating via tilted EWA, and extending to unknown non-stationarity via Bandit-over-Bandit meta-learning.

b) Constrained Markov Decision Processes

In simulation-based algorithms for CMDPs, TEWA-SE is represented by FTAL and AUER, where only constraint-satisfying (feasible) policies are considered awake in each iteration. Convergence is guaranteed both in expectation and almost surely, with computational overhead depending only on the number of policies, independently of the state/action space size.

c) Online Fairness and Subgroup Adaptivity

Sleeping experts algorithms connected to TEWA-SE allow for online, subgroup-fair learning: for overlapping demographic populations, fairness is rephrased as low regret for "sleeping experts" corresponding to subgroup-specific predictors. TEWA-SE guarantees each group’s performance is close to its best expert, but also reveals structural impossibility for some fairness metrics (e.g., unweighted average of FPR/FNR).

d) Piecewise Regular Function Estimation

Modified sleeping experts aggregation, truncated for robustness, yields spatially adaptive estimators with simultaneous oracle risk bounds for all dyadic subregions:

$$\mathbb{E}\left[\|\hat{\theta}_{\mathsf{T}} - \theta^*_{\mathsf{T}}\|^2\right] \leq \min_{P, \theta \in \Theta_P} \|\theta^*_{\mathsf{T}} - \theta\|^2 + C \cdot |P| \log |\mathcal{S}|$$

This delivers online minimax optimality for bounded variation and piecewise polynomial classes, notably exceeding guarantees of classical batch estimators regarding local adaptivity.

e) Adaptive Expert Aggregation in Forecasting

Integration of the sleeping expert framework into expert aggregation for applications such as temperature prediction allows sometimes-biased predictors (specialists for extremes) to contribute adaptively. Activation of these experts is learned online, e.g., via gradient boosted regression trees, with second-order regret guarantees and meta-aggregation (FTL) for robustness.

4. Regret Guarantees, Bounds, and Surrogate Losses

TEWA-SE and its variants are associated with strong, sometimes minimax-optimal, regret bounds under non-stationarity:

• Interval/adaptive regret: $O(\sqrt{B})$ for intervals of length $B$ (strongly convex loss).
• Switching and dynamic regret: Scaling as $\sqrt{S T}$, $\Delta^{1/3} T^{2/3}$, or their appropriate equivalents for the problem.
• Per-action and per-group regret: Bounds of $O(\sqrt{T(g) \log N})$ for group size $T(g)$, expert set size $N$.

These are typically achieved by applying loss estimates (often surrogates: e.g., one-point gradient estimates, quadratic relaxations) and exponential weighting restricted to the awake set, with tilting by learning rate or prior.

5. Computational and Practical Considerations

The computational cost of TEWA-SE scales polylogarithmically in the horizon and polynomially in dimension and the number of experts:

• Scalability: For CMDPs and BCO, computational complexity is independent of state/action space size, given feasible simulation or gradient estimation per expert.
• Simulation and feedback: All major TEWA-SE instantiations are designed for online operation with stochastic, partial, or bandit feedback, for which only local ("awake") updates are needed per active expert per round.
• Extensibility: The framework supports black-box, simulation-based, or adversarial-environment applications, accommodating uncertainty in non-stationarity via meta-learning wrappers (e.g., Bandit-over-Bandit).

6. Open Problems and Limitations

TEWA-SE variants face several theoretical and practical limitations:

• Fairness metrics limitation: Under overlapping population constraints, certain fairness metrics (e.g., FPR/FNR averaging) are unachievable for all groups using any online sleeping expert approach.
• Incentive compatibility vs. efficiency: Achieving both individual rationality and incentive compatibility requires potentially exponential computation, which is infeasible for many subgroups or intersectional structures.
• Regret bounds in high dimension or for general convexity: While minimax rates are achieved for strongly convex losses, path-length regret for general convex losses remains suboptimal (typically $T^{3/4}$ rate with bandit feedback).
• Optimality under partial feedback: Regret bounds for the “one-sided feedback” (apple-tasting) model are not tight for all scenarios, especially with many subgroups or arms.

7. Broader Impact and Directions

TEWA-SE forms a cornerstone methodology for robust, adaptive, and scalable online learning in dynamic, constrained, and partial-information environments. Its algorithmic structure facilitates unifying analysis, implementation simplicity (through exponential weighting and local updates), and rigorous performance guarantees for highly non-stationary problems. Future research may further close suboptimality gaps for general convex losses, develop computationally efficient incentive-compatible versions for group fairness, and extend the approach to richer feedback models and contextual action spaces.