---
title: Problem-Dependent Regret Measure
url: https://www.emergentmind.com/topics/problem-dependent-regret-measure
type: topic
---

# Problem-Dependent Regret Measure

A problem-dependent regret measure is any notion of regret in sequential decision-making or online learning whose upper or lower bound is not uniform over all problem instances, but instead depends on specific parameters, structure, or observed data of the instance at hand. This category encompasses a wide range of formally distinct regret functionals and instance-dependent complexity measures developed across online convex optimization, bandits, reinforcement learning, decision theory, and control. Problem-dependent regret arises whenever the measure of learner performance adaptively scales with aspects such as gaps, variances, loss sequences, gradients, or ambiguity weights, thereby sharply distinguishing "easy" from "hard" problem instances.

## 1. Definitions and Formal Criteria

In classical minimax analysis, regret is typically bounded in terms of adversarial or worst-case instance parameters (e.g., time horizon $T$, number of arms $K$, dimension $d$, horizon $H$) with no reference to special characteristics of a given problem instance. In contrast, a regret measure is termed problem- or instance-dependent if it scales with observable or structural parameters that vary meaningfully between instances:

- **Gap-dependent regret:** Depends on gaps between optimal and suboptimal actions or policies, e.g., $\sum_{i: \Delta_i>0}\Delta_i / KL(\mu_i,\mu^*)$ in bandits, or policy/action gaps in MDPs [2106.13013, 2203.03110].
- **Variance-dependent regret:** Scales with empirical variances of the reward/noise, e.g., $\widetilde O(d\sqrt{\sum_k \sigma_k^2})$ for bandits/RL [2302.10371].
- **Gradient-variation/variation-based regret:** In online convex optimization, the regret is bounded by cumulative changes in the gradients or variations of loss functions, e.g., $O(\sqrt{V_T})$ where $V_T = \sum_{t=2}^T \sup_x \|\nabla f_t(x) - \nabla f_{t-1}(x)\|^2$ [2511.19937, 2112.14368].
- **Small-loss/adaptive regret:** Scales with cumulative loss of the best comparator or adaptive comparators over intervals, e.g., $O(\sqrt{L_r^s})$ where $L_r^s$ is interval cumulative loss [1904.11681].
- **Data-dependent regret:** Explicitly depends on the realized loss sequence, often via empirical range, variance, or structure of the observed data [2303.06526, 2505.20010].
- **Ambiguity-weighted regret:** Uses a weighted set of priors (e.g., MWER), where bounds vary depending on the agent's subjective confidence in model components [1302.5681].
- **Instance-complexity based regret:** Scales with intrinsic measures such as instance-specific effective dimension, shell packings, or other instance-wise complexity terms (e.g., kernelized bandits with shell packing sum $C(\Delta)$) [2203.06297].

By incorporating such quantities, the problem-dependent regret measure captures the true difficulty of an instance, potentially providing tighter and more informative theoretical and algorithmic guarantees than minimax analysis.

## 2. Key Examples Across Frameworks

### Decision Theory and MWER

- **Minimax Weighted Expected Regret (MWER):**
  $$
  \mathrm{MWER}_{M, P^+}(f) = \sup_{\mu \in P} \alpha_\mu \cdot \mathbb{E}_\mu\left[\max_{g \in M} u(g(s)) - u(f(s))\right]
  $$
  Here, the $\alpha_\mu$ encode *problem-dependent beliefs* (e.g., based on expert knowledge, historical data, or likelihood updates), producing an adaptive regret criterion. Likelihood updating of the $\alpha_\mu$ ensures that, as data accrues, MWER converges to subjective expected utility under the true generating measure, adapting to each instance [1302.5681].

### Online Learning and Bandits

- **Range- and variance-based regret**: For adversarial bandits or experts, data-dependent regret can be written as $R_T \leq A\max_t\Delta_t + B\sqrt{\sum_{t=1}^T\Delta_t^2}$, where $\Delta_t$ are realized per-round loss ranges, capturing the problem's heteroscedasticity or observed variability [2303.06526].
- **Kernelized bandits, instance-complexity**: Lower bound and algorithmic guarantee are determined by $C(\Delta) = \sum_k m_k/(2^{k+2}\Delta)$, where $m_k$ are packing numbers of shells defined by the function's value structure around its maximizer. Thus the regret interpolation strongly depends on instance geometry and the function's local behavior [2203.06297].
- **Constrained MABs, decomposed regret**: Regret bounds take the form $O\left(\frac{K}{\rho}\sqrt{\sum_t (\ell_t^\top(x^\diamond - x^*))^2} + \frac{1}{\rho}\sqrt{K\sum_t \ell_t^\top x^*}\right)$, where the first term is a data-dependent safety complexity and the second is a bandit-complexity, both problem-specific [2505.20010].

### Reinforcement Learning

- **Gap-based lower bounds for MDPs:** Problem-dependent lower bounds for regret in MDPs are given by $\min_{\eta\ge0} \sum_{s,a,h} \eta_h(s,a)\Delta_h(s,a)$ subject to information-theoretic and flow constraints determined by the MDP's structure, with the solution scaling with policy gaps and dynamics [2106.13013].
- **Variance/environmental norm in RL:** High-probability regret bounds of the form $\tilde{O}(\sqrt{S A T} + \sigma_*)$, where $\sigma_*$ is the "environmental norm," (maximum conditional variance of one-step value) adaptively interpolate between worst-case and easy instances with flat or deterministic transitions [1901.00210].
- **Variance-dependent regrets in linear RL:** Horizon-free, problem-specific regret scaling as $O\left(d\sqrt{\mathrm{Var}_K^*} + d^2\right)$, where $\mathrm{Var}_K^*$ is the cumulative variance under the optimal policy [2302.10371].
- **Gap-dependent, risk-sensitive RL:** "Cascaded gaps" generalize risk-neutral gap analysis to risk-sensitive settings, controlling regret as $O\left((e^{|\beta|H} - 1)^2/(\beta^2\,\Delta_{\min,\beta})\right)$, where $\Delta_{\min,\beta}$ encodes the cascaded problem-dependent gaps related to the nonlinearity and risk parameter $\beta$ [2203.03110].

### Online Convex Optimization

- **Gradient-variation regret:** Adaptive regret bounds such as $O(\sqrt{V_T})$ (gradient-variation), $O(\sqrt{L_r^s})$ (interval-comparator loss), and best-of-small-loss or best-of-gradient-variance rates provide instance-adaptive learning guarantees [2511.19937, 2112.14368, 1904.11681, 2303.06526].

## 3. Methodologies for Achieving Problem-Dependent Regret

Several methodological paradigms are regularly employed to realize or analyze problem-dependent regret:

- **Algorithmic adaptivity:** Design of algorithms (often via second-order, scale-free, or meta-learning techniques) that do not require prior knowledge of the problem-dependent quantity but automatically adapt to its realized value (e.g., gradient-var adaptive OCO, data-adaptive bandits, likelihood-updating of ambiguity weights) [2511.19937, 1904.11681, 2505.20010, 1302.5681].
- **Information-theoretic constraints:** Lower bounds derived via change-of-measure or KL-based arguments where the required "exploration budget" per instance arises from the instance's intrinsic statistical or structural distinguishability properties [2106.13013, 2106.14338].
- **Instance-specific complexity measures:** Introduction of explicit complexity measures (e.g., environmental norm $\sigma_*$, shell-packing numbers, cascaded gaps, path length, or comparator class complexity $W$) which capture the structural elements governing the learning challenge in a concrete instance [1901.00210, 2203.06297, 2203.03110, 2303.06526].
- **Dynamic updating and belief revision:** Bayesian, likelihood-based, or menu-expansion-weight updating schemes ensure that regret adapts as learning progresses (e.g., MWER's weight updating, dynamic ensemble OCO frameworks) [1302.5681, 2112.14368].

## 4. Axiomatic and Structural Foundations

Problem-dependent regret measures often require nuanced axiomatization to justify and delineate their properties:

- **MWER characterization:** An explicit axiomatization (transitivity, monotonicity, ambiguity aversion, etc.) yields an equivalence between preference relations and minimax weighted expected regret, with further dynamic consistency axioms ensuring stability under information flow [1302.5681].
- **Dynamic decision theory:** Menu-selection rules parameterize which set of options enter the regret calculation, inducing problem-dependence (e.g., inclusion or exclusion of forgone opportunities), with dynamic consistency tied to the belief structure and updating mode [1502.00152].
- **Regret kernels and stochastic dominance:** For probabilistic lotteries, axiomatic requirements such as transitivity, stochastic dominance, and super-additivity uniquely characterize specific (problem-dependent) regret kernels that match observed choice behavior [2301.02447].

## 5. Practical and Theoretical Consequences

Problem-dependent regret analysis delivers several theoretical and algorithmic benefits:

- **Sharper performance guarantees:** Boundaries between easy and hard instances become explicit (e.g., constant regret in deterministic or concentrated loss settings; logarithmic rather than $\sqrt{T}$ scaling in small-loss regimes) [2303.06526, 2505.20010, 1904.11681].
- **Learning and inference:** In settings with ambiguity (e.g., MWER), adaptive revision of confidence in hypotheses allows regret to converge to the fundamental limit as the learner identifies the true environment [1302.5681].
- **Policy design and selection:** In dynamic or menu-dependent settings, the designer can select menus, update rules, and belief structures to recover or break time-consistency, adjust the extent of exploration, or mitigate undesirable behaviors such as procrastination [1502.00152].
- **Instance-optimality:** Lower and upper bounds, incorporating problem-specific structure, certify that no uniform algorithm can improve over the problem-dependent scaling (instance-optimality) [2106.13013, 2203.06297].
- **Robustness and universality:** Meta-ensembles and scale-free strategies ensure minimax optimality in worst-case regimes while automatically adapting to benign structure, thus obviating the need for adversarial tuning [2511.19937].

## 6. Limitations and Open Questions

Despite their advantages, problem-dependent regret measures can present several subtleties:

- **Menu- and set-dependence:** Menu-dependent regret (as in MWER and dynamic regret minimization) can violate classic decision-theoretic axioms (e.g., the independence of irrelevant alternatives), creating subtleties in both static and dynamic modeling [1302.5681, 1502.00152].
- **Necessity of careful updating rules:** Without appropriate likelihood updating or dynamic consistency axioms, time-inconsistent or even paradoxical preference reversals may occur [1302.5681, 1502.00152].
- **Instance-complexity estimation:** For certain measures (e.g., environmental norm in RL, shell packing in bandits), computation of the problem complexity may not be tractable without post hoc data or may demand domain knowledge [1901.00210, 2203.06297].
- **Incomplete generality:** Not every sharp upper bound is known to be instance-optimal for all settings; in some domains matching upper/lower bounds or robustly adaptive methods remain open research directions [2302.10371].

## 7. Comparative Summary Table

| Domain                      | Problem-Dependent Measure     | Instance Parameter(s)                        | Reference        |
|-----------------------------|------------------------------|----------------------------------------------|------------------|
| Decision theory             | MWER (weighted regret)       | Ambiguity weights $\alpha_\mu$               | [1302.5681]      |
| Bandits (general)           | Data-dependent regret        | Per-round observed ranges, variances         | [2303.06526]     |
| Kernel bandits              | Instance complexity $C(\Delta)$ | Shell packing/geometry of $f$                | [2203.06297]     |
| Convex OCO                  | Gradient-variation regret    | $V_T$, path-length, comparator loss          | [2511.19937]     |
| RL (episodic tabular)       | Environmental norm           | $\sigma_\star^2 = \max_{s,a,t} \Var[R+V]$    | [1901.00210]     |
| RL (finite-horizon)         | Gap-based LP                 | Policy/action gaps, KL under dynamics        | [2106.13013]     |
| RL (risk-sensitive)         | Cascaded gap                 | Nonlinear Bellman gap, $\beta$, $H$          | [2203.03110]     |
| Linear RL/bandits           | Variance-dependent regret    | Realized noise/variance sequence             | [2302.10371]     |
| Constrained bandits         | Decomposed (safety/bandit)   | Loss gap of Slater point, loss sum           | [2505.20010]     |

Problem-dependent regret thus provides a unified yet highly granular quantification of learning complexity, simultaneously enabling sharper quantitative analysis, improved instance-adaptive algorithms, and deeper structural insights into the foundations of sequential decision-making.

Source: https://www.emergentmind.com/topics/problem-dependent-regret-measure