Graves-Lai Optimization Problem

• The Graves–Lai optimization problem is a framework for deriving exact asymptotic regret bounds and guides the optimal exploration in structured bandits.
• It generalizes the classical Lai–Robbins lower bound to settings like combinatorial semi-bandits and multimodal bandits using constrained optimization techniques.
• A convex reduction with sparse decomposition enables a polynomial-time solution, making the approach computationally tractable for practical bandit models.

The Graves–Lai optimization problem is the lower-bound/optimization framework that identifies the exact asymptotic constant governing regret for uniformly good algorithms in structured bandits, and it also specifies the exploration rates required by asymptotically optimal strategies. In combinatorial semi-bandits with uncorrelated Gaussian rewards, it characterizes the minimum logarithmic-scale sampling effort needed to distinguish the true parameter from statistically confusing alternatives, while in later work on multimodal bandits it plays the same role for tree-structured mean-reward functions (Cuvelier et al., 2021).

1. Conceptual role in asymptotic bandit theory

The Graves–Lai framework generalizes the Lai–Robbins lower bound from classical bandits to structured bandits. Its central object is an optimization problem whose optimum gives the constant in the logarithmic regret lower bound for uniformly good algorithms, where “uniformly good” means that regret is subpolynomial in $T$ for every fixed instance. In the notation used for combinatorial semi-bandits, any uniformly good algorithm satisfies

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$

where $C(\theta)$ is the optimum value of the Graves–Lai optimization problem $P_{GL}$ (Cuvelier et al., 2021).

A recurring misconception is to regard the Graves–Lai problem as only a lower-bound certificate. The converse statement recorded for the combinatorial semi-bandit setting is that, if one can solve the Graves–Lai optimization problem efficiently, then there exist asymptotically optimal algorithms whose regret matches this lower bound asymptotically: $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$ The optimization therefore functions not only as an information-theoretic benchmark but also as the prescription for how much exploration each suboptimal action must receive in any asymptotically optimal strategy (Cuvelier et al., 2021).

In later multimodal bandit work, the same interpretation is made explicit through an asymptotic sampling-rate vector $\boldsymbol{\eta}$, with

$\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$

for each suboptimal arm $k$, where $\boldsymbol{\eta}^\star$ solves the relevant Graves–Lai program. This reinforces the status of the problem as the canonical asymptotic exploration design principle across structured bandit models (Réveillard et al., 29 Oct 2025).

2. Specialized formulation for combinatorial semi-bandits

In the combinatorial semi-bandit model, the learner chooses a binary decision vector $x(t)\in X\subset\{0,1\}^d$. The environment generates

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$0

so the coordinates $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$1 are independent Gaussian rewards with means $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$2 and variance $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$3. The learner observes semi-bandit feedback $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$4, the scalar reward is

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$5

and regret is measured against the best fixed action $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$6: $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$7 The combinatorial structure is essential because $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$8 is typically exponential in $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$9, so direct optimization over actions is generally not polynomial-time in the ambient dimension (Cuvelier et al., 2021).

For this setting, the Graves–Lai program is specialized as

$C(\theta)$0

subject to

$C(\theta)$1

Here

$C(\theta)$2

is the gap of action $C(\theta)$3, and

$C(\theta)$4

is the set of non-optimal items, namely those coordinates that never appear in any optimal decision (Cuvelier et al., 2021).

The variable $C(\theta)$5 represents how often action $C(\theta)$6 is played on the logarithmic scale, approximately $C(\theta)$7 times. The objective $C(\theta)$8 is exactly the asymptotic regret cost of sampling each suboptimal action at that rate. The formulation thus converts asymptotic exploration design into a constrained optimization over action frequencies (Cuvelier et al., 2021).

3. Statistical meaning of the constraints

The constraint

$C(\theta)$9

encodes the statistical indistinguishability requirement. The denominator

$P_{GL}$0

is the asymptotic number of observations of coordinate $P_{GL}$1. The left-hand side therefore measures whether the available information is sufficient to rule out the suboptimal action $P_{GL}$2: if the quantity is too large, the learner has not collected enough evidence on the non-optimal coordinates relevant for distinguishing $P_{GL}$3 from the optimum (Cuvelier et al., 2021).

The set $P_{GL}$4 isolates exactly the coordinates whose means must be learned through explicit exploration. If a coordinate can appear in an optimal action, it can be learned “for free” by repeatedly sampling optimal actions; if it never appears in an optimal action, then asymptotically optimal learning must allocate dedicated exploration to it. This distinction is built directly into the specialized combinatorial semi-bandit program through the index set $P_{GL}$5 (Cuvelier et al., 2021).

The specialized program is derived from the more general Graves–Lai lower-bound form

$P_{GL}$6

with

$P_{GL}$7

Because rewards are Gaussian with variance $P_{GL}$8,

$P_{GL}$9

The derivation decomposes $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$0 by suboptimal action $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$1 and offset $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$2, solves the inner quadratic minimization by KKT conditions, and recovers the explicit constraint above (Cuvelier et al., 2021).

4. Naive size and tractability barriers

Although the specialized program is concise at the level of notation, its naive representation is computationally prohibitive. It has $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$3 variables and $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$4 constraints, and $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$5 is typically exponential in $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$6. As a result, even checking feasibility is not obviously polynomial-time, and outputting a full solution is itself infeasible unless the optimal support is sparse (Cuvelier et al., 2021).

The tractability question is especially relevant for canonical combinatorial families in which the action set is exponentially large but linear optimization is tractable. The polynomial-time result in the Gaussian semi-bandit setting applies to $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$7-sets, spanning trees, matroid bases, source-destination paths in DAGs, bipartite matchings, and the intersection of two matroids. These are the classes for which the paper establishes the first method, to the best of its knowledge, for computing the solution of the Graves–Lai optimization problem in polynomial time (Cuvelier et al., 2021).

The assumptions used to obtain this tractability are explicit. They are: covering; integrality $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$8, together with an explanation that real-valued $$\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).$$9 can be discretized to arbitrary accuracy; polynomial-time linear maximization $\boldsymbol{\eta}$0; polynomial-time budgeted linear maximization $\boldsymbol{\eta}$1, exactly or approximately; and a compact convex-hull representation

$\boldsymbol{\eta}$2

with $\boldsymbol{\eta}$3 of polynomial size. These are presented as standard tractability assumptions for the covered combinatorial classes (Cuvelier et al., 2021).

5. Polynomial-time solution for the combinatorial Gaussian case

The first technical step is a dimensionality reduction from action frequencies $\boldsymbol{\eta}$4 to coordinate-sample counts

$\boldsymbol{\eta}$5

The paper proves that the optimum of $\boldsymbol{\eta}$6 can be recovered from a reduced problem $\boldsymbol{\eta}$7 with only $\boldsymbol{\eta}$8 variables: $\boldsymbol{\eta}$9 subject to

$\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$0

The constants are

$\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$1

The reduction is based on the fact that

$\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$2

lies in a set described by the convex hull of $\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$3, allowing the relation $\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$4 with $\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$5 to be converted into $\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$6, while the original objective becomes $\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$7 (Cuvelier et al., 2021).

The reduced problem remains constrained by one combinatorial inequality for each $\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$8, so the paper introduces penalization. Defining

$\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T$9

it solves

$k$0

subject to $k$1, $k$2, and $k$3, where $k$4. The optimization is carried out by projected subgradient descent,

$k$5

with

$k$6

and output

$k$7

The explicit gradient is

$k$8

The approximate maximizer $k$9 of the constraint violation can be obtained by solving $\boldsymbol{\eta}^\star$0 exactly or approximately a polynomial number of times (Cuvelier et al., 2021).

With suitable parameters $\boldsymbol{\eta}^\star$1, the rescaled average $\boldsymbol{\eta}^\star$2 is proved to be an $\boldsymbol{\eta}^\star$3-optimal solution to $\boldsymbol{\eta}^\star$4. The guarantees are stated as

$\boldsymbol{\eta}^\star$5

in the exact version, and

$\boldsymbol{\eta}^\star$6

in the approximate version, while all constraints remain satisfied. The running time is polynomial in $\boldsymbol{\eta}^\star$7, $\boldsymbol{\eta}^\star$8, and $\boldsymbol{\eta}^\star$9 (Cuvelier et al., 2021).

A final reconstruction step is needed to recover an action-space solution $x(t)\in X\subset\{0,1\}^d$0. This is achieved by a Carathéodory-style iterative decomposition showing that

$x(t)\in X\subset\{0,1\}^d$1

can be represented with only $x(t)\in X\subset\{0,1\}^d$2 nonzero coefficients, and more specifically

$x(t)\in X\subset\{0,1\}^d$3

with positive coefficients. The procedure runs in polynomial time because each $x(t)\in X\subset\{0,1\}^d$4 is obtained by linear maximization over $x(t)\in X\subset\{0,1\}^d$5 using $x(t)\in X\subset\{0,1\}^d$6. The resulting GLPG algorithm computes a $x(t)\in X\subset\{0,1\}^d$7-accurate solution to $x(t)\in X\subset\{0,1\}^d$8 in time polynomial in $x(t)\in X\subset\{0,1\}^d$9, $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$00, and $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$01; if only approximate budgeted maximization is available, the solution is $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$02-optimal and yields asymptotic regret within a factor $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$03 of the Graves–Lai lower bound (Cuvelier et al., 2021).

6. Broader variants and later tractable instances

The computational role of the Graves–Lai problem extends beyond combinatorial semi-bandits. In multimodal bandits on a known tree $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$04, where the unknown mean vector is assumed to have at most $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$05 modes, the lower bound is again expressed through a Graves–Lai program: $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$06 with

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$07

Here the difficulty is no longer an exponentially large action family but the set $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$08, which for $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$09 is highly nonconvex and disconnected (Réveillard et al., 29 Oct 2025).

That work proves that representing $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$10 as a union of convex sets requires exponentially many components in $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$11: $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$12 It then introduces structural reductions, including decomposition by the wrong optimal arm, a compact reformulation for non-neighbor arms, and a location-of-modes lemma stating that for the relevant optimizer $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$13,

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$14

This leads to subproblems indexed by $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$15, discretization on a grid

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$16

and a dynamic program on the rooted tree $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$17 (Réveillard et al., 29 Oct 2025).

The dynamic program computes each $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$18 subproblem in time and memory $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$19, giving total complexity $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$20 and space $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$21 per iteration of the outer loop, with an appendix improvement reducing the discrete subproblem to $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$22. The outer Graves–Lai optimization is then handled by a penalized projected subgradient method over $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$23, producing a feasible approximate solution after $$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$24 iterations with total time

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$25

and space

$$\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),$$26

The paper states that this is the first known computationally tractable algorithm for computing the solution to the Graves–Lai optimization problem in that multimodal setting (Réveillard et al., 29 Oct 2025).

A plausible implication is that the Graves–Lai optimization problem is a unifying object across disparate structured bandit classes, while the algorithmic route to tractability depends sharply on the geometry of the alternative set: exponential action spaces in combinatorial semi-bandits can be handled by convex reduction and sparse decomposition, whereas disconnected structural constraints in multimodal bandits require decomposition, discretization, and tree dynamic programming. Both lines of work, however, preserve the same central interpretation: the Graves–Lai solution is the asymptotically optimal exploration profile.