---
title: Graves-Lai Optimization Problem
url: https://www.emergentmind.com/topics/graves-lai-optimization-problem
type: topic
---

# Graves-Lai Optimization Problem

The Graves–Lai optimization problem is the lower-bound/optimization framework that identifies the exact asymptotic constant governing regret for uniformly good algorithms in structured bandits, and it also specifies the exploration rates required by asymptotically optimal strategies. In combinatorial semi-bandits with uncorrelated Gaussian rewards, it characterizes the minimum logarithmic-scale sampling effort needed to distinguish the true parameter from statistically confusing alternatives, while in later work on multimodal bandits it plays the same role for tree-structured mean-reward functions [2102.07254].

## 1. Conceptual role in asymptotic bandit theory

The Graves–Lai framework generalizes the Lai–Robbins lower bound from classical bandits to structured bandits. Its central object is an optimization problem whose optimum gives the constant in the logarithmic regret lower bound for uniformly good algorithms, where “uniformly good” means that regret is subpolynomial in \(T\) for every fixed instance. In the notation used for combinatorial semi-bandits, any uniformly good algorithm satisfies
\[
\liminf_{T\to\infty}\frac{R(T,\theta)}{\ln T}\ge C(\theta),
\]
where \(C(\theta)\) is the optimum value of the Graves–Lai optimization problem \(P_{GL}\) [2102.07254].

A recurring misconception is to regard the Graves–Lai problem as only a lower-bound certificate. The converse statement recorded for the combinatorial semi-bandit setting is that, if one can solve the Graves–Lai optimization problem efficiently, then there exist asymptotically optimal algorithms whose regret matches this lower bound asymptotically:
\[
\limsup_{T\to\infty}\frac{R(T,\theta)}{\ln T}\le C(\theta).
\]
The optimization therefore functions not only as an information-theoretic benchmark but also as the prescription for how much exploration each suboptimal action must receive in any asymptotically optimal strategy [2102.07254].

In later multimodal bandit work, the same interpretation is made explicit through an asymptotic sampling-rate vector \(\boldsymbol{\eta}\), with
\[
\mathbb{E}[N_k(T)] \approx \eta_k^\star \log T
\]
for each suboptimal arm \(k\), where \(\boldsymbol{\eta}^\star\) solves the relevant Graves–Lai program. This reinforces the status of the problem as the canonical asymptotic exploration design principle across structured bandit models [2510.25811].

## 2. Specialized formulation for combinatorial semi-bandits

In the combinatorial semi-bandit model, the learner chooses a binary decision vector \(x(t)\in X\subset\{0,1\}^d\). The environment generates
\[
Y(t)\sim \mathcal{N}(\theta,\tfrac12 I_d),
\]
so the coordinates \(Y_i(t)\) are independent Gaussian rewards with means \(\theta_i\) and variance \(1/2\). The learner observes semi-bandit feedback \(x(t)\odot Y(t)\), the scalar reward is
\[
Y(t)^\top x(t),
\]
and regret is measured against the best fixed action \(x^\star\in\arg\max_{x\in X}\theta^\top x\):
\[
R(T,\theta)=T\left(\max_{x\in X}\theta^\top x\right)-\sum_{t=1}^T \mathbb{E}\big[\theta^\top x(t)\big].
\]
The combinatorial structure is essential because \(|X|\) is typically exponential in \(d\), so direct optimization over actions is generally not polynomial-time in the ambient dimension [2102.07254].

For this setting, the Graves–Lai program is specialized as
\[
\underset{\alpha \in \mathbb{R}_+^{|X|}}{\text{minimize}} \quad \sum_{x \in X}\alpha_x \Delta_x
\]
subject to
\[
\sum_{i \in I}\frac{x_i}{\sum_{y\in X}y_i\alpha_y}\le \Delta_x^2,\qquad \forall x\in X.
\]
Here
\[
\Delta_x=\theta^\top x^\star-\theta^\top x
\]
is the gap of action \(x\), and
\[
I=\left\{i\in\{1,\dots,d\}:\max_{x\in X:x_i=1}\theta^\top x<\max_{x\in X}\theta^\top x\right\}
\]
is the set of non-optimal items, namely those coordinates that never appear in any optimal decision [2102.07254].

The variable \(\alpha_x\) represents how often action \(x\) is played on the logarithmic scale, approximately \(\alpha_x\ln T\) times. The objective \(\sum_x\alpha_x\Delta_x\) is exactly the asymptotic regret cost of sampling each suboptimal action at that rate. The formulation thus converts asymptotic exploration design into a constrained optimization over action frequencies [2102.07254].

## 3. Statistical meaning of the constraints

The constraint
\[
\sum_{i\in I}\frac{x_i}{\sum_{y\in X}y_i\alpha_y}\le \Delta_x^2
\]
encodes the statistical indistinguishability requirement. The denominator
\[
\sum_{y\in X}y_i\alpha_y
\]
is the asymptotic number of observations of coordinate \(i\). The left-hand side therefore measures whether the available information is sufficient to rule out the suboptimal action \(x\): if the quantity is too large, the learner has not collected enough evidence on the non-optimal coordinates relevant for distinguishing \(x\) from the optimum [2102.07254].

The set \(I\) isolates exactly the coordinates whose means must be learned through explicit exploration. If a coordinate can appear in an optimal action, it can be learned “for free” by repeatedly sampling optimal actions; if it never appears in an optimal action, then asymptotically optimal learning must allocate dedicated exploration to it. This distinction is built directly into the specialized combinatorial semi-bandit program through the index set \(I\) [2102.07254].

The specialized program is derived from the more general Graves–Lai lower-bound form
\[
\min_{\alpha\in\mathbb{R}_+^{|X|}} \sum_{x\in X}\alpha_x\Delta_x
\quad\text{subject to}\quad
\min_{\lambda\in B(\theta)} \left\{\sum_{i=1}^d\sum_{x\in X}\alpha_x x_i D(\theta_i,\lambda_i)\right\}\ge 1,
\]
with
\[
B(\theta)=\left\{\lambda\in\mathbb{R}^d: \lambda^\top x^\star<\max_{x\in X}\lambda^\top x,\ \theta_i=\lambda_i\ \forall i\notin I\right\}.
\]
Because rewards are Gaussian with variance \(1/2\),
\[
D(\theta_i,\lambda_i)=(\theta_i-\lambda_i)^2.
\]
The derivation decomposes \(B(\theta)\) by suboptimal action \(x\) and offset \(v>0\), solves the inner quadratic minimization by KKT conditions, and recovers the explicit constraint above [2102.07254].

## 4. Naive size and tractability barriers

Although the specialized program is concise at the level of notation, its naive representation is computationally prohibitive. It has \(|X|\) variables and \(|X|\) constraints, and \(|X|\) is typically exponential in \(d\). As a result, even checking feasibility is not obviously polynomial-time, and outputting a full solution is itself infeasible unless the optimal support is sparse [2102.07254].

The tractability question is especially relevant for canonical combinatorial families in which the action set is exponentially large but linear optimization is tractable. The polynomial-time result in the Gaussian semi-bandit setting applies to \(m\)-sets, spanning trees, matroid bases, source-destination paths in DAGs, bipartite matchings, and the intersection of two matroids. These are the classes for which the paper establishes the first method, to the best of its knowledge, for computing the solution of the Graves–Lai optimization problem in polynomial time [2102.07254].

The assumptions used to obtain this tractability are explicit. They are: covering; integrality \(\theta\in\mathbb{N}^d\), together with an explanation that real-valued \(\theta\) can be discretized to arbitrary accuracy; polynomial-time linear maximization \(P_{LM}\); polynomial-time budgeted linear maximization \(P_{BLM}\), exactly or approximately; and a compact convex-hull representation
\[
\mathrm{conv}(X)=\{w\in\mathbb{R}^d:Aw=b,\ w\ge 0\},
\]
with \(A,b\) of polynomial size. These are presented as standard tractability assumptions for the covered combinatorial classes [2102.07254].

## 5. Polynomial-time solution for the combinatorial Gaussian case

The first technical step is a dimensionality reduction from action frequencies \(\alpha_x\) to coordinate-sample counts
\[
w_i=\sum_{x\in X}x_i\alpha_x.
\]
The paper proves that the optimum of \(P_{GL}\) can be recovered from a reduced problem \(P_{GL}'\) with only \(d\) variables:
\[
\underset{w\in\mathbb{R}^d}{\text{minimize}} \quad q^\top w
\]
subject to
\[
\sum_{i\in I}\frac{x_i}{w_i}\le \Delta_x^2,\quad \forall x\in X,\qquad Mw=0,\qquad w\ge 0,\qquad \min_{i\in I}w_i\ge \underline w.
\]
The constants are
\[
M\equiv A-\frac{bb^\top A}{\|b\|^2},\qquad
q\equiv (\theta^\top x^\star)\frac{b^\top A}{\|b\|^2}-\theta,\qquad
\underline w\equiv (m\|\theta\|_\infty)^{-2}.
\]
The reduction is based on the fact that
\[
\left(\sum_x x\alpha_x,\ \sum_x\alpha_x\right)
\]
lies in a set described by the convex hull of \(X\), allowing the relation \(Aw=vb\) with \(v=\sum_x\alpha_x\) to be converted into \(Mw=0\), while the original objective becomes \(q^\top w\) [2102.07254].

The reduced problem remains constrained by one combinatorial inequality for each \(x\in X\), so the paper introduces penalization. Defining
\[
h_x(w)=\left(\sum_{i\in I}\frac{x_i}{w_i}\right)-\Delta_x^2,
\]
it solves
\[
\underset{w\in\mathbb{R}^d}{\text{minimize}} \quad q^\top w+\lambda \max_{x\in X}\big(h_x(w)\big)^+
\]
subject to \(Mw=0\), \(w\ge 0\), and \(\min_{i\in I}w_i\ge \underline w\), where \((\cdot)^+=\max(\cdot,0)\). The optimization is carried out by projected subgradient descent,
\[
w^{t+1}=\Pi_M\{w^t-\eta g^t\},
\]
with
\[
g^t=q+\lambda \varepsilon \nabla h_{x^t}(\varepsilon w^t)\,\mathbf{1}\{h_{x^t}(\varepsilon w^t)>0\},
\]
and output
\[
\bar w=\frac1T\sum_{t=1}^T w^t.
\]
The explicit gradient is
\[
\nabla h_{x^t}(\varepsilon w^t) = -\left(\frac{x_1^t}{(\varepsilon w_1^t)^2},\dots,\frac{x_d^t}{(\varepsilon w_d^t)^2}\right).
\]
The approximate maximizer \(x^t\) of the constraint violation can be obtained by solving \(P_{BLM}\) exactly or approximately a polynomial number of times [2102.07254].

With suitable parameters \(\delta_1,\delta_2,\lambda,T,\eta\), the rescaled average \(\bar w'=(1+\delta_2)\bar w\) is proved to be an \((\varepsilon,\delta)\)-optimal solution to \(P_{GL}'\). The guarantees are stated as
\[
q^\top \bar w' \le q^\top(w^\star/\varepsilon)+\delta
\]
in the exact version, and
\[
q^\top \bar w' \le \frac{1}{\varepsilon}C(\theta)+\delta
\]
in the approximate version, while all constraints remain satisfied. The running time is polynomial in \(d\), \(\delta\), and \(\|\theta\|_\infty\) [2102.07254].

A final reconstruction step is needed to recover an action-space solution \(\alpha\in\mathbb{R}_+^{|X|}\). This is achieved by a Carathéodory-style iterative decomposition showing that
\[
w^\star=\sum_{x\in X}x\,\alpha_x
\]
can be represented with only \(O(d)\) nonzero coefficients, and more specifically
\[
w^\star=\sum_{k=1}^d x^k\alpha_{x^k},
\]
with positive coefficients. The procedure runs in polynomial time because each \(x^k\) is obtained by linear maximization over \(X\) using \(P_{LM}\). The resulting GLPG algorithm computes a \(\delta\)-accurate solution to \(P_{GL}\) in time polynomial in \(d\), \(\delta\), and \(\|\theta\|_\infty\); if only approximate budgeted maximization is available, the solution is \((\varepsilon,\delta)\)-optimal and yields asymptotic regret within a factor \(1/\varepsilon\) of the Graves–Lai lower bound [2102.07254].

## 6. Broader variants and later tractable instances

The computational role of the Graves–Lai problem extends beyond combinatorial semi-bandits. In multimodal bandits on a known tree \(G=(V,E)\), where the unknown mean vector is assumed to have at most \(m\) modes, the lower bound is again expressed through a Graves–Lai program:
\[
\min_{\boldsymbol{\eta}\ge 0}\quad \boldsymbol{\eta}^\top \boldsymbol{\Delta}
\quad\text{s.t.}\quad
\inf_{\boldsymbol{\lambda}\in\mathcal{B}(m,\boldsymbol{\mu})} \boldsymbol{\eta}^\top d(\boldsymbol{\mu},\boldsymbol{\lambda}) \ge 1,
\]
with
\[
\mathcal{B}(m,\boldsymbol{\mu}) = \left\{ \boldsymbol{\lambda}\in\mathcal{F}_{\le m}: \lambda_{k^\star(\boldsymbol{\mu})}=\mu^\star,\; k^\star(\boldsymbol{\lambda})\ne k^\star(\boldsymbol{\mu}) \right\}.
\]
Here the difficulty is no longer an exponentially large action family but the set \(\mathcal{B}(m,\boldsymbol{\mu})\), which for \(m>1\) is highly nonconvex and disconnected [2510.25811].

That work proves that representing \(\mathcal{B}(m,\boldsymbol{\mu})\) as a union of convex sets requires exponentially many components in \(m\):
\[
\mathcal{U}(K,m)\ge \frac{((\deg(G)-1)m)!}{(\deg(G)m)!} \bigl(K-(\deg(G)+1)m\bigr)^m.
\]
It then introduces structural reductions, including decomposition by the wrong optimal arm, a compact reformulation for non-neighbor arms, and a location-of-modes lemma stating that for the relevant optimizer \(\boldsymbol{\lambda}^\star\),
\[
\mathcal{M}(\boldsymbol{\lambda}^\star)\subset \mathcal{M}(\boldsymbol{\mu})\cup\{k\}.
\]
This leads to subproblems indexed by \((k,k')\), discretization on a grid
\[
D(n,\boldsymbol{\mu}) = \left\{\mu_\star+\frac{i}{n}(\mu^\star-\mu_\star): i\in[n]\right\},
\]
and a dynamic program on the rooted tree \(G^k\) [2510.25811].

The dynamic program computes each \((k,k')\) subproblem in time and memory \(O(nK)\), giving total complexity \(O(K^2mn)\) and space \(O(Kn)\) per iteration of the outer loop, with an appendix improvement reducing the discrete subproblem to \(O(Kn)\). The outer Graves–Lai optimization is then handled by a penalized projected subgradient method over \(\boldsymbol{\eta}\), producing a feasible approximate solution after \(t\) iterations with total time
\[
O(K^2mnt)
\]
and space
\[
O(Knt).
\]
The paper states that this is the first known computationally tractable algorithm for computing the solution to the Graves–Lai optimization problem in that multimodal setting [2510.25811].

A plausible implication is that the Graves–Lai optimization problem is a unifying object across disparate structured bandit classes, while the algorithmic route to tractability depends sharply on the geometry of the alternative set: exponential action spaces in combinatorial semi-bandits can be handled by convex reduction and sparse decomposition, whereas disconnected structural constraints in multimodal bandits require decomposition, discretization, and tree dynamic programming. Both lines of work, however, preserve the same central interpretation: the Graves–Lai solution is the asymptotically optimal exploration profile.

Source: https://www.emergentmind.com/topics/graves-lai-optimization-problem