Optimal Kelley-like Methods for Nonsmooth Optimization

• The paper proposes an optimal Kelley-like method that integrates affine cutting-plane models with trust-region regularization to achieve minimax optimal convergence rates.
• It combines dynamic step adaptation and explicit history dependence to balance convergence speed with computational efficiency.
• The method delivers subgame perfect guarantees, ensuring that each iteration attains the lowest possible objective gap given the problem’s parameters.

Optimal Kelley-like methods for nonsmooth convex optimization form a class of first-order algorithms that leverage affine (cutting-plane) models, trust-region or prox-level stabilization, and explicit history dependence to achieve information-theoretic optimal rates in the minimization of convex, Lipschitz-continuous functions subject to constraints or within Euclidean balls. Unlike classical subgradient or Kelley’s original cutting-plane methods, these approaches incorporate regularization and dynamic adaptation that yield improved worst-case and instance-dependent performance while requiring minimal problem parameter knowledge.

1. Problem Setting and Performance Metrics

Optimal Kelley-like methods operate on the convex, nonsmooth minimization problem

$\min_{x\in\mathbb{R}^d} f(x)$

where $f: \mathbb{R}^d \to \mathbb{R}$ is convex and $M$-Lipschitz,

$$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$

and access to $f$ is via a first-order (subgradient) oracle returning $(f(x), g)$ for $g \in \partial f(x)$. The initial point $x_0$ is such that $\|x_0 - x_\star\| \le R$ for some minimizer $x_\star$.

The primary performance metric is the worst-case final objective gap after $f: \mathbb{R}^d \to \mathbb{R}$0 iterations,

$f: \mathbb{R}^d \to \mathbb{R}$1

where $f: \mathbb{R}^d \to \mathbb{R}$2 is the set of convex, $f: \mathbb{R}^d \to \mathbb{R}$3-Lipschitz functions with a minimizer within radius $f: \mathbb{R}^d \to \mathbb{R}$4 of $f: \mathbb{R}^d \to \mathbb{R}$5. Minimax optimality seeks methods with lowest possible worst-case gap; subgame perfect optimality demands that this guarantee holds adaptively for every realized subproblem after any sequence of oracle answers (Grimmer et al., 17 Nov 2025).

2. Methodologies: Kelley-like and Bundle-level Approaches

Kelley’s Original Method

Kelley’s 1960 method incrementally builds a bundle of affine minorants (cuts) of $f: \mathbb{R}^d \to \mathbb{R}$6. Each iteration solves

$f: \mathbb{R}^d \to \mathbb{R}$7

However, this procedure can exhibit arbitrarily slow convergence (oscillations) and is not minimax optimal (Drori et al., 2014, Grimmer et al., 17 Nov 2025).

Bundle-level and Accelerated Prox-Level Methods

Bundle-level methods introduce regularization via trust-region constraints or prox-terms. The accelerated bundle-level (ABL) and accelerated prox-level (APL) methods (Lan, 2013) update via

$f: \mathbb{R}^d \to \mathbb{R}$8

where $f: \mathbb{R}^d \to \mathbb{R}$9 is the maximal cutting-plane model and $M$0 is a “level” determined by historic lower and upper bounds.

The ABL method achieves an $M$1 rate for approximation error, matching the Nemirovski–Yudin lower bound for black-box nonsmooth convex minimization, without inputting the Lipschitz constant $M$2 or feasible set diameter $M$3. The APL variant restricts memory by keeping only the last $M$4 cuts, preserving optimality and computational tractability.

Optimal Variant: Kelley-like Method (KLM)

The optimal Kelley-like method (KLM) (Drori et al., 2014, Grimmer et al., 17 Nov 2025) runs as follows:

• Builds a polyhedral cutting-plane model with a quadratic/trust-region term, solving at each (standard) step:

$M$5

subject to

$M$6

• Allows “easy” subgradient steps between standard steps: $M$7, with step size $M$8.
• After $M$9 steps, produces a carefully constructed convex combination of iterates as the output.

At every iteration, it explicitly incorporates the actual bundle, minimum observed function value, Lipschitz bound, and the remaining step budget.

3. Optimality and Convergence Rates

The methods above are optimal in the following precise senses:

• ABL/APL (Bundle-level, Parameter-Free Optimality): Achieve $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$0, requiring no a priori knowledge of $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$1 or $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$2, and operate with fully dynamic bounds, matching black-box lower bounds (Lan, 2013).
• KLM (Minimax and Subgame Perfect Optimality):

$$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$3

This constant matches the exact minimax rate for this class as established by Nemirovski–Yudin and Nesterov (Drori et al., 2014, Grimmer et al., 17 Nov 2025).

• Subgame Perfection: KLM achieves subgame perfect guarantees: after any observed history, it attains the tightest possible minimax bound for the residual problem, adapting optimally to all information revealed so far. In practical terms, for any instance and any observed sequence of oracle answers, the KLM bound $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$4 never exceeds what any competitor could prove for that subinstance (Grimmer et al., 17 Nov 2025).

The following table summarizes complexity and optimality guarantees:

Method	Complexity Bound	Needs $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$5, $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$6?	Subgame Perfect
Kelly (1960)	Arbitrarily slow	No	No
ABL/APL	$$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$7	No	No*
KLM	$$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$8	Yes (explicit)	Yes

*ABL/APL achieve minimax optimality but are not subgame perfect as defined in (Grimmer et al., 17 Nov 2025).

4. Algorithmic Structure and Implementation Details

The core steps for these methods are:

• Cutting-plane modeling. Maintain a model of the feasible set comprising all historic affine minorants, i.e., cut planes.
• Regularization. Introduce a trust-region (quadratic penalty) or "level set" to avoid oscillations and ensure sufficient progress.
• Adaptive step selection. In ABL/APL, acceleration is via multi-step extrapolation; in KLM, step size and trust-region size are chosen by solving a history-aware convex program (second-order cone program).
• Upper and lower bound management. All methods track both best-observed and model-predicted function values for robust stopping.

The KLM algorithm requires the number of steps $$|f(x) - f(y)| \le M \|x - y\| \quad \forall x, y \in \mathbb{R}^d,$$9 in advance to set the optimal step size $f$0. Each standard step involves solving a convex quadratic or second-order cone subproblem of size $f$1 (where $f$2 is the ambient dimension). Each easy step is a direct subgradient move. APL implements memory restriction by keeping only $f$3 past planes, leading to smaller subproblems with retained convergence rates (Lan, 2013, Drori et al., 2014).

Variants allow for known lower bounds, $f$4-subgradients, and cases with composite structure.

5. Theoretical Foundations: Lower Bounds, Interpolation, and Zero-Chain Property

Any black-box (first-order oracle) method for nonsmooth convex minimization requires at least $f$5 function–subgradient calls to achieve $f$6 accuracy (Lan, 2013). For the radius-bounded, Lipschitz-constrained setting, the minimax absolute error after $f$7 steps is at least $f$8, and KLM attains this exactly.

The key technical device is an interpolation lemma (Grimmer et al., 17 Nov 2025, Drori et al., 2014): for any bundle $f$9 with

$(f(x), g)$0

there exists a convex $(f(x), g)$1-Lipschitz function matching all data. This underlies the history-aware SOCP (second-order cone program) used for KLM’s next-step planning.

The zero-chain property ensures that for subgradient-span methods, an adversarial oracle can always force the new subgradient $(f(x), g)$2 to be orthogonal to the prior span, restricting information gain per iteration. As a result, any such method cannot improve on the prescribed rate; KLM is constructed to saturate this lower bound at each (sub)problem (Grimmer et al., 17 Nov 2025).

6. Adaptivity, Beyond-Worst-Case Behavior, and Practical Aspects

KLM’s explicit history dependence (SOCP is recalculated at every iteration using actual observed cuts and gradients) allows it to adapt to the revealed subclass of functions. In cases where the observed sequence indicates better conditioning or lower effective Lipschitz constants, the worst-case bound $(f(x), g)$3 can decrease strictly faster than the uniform $(f(x), g)$4 rate. Thus, KLM provides a "beyond worst-case" guarantee while retaining global optimality (Grimmer et al., 17 Nov 2025).

In practice, the computational bottleneck is solving the low-dimensional convex subproblems; the cost is manageable for moderate $(f(x), g)$5 and $(f(x), g)$6. APL’s restricted memory variant is attractive for large-scale applications (Lan, 2013). Both the lower and upper bound management is fully explicit and does not require prior knowledge of $(f(x), g)$7 or $(f(x), g)$8 for bundle-level methods.

7. Comparison with Related Methodologies and Extensions

Optimal Kelley-like methods are strictly more effective and theoretically robust than classical Kelley or pure subgradient methods. Bundle-level approaches (ABL, APL) and KLM both use ◦ explicit model regularization (prox/level or trust region) and ◦ full history of cuts, but only KLM (and its subgame perfect proximal point analogue) deliver subgame perfect performance. Other bundle or level-type methods require additional parameter knowledge or do not enforce instance-wise tight guarantees at each history node (Drori et al., 2014, Grimmer et al., 17 Nov 2025).

Extensions to saddle-point problems, composite structures, and stochastic programming have been demonstrated for bundle-level and prox-level methods (Lan, 2013). KLM’s construct extends naturally to proximal oracles as well as to settings where ε-subgradients are permitted; the performance guarantee shifts by at most an additive ε when $(f(x), g)$9 is used (Drori et al., 2014).

• "Bundle-Level Type Methods Uniformly Optimal for Smooth and Nonsmooth Convex Optimization" (Lan, 2013)
• "An optimal variant of Kelley's cutting-plane method" (Drori et al., 2014)
• "Subgame Perfect Methods in Nonsmooth Convex Optimization" (Grimmer et al., 17 Nov 2025)