An Accelerated Proximal Bundle Method with Momentum

Abstract: Proximal bundle methods (PBM) are a powerful class of algorithms for convex optimization. Compared to gradient descent, PBM constructs more accurate surrogate models that incorporate gradients and function values from multiple past iterations, which leads to faster and more robust convergence. However, for smooth convex problems, PBM only achieves an O(1/k) convergence rate, which is inferior to the optimal O(1/k^2) rate. To bridge this gap, we propose an accelerated proximal bundle method (APBM) that integrates Nesterov's momentum into PBM. We prove that under standard assumptions, APBM achieves the optimal O(1/k^2) convergence rate. Numerical experiments demonstrate the effectiveness of the proposed APBM.

• The paper introduces APBM which achieves an optimal O(1/k^2) convergence rate for smooth convex optimization by integrating proximal bundle models with Nesterov acceleration.
• The method eliminates the need for inner loops, enhancing computational efficiency especially in high-dimensional settings.
• Experimental results show APBM converges at least three times faster than classical proximal bundle methods while maintaining robust step-size performance.

An Accelerated Proximal Bundle Method with Momentum

Introduction

The paper "An Accelerated Proximal Bundle Method with Momentum" (2604.00569) presents a novel algorithmic framework for unconstrained convex and smooth optimization, integrating the robust modeling of Proximal Bundle Methods (PBM) with the optimal convergence rates of Nesterov-type acceleration. The proposed method, denoted as Accelerated Proximal Bundle Method (APBM), is motivated by the observed limitations of classical PBM—namely, their $O(1/k)$ rate for smooth objectives—and aims to achieve the optimal $O(1/k^2)$ rate without incurring the algorithmic complexity associated with nested or inner loops seen in prior work.

Algorithmic Framework

APBM operates on general convex and differentiable objectives $f:\mathbb{R}^n \to \mathbb{R}$, updating its iterates using a surrogate model $\hat{f}^k$ at each iteration. The algorithm synthesizes two powerful components:

1. Proximal Bundle Model: Constructs surrogate minorants of $f$ using historical gradient and value information over a bundle of points, yielding an update by minimizing $\hat{f}^k(x) + \frac{1}{2\gamma}\|x - y^k\|^2$. Compared to the proximal-linear (first-order) model in Gradient Descent (GD), this surrogate ensures tighter bounds and improved step-size robustness.
2. Nesterov Acceleration: Retains the classical coefficient and extrapolation sequence, extrapolating iterates via $$y^{k+1} = x^k + \frac{t_k - 1}{t_{k+1}}(x^k - x^{k-1})$$, where $t_k$ controls the momentum. The minimization is performed at the extrapolated point as in the Fast Iterative Shrinkage-Thresholding Algorithm (FISTA).

A critical distinction is that, in contrast to related accelerated bundle schemes, APBM eliminates the need for any internal loop, resulting in a more parsimonious and efficient structure.

Surrogate Models in APBM

The selection of the surrogate $\hat{f}^k$ is central to achieving both theoretical guarantees and computational efficiency. The paper proposes and justifies several forms:

• Polyak model: Minorizes $f$ by the maximum of its tangent plane at the extrapolated point and a global lower bound.
• Cutting-Plane model: Aggregates supporting hyperplanes constructed at a subset of previous iterates.
• Polyak Cutting-Plane model: Enhances the model further by incorporating both approaches.
• Two-Cut model: Iteratively combines previous minorants with new tangents to promote accuracy.

Each is shown to satisfy convexity and minorization assumptions, enabling the required theoretical analysis.

Figure 1: Three representative surrogate functions showcasing the Polyak, cutting-plane, and Polyak cutting-plane models; each incorporates additional historical or lower-bounding information to construct a tighter minorant of the objective.

Computation of the Proximal Bundle Step

With piecewise-linear $O(1/k^2)$0, the APBM update reduces to a convex quadratic program (QP) that can be efficiently solved. For high-dimensional $O(1/k^2)$1 but modest bundle size $O(1/k^2)$2, the dual QP—posed over the simplex—admits fast projected-gradient methods with $O(1/k^2)$3 cost per projection, leveraging the low cardinality of the bundle to maintain efficiency. This is essential for high-dimensional learning problems and is a concrete advantage over methods with nested inner iterations.

Convergence Analysis

Under standard smoothness and convexity assumptions, the paper rigorously proves that APBM attains the optimal $O(1/k^2)$4 convergence rate for smooth convex optimization. The main theorem establishes that

$O(1/k^2)$5

where $O(1/k^2)$6 is an optimal solution, $O(1/k^2)$7 is the Lipschitz constant of $O(1/k^2)$8, and $O(1/k^2)$9 counts the iterations. Notably, this matches the lower complexity bound for first-order methods and is achieved without losing the robust step-size selection and fast practical performance of PBM.

Numerical Validation

Through experiments on large-scale least squares problems ($f:\mathbb{R}^n \to \mathbb{R}$0), APBM is empirically compared with GD, PBM, and AGD. The bundle size $f:\mathbb{R}^n \to \mathbb{R}$1 is used for PBM and APBM, while all methods are fine-tuned for optimal step-size. The results explicitly show:

• Superior practical speed: APBM achieves far lower objective residuals in fewer iterations, converging at least three times faster than PBM.
• Robustness: APBM maintains convergence for a much larger range of step-sizes $f:\mathbb{R}^n \to \mathbb{R}$2 relative to AGD, easing practical implementation.
• Benefit from restart: Incorporating a fixed-restart mechanism further improves convergence—especially for APBM—demonstrating that the method synergizes with modern acceleration strategies.

Figure 2: Demonstrates convergence rates of GD, PBM, AGD, and APBM ($f:\mathbb{R}^n \to \mathbb{R}$3) on a least squares instance, highlighting the faster decay rate and step-size robustness of APBM.

Figure 3: Incorporating a restart scheme further improves the convergence profile of APBM, delivering lower optimality residuals and retaining robust performance for larger step-sizes compared to AGD.

Theoretical and Practical Implications

The APBM framework establishes that integrating bundle models with Nesterov acceleration is both theoretically sound and computationally superior for smooth convex optimization. The absence of inner loops makes the method substantially more practical for high-dimensional problems, where solving a low-dimensional dual is significantly cheaper than primal approaches when the bundle size remains controlled. This design ensures that the algorithm's advantages persist in modern large-scale machine learning and control applications.

Further, APBM's improved step-size robustness, inherited from PBM, mitigates the tuning issues often faced by classical accelerated schemes, and the bundle-based surrogates enable extrapolation beyond strict smoothness, opening paths toward future generalizations to nonsmooth or composite settings.

Conclusion

This work provides a convergence-optimal proximal bundle framework augmented with momentum that closes the rate gap between PBM and accelerated gradient methods for smooth convex objectives. The derivation and analysis demonstrate that high-fidelity surrogate models, when coupled with Nesterov-style extrapolation, yield an algorithm that is both theoretically optimal and empirically robust. The potential for future extensions includes further improvements via adaptive restarts, application to composite and distributed optimization, and optimization in the presence of stochasticity or nonsmoothness.