The Method of Ellipcenters for Strongly Convex Functions

Abstract: The Method of Ellipcenters (ME), introduced in~\cite{ME2025} for strongly convex quadratic minimization, uses two gradient evaluations per iteration: one at the current iterate and one at a companion point on the same level set. We extend ME to the broader class of strongly convex functions with Lipschitz continuous gradient, and prove that ME matches the convergence rate of gradient descent with exact line search on this class. When the two gradient directions are linearly independent, a midpoint argument exploiting the level-set symmetry yields a further per-step improvement, which is global when the angle between the two gradients is uniformly bounded away from zero. ME also converges in at most two steps in dimension two. Numerical experiments on regularized logistic regression confirm the theoretical predictions.

• The paper presents an extension of the ME method from quadratic to general L-smooth, μ-strongly convex functions using a two-gradient evaluation strategy.
• It leverages a two-dimensional subproblem to exploit the local geometry of level sets, yielding improved per-iteration descent compared to classical gradient descent.
• Empirical evaluations confirm that ME achieves sharper convergence rates and reduced gradient evaluations, especially when gradients are linearly independent.

The Method of Ellipcenters for Strongly Convex Functions

Extension of the Method of Ellipcenters to General Strongly Convex Settings

The Method of Ellipcenters (ME), introduced for unconstrained minimization of strongly convex quadratic functions, is extended to encompass $L$-smooth, $\mu$-strongly convex functions. The method iteratively computes the minimizer of $f$ over a plane spanned by two gradients evaluated at special companion points lying on the same level set. For general $f$, this involves two gradient evaluations per iteration: at the current iterate $x^k$ and a companion point $y^k = x^k - t_k \nabla f(x^k)$, where $t_k > 0$ is chosen so that $f(y^k) = f(x^k)$ is maintained, guaranteeing that $y^k$ always exists and is unique.

The generated affine plane $$\Pi_k = \{ x^k + \alpha \nabla f(x^k) + \beta \nabla f(y^k): (\alpha, \beta) \in \mathbb{R}^2 \}$$ is the search domain for each iterate. If the gradients $\mu$0 and $\mu$1 are linearly dependent, ME reverts to a segment search, otherwise one solves a two-dimensional subproblem for the minimum over $\mu$2. The method is analogous to the ellipsoidal structure of quadratic levels sets, but for general $\mu$3 relies on the smooth, strictly convex shape of level sets and their local geometry.

Theoretical Properties and Convergence Analysis

Existence, Uniqueness, and Computation

Rigorous arguments are presented demonstrating that the companion point $\mu$4 exists and is unique for any differentiable, strongly convex $\mu$5 at every non-stationary $\mu$6, with at most two points on a line sharing the same objective value. For general $\mu$7, a bisection search provides the required step size $\mu$8. For quadratic $\mu$9, a closed-form is available.

Fundamental Descent Property

Central to ME’s improved convergence is the exploitation of the Baillon--Haddad inequality:

$f$0

where $f$1 is the Lipschitz constant of the gradient. The two-dimensional nature of the update ensures that $f$2 is orthogonal to both $f$3 and $f$4 in the linearly independent (LI) case, thereby annihilating cross terms and enforcing sharper descent than gradient descent. This structure is nontrivial and does not occur in vanilla gradient methods.

Linear Convergence and Per-step Rate Improvements

ME is shown to achieve the classical gradient descent with exact line search rate, $f$5, for $f$6-smooth $f$7-strongly convex $f$8, where $f$9 is the condition number:

$f$0

In the LI case, a further per-step improvement is rigorously established. The method’s per-step contraction factor is

$f$1

where $f$2 is the angle between $f$3 and $f$4. The strict improvement $f$5 is achieved whenever the two gradients are not parallel, and a uniform gap is present if the directional angle is bounded away from zero for all $f$6. Notably, ME always dominates the function value achieved by gradient descent with exact line search, step-for-step.

If both gradients are parallel, the method reverts to an exact linesearch, matching the classical rate. In dimension two, ME achieves the solution in at most two iterations due to the planar span.

Empirical Evaluation

Extensive numerical experiments are provided to validate the theoretical convergence rates on non-quadratic, regularized logistic loss minimization tasks. In these experiments, ME outperforms classical gradient descent, gradient descent with exact line search, and Nesterov's accelerated gradient (Fast-GD) in terms of gradient evaluations, particularly at moderate condition numbers.

Terminal suboptimality with ME matches or surpasses all competitors and the per-iteration empirical contraction closely tracks, and often exceeds, the theoretical worst-case bounds. Importantly, the linearly independent condition for the two gradients is always satisfied in practical experiments, ensuring the improved rate guarantee applies uniformly.

Convergence behavior is illustrated by plotting the logarithm of suboptimality versus gradient evaluations.

Figure 1: Convergence on regularized logistic regression showing ME requires fewer gradient evaluations than GD-exact, closely matching the theoretical per-step improvement.

Practical and Theoretical Implications

The extension of ME broadens its applicability to any differentiable, strongly convex, and gradient-Lipschitz objective. The method inherits from both conjugate gradient and two-step acceleration principles without recourse to higher-order information or acceleration coefficients. The per-iteration work is more involved than gradient descent but requires only two gradient evaluations and a bisection.

Strong empirical and theoretical superiority in terms of function value contraction is demonstrated stepwise, exceeding the performance of methods based upon exact linesearch when gradient independence holds. In dimension two, this yields trivialization of the problem, solving in two steps for all $f$7-smooth, $f$8-strongly convex $f$9.

Open Problems and Future Directions

The most prominent theoretical gap relates to the convergence rate in iterate distance for general $x^k$0, which still lags the optimal rate achieved for quadratics. Development of geometric Lyapunov functions tailored to the evolving plane geometry may close this gap.

Other open directions include efficient algorithms for the two-dimensional plane minimization subproblem (potentially using curvature information or surrogate quadratic fits); extension to composite and nonsmooth optimization by defining analogs of the companion point; and generalization to block or stochastic variants by selecting planes using other directional information.

Conclusion

The paper rigorously extends the Method of Ellipcenters from strongly convex quadratics to general $x^k$1-smooth, $x^k$2-strong convex functions, establishing sharp per-iteration descent bounds, demonstrating functional value contraction at or above classical exact-linesearch rates, and guaranteeing at most two iterations to solution in dimension two. Numerical results on nontrivial objectives robustly confirm the theoretical predictions, and the method's stepwise dominance invites further study into higher-order and nonsmooth variants, as well as enhanced initialization and solution techniques for the internal step.

Reference: "The Method of Ellipcenters for Strongly Convex Functions" (2604.21132)