Linear Convergence and Error Bounds for Optimization Without Strong Convexity (2510.27540v1)

Abstract: Many optimization algorithms$\unicode{x2013}$including gradient descent, proximal methods, and operator splitting techniques$\unicode{x2013}$can be formulated as fixed-point iterations (FPI) of continuous operators. When these operators are averaged, convergence to a fixed point is guaranteed when one exists, but the convergence is generally sublinear. Recent results establish linear convergence of FPI for averaged operators under certain conditions. However, such conditions do not apply to common classes of operators, such as those arising in piecewise linear and quadratic optimization problems. In this work, we prove that a local error-bound condition is both necessary and sufficient for the linear convergence of FPI applied to averaged operators. We provide explicit bounds on the convergence rate and show how these relate to the constants in the error-bound condition. Our main result demonstrates that piecewise linear operators satisfy local error bounds, ensuring linear convergence of the associated optimization algorithms. This leads to a general and practical framework for analyzing convergence behavior in algorithms such as ADMM and Douglas-Rachford in the absence of strong convexity. In particular, we obtain convergence rates that are independent of problem data for linear optimization, and depend only on the condition number of the objective for quadratic optimization.

• The paper presents local error-bound conditions as necessary and sufficient for linear convergence of fixed-point iterations in non-strongly convex settings.
• It applies the framework to PLQ optimization problems, such as ADMM and Douglas-Rachford, demonstrating that convergence rates depend on quadratic term conditioning rather than constraints.
• The study provides explicit convergence rate bounds that inform parameter tuning and scaling strategies for robust implementation of iterative algorithms.

Linear Convergence and Error Bounds for Optimization Without Strong Convexity

Introduction

The paper "Linear Convergence and Error Bounds for Optimization Without Strong Convexity" explores the conditions under which fixed-point iterations (FPI) for averaged operators exhibit linear convergence. The importance of this investigation lies in extending linear convergence guarantees to broader classes of optimization problems that do not satisfy strong convexity, a common assumption in optimization theory. Specifically, the authors focus on establishing a necessary and sufficient condition for such convergence using local error-bound conditions for piecewise linear and quadratic (PLQ) optimization problems.

Main Contributions

The core contribution is the identification of local error-bound conditions as both necessary and sufficient for linear convergence of FPI applied to averaged operators. This is a significant extension of previous results that required either global error bounds or strong convexity assumptions, which limited their applicability. The paper demonstrates the following key aspects:

1. Error-Bound Conditions: The authors formalize that a local error-bound condition is critical for linear convergence. They demonstrate that for a fixed-point operator $F$, linear convergence is intrinsically linked to satisfying a condition of the form:

$\text{dist}(x, _F) \leq K_F \cdot \|F(x) - x\|,$

where $_F$ denotes the set of fixed points of $F$. This relationship provides a practical criterion for assessing convergence speed in various iterative algorithms.

1. Application to Piecewise Linear Operators: The paper provides a framework for applying these theoretical findings to PLQ optimization algorithms such as ADMM and Douglas-Rachford. It is shown that PLQ problems satisfy the proposed error-bound conditions, thus ensuring linear convergence without strong convexity.
2. Bound Calculations for Specific Algorithms: For linear and quadratic optimization problems, the paper calculates explicit bounds on the convergence rate and discusses conditions under which these bounds hold. Notably, it is shown that these convergence rates depend on the conditioning of the quadratic terms rather than on the constraint systems, offering insights into the stability and efficiency of scaled algorithms.

Implementation Considerations

The practical ramifications of these findings are significant for implementing optimization algorithms in scenarios where strong convexity cannot be assumed:

• PLQ Optimization in Software: When implementing PLQ optimization algorithms, ensuring that operators are piecewise linear facilitates the application of the paper's results. This may involve designing the algorithm components, such as gradient and proximal operations, to maintain the piecewise linearity.
• Error-Bound Condition Verification: In practice, verifying local error-bound conditions as outlined can guide tuning of parameters such as step sizes or regularization parameters to ensure linear convergence. This verification can typically be assessed by observing the reduction in residuals during the iteration process.
• ADMM and DR Algorithm Scaling: The insights regarding the independence of linear convergence rates from constraint conditioning suggest that practitioners can scale problems effectively, optimizing the quadratic terms to improve convergence without negatively impacting the solution's quality or the convergence speed due to the constraints.

Conclusion

The results presented in the paper provide a robust theoretical foundation for achieving linear convergence in non-strongly convex settings through local error-bound conditions. This opens new avenues for efficiently implementing and extending optimization algorithms to broader classes of practical problems where previous theoretical limitations might have hindered practical applications. The work stands as a pivotal step in bridging theoretical advances with real-world optimization challenges in fields such as machine learning, operations research, and beyond.