Inexact Local Solvers in Optimization

Updated 4 July 2025

Inexact local solvers are algorithmic components that approximately solve local subproblems within an optimization framework, balancing computational cost and accuracy.
They employ controlled error tolerances and adaptive strategies to achieve sufficient improvement per iteration, making them ideal for large-scale, nonlinear, or distributed problems.
Their design focuses on maintaining global convergence through majorant conditions and error propagation control, ensuring efficient and reliable performance in complex systems.

Inexact local solvers are algorithmic components that compute approximate solutions to local subproblems within an overarching optimization or equation-solving framework, allowing for controlled error and lower computational cost compared to exact solves. They are especially relevant for large-scale nonlinear, constrained, or distributed problems where global precision at every substep is infeasible or unnecessary. Fundamentally, inexact local solvers provide sufficient improvement or stationarity per step, under conditions that ensure the convergence of the global algorithm to a solution or stationary point, potentially at optimal or near-optimal computational complexity.

1. Fundamental Principles and Motivations

Inexact local solvers are rooted in the recognition that demanding high-precision in all local computations (e.g., local linear or nonlinear subproblems, projections, or inner optimization loops) can be computationally prohibitive without proportional global benefit, especially for large or structured systems. Instead, if local subproblems are solved only approximately—subject to explicit error conditions that are rigorously tied to the progress of the global algorithm—similar or same global convergence properties can be achieved at much greater efficiency.

Motivations include:

Reducing per-iteration or per-update computational and memory cost.
Scalability for large-scale, high-dimensional, or distributed problems.
Enabling the use of fast heuristics, iterative, or matrix-free methods as local subproblem solvers.
Handling local subproblems where exact solves are intractable due to NP-hardness, domain complexity, or hardware limitation.

A key principle is to quantify and control inexactness (often as a relative or absolute residual, or via error bounds) so that the global convergence theory accommodates approximate local computations (1008.1916, 1810.00303, 1608.00413).

2. Convergence Theory Under Inexactness

Rigorous convergence guarantees for algorithms with inexact local solvers require the development of analytical frameworks that relate the error in each local step to global progress. Two common tools are:

Majorant Conditions: These generalize the classical Lipschitz or Hölder continuity conditions, replacing them with inequalities involving a scalar majorant function that governs the allowed growth of local nonlinearity or inexactness. This broadens applicability to cases where the derivatives are non-Lipschitz or analytic (1008.1916, 1309.4734, 1705.07684).
Residual and Error Tolerance Schemes: For Newton-type or Gauss-Newton-like methods, requiring that the step solves the linearized equation up to a relative or absolute residual (e.g., $\|F(x_k) + V_k(y_k - x_k)\| \leq \eta_k\|F(x_k)\|$ ), and tuning $\eta_k$ adaptively or guaranteeing it decays, ensures convergence at rates (linear, superlinear, quadratic) linked to the problem's regularity and the error schedule (1309.4734, 1810.11640).
Error Propagation: Inexactness in local subproblems, if properly bounded and summable, does not destroy local convergence; the rate depends on the decay of inexactness. For example, absolute summability of error sequences or decay as $\mathcal{O}(1/k^{1+\kappa})$ can guarantee convergence of sublinear or linear order for first-order methods (1608.00413, 1708.07010). Inexact projection criteria in Levenberg-Marquardt or Newton-like methods preserve local superlinear convergence under error bounds or semismoothness assumptions (1908.06118, 2105.01781).

3. Methodological Realizations

Inexact local solvers have been realized in diverse algorithmic frameworks:

Inexact Newton and Gauss-Newton Methods: At each iteration, the linear system is solved only approximately, with residuals controlled by relative tolerances. These methods have been analyzed under both majorant and error bound conditions, and extended to Riemannian manifolds (1008.1916, 1309.4734, 2303.10554).
Inexact Conditional Gradient Methods: Projection-free algorithms that replace exact projection with a (possibly inexact) Frank-Wolfe step, allowing for efficient handling of complex constraints (1705.07684).
Inexact Proximal-Gradient Algorithms: Especially in nonconvex regularized optimization, proximal or subgradient steps may be computed inexactly, as long as sufficient decrease and subgradient bound conditions are met (1708.07010).
Inexact Solvers in Distributed and Domain Decomposition Methods: Local subdomain problems in methods such as IETI-DP, FETI-type, or AFEM are solved with iterative methods (e.g., multigrid, fast diagonalization, or Krylov subspace solvers) instead of direct factorizations, enabling memory efficiency and scalability in high-degree, multi-patch discretizations (1705.04531, 2001.09626, 2206.08416).
Inexact Proximal Newton and Primal-Dual Methods: Composite convex and structured optimization problems leverage inexactness in both oracles (function, gradient, Hessian) and inner minimization, structured under global or local inexact oracle definitions (1808.02121).
Spectrahedral and General Feasibility Problems: Where projection onto the constraint set is expensive, feasible inexact projections (via conditional gradient or other approximations) facilitate rapid iterations while guaranteeing feasibility and convergence (1810.11640, 1908.06118, 2105.01781).

4. Practical Implications and Limitations

The deployment of inexact local solvers offers broad practical benefits:

Computational efficiency: Drastic reductions in local computation, enabling real-time or large-scale implementation.
Memory savings: Avoidance of dense factorization or storage in large-scale finite element, isogeometric, or domain decomposition settings.
Flexibility and scalability: Enabling parallel and distributed implementations, particularly in distributed optimization and multi-patch simulations (1608.00413, 2001.09626, 1705.04531).
Robustness: The flexibility to accommodate various solver accuracies without loss of global convergence, crucial for computational platforms with variable resources or for problems where exact local solutions are infeasible.
Explicit error control: Providing mechanisms (e.g., adaptive tolerances, certification strategies for local solver iterations) to match accuracy requirements dynamically to the current state of the algorithm (1608.00413).

Limitations include:

The need to construct or estimate a majorant function in Gauss-Newton/Newton analyses, which for arbitrary nonlinear problems may require delicate mathematical work.
Most guarantees are local; global convergence may require additional strategies (trust region, globalization via line search, error bounds that hold globally) (1008.1916, 1908.06118).
In projection-based methods, the theoretical guarantees require that the inexact projections maintain feasibility and are controlled relative to the step size (1810.11640, 2105.01781).
Inexactness schedules that relax too quickly can degrade convergence; error decay rates must be matched to algorithm and problem structure for optimal performance (1608.00413, 1708.07010).

5. Notable Application Domains

Inexact local solvers have been successfully applied and analyzed in multiple domains, including:

Adaptive Finite Element Methods: Achieve linear-time complexity for nonlinear elliptic PDEs by solving only one full nonlinear solve and using single inexact Newton updates on refined meshes (1107.2143).
Isogeometric Analysis and Domain Decomposition: Enable high-order, multi-patch simulations by replacing local direct solves with fast diagonalization or multigrid, realizing scalability and memory savings for problems in engineering simulation and continuum mechanics (1705.04531, 2001.09626, 2206.08416).
Nonlinear Optimization and Machine Learning: Improve large-scale Newton-type methods in high-dimensional empirical risk minimization, mixture models, and regularized learning, allowing for indefinite or singular Hessians (1810.00303).
Constrained Nonsmooth and Manifold Equations: Facilitate new classes of algorithms for structured feasibility problems and variational inequalities, including on Riemannian manifolds (2303.10554).
Sparse or Nonconvex Regularized Problems: Allow local or block inexactness in sparse optimization (e.g., $\ell_p$ -regularization for $0 < p < 1$) while proving sharp local rates (1708.07010).

6. Convergence Rate, Certification, and Error Schedules

A key technical focus is establishing clear, practically checkable terms for the permissible inexactness per local solve. Typical results relate:

Linear convergence to absolutely summable error or inexactness decaying at $\mathcal{O}(1/k^{1+\kappa})$ per iteration (1608.00413, 1708.07010).
Superlinear or quadratic convergence as inexactness (or its square root) decreases proportionally to the current residual or step size (1309.4734, 1810.11640).
Explicit certification of the number of local solver steps required, particularly for quadratic objectives in distributed optimization (1608.00413).

Adaptive or certified error schedules both safeguard convergence and match computational resources to the most expensive steps in practice.

7. Outlook and Research Directions

Substantial research opportunities remain, including:

Automatic or data-driven majorant construction for nonlinear least squares and Newton-type local analysis (1008.1916).
Designing globally convergent schemes that flexibly integrate inexact local solvers, especially via nonmonotone or composite error control strategies (1908.06118).
Parallel and distributed algorithm design that fully exploits the flexibility of inexact local computation, particularly in networked or resource-constrained environments (1608.00413).
Extending the frameworks to broader problem classes, including nonsmooth, nonconvex, or semi-infinite optimization, and high-dimensional manifold-valued settings (1810.11640, 2303.10554).

Application Domain	Inexact Local Solver Strategy	Convergence/Complexity Characteristic
Nonlinear least squares	Gauss-Newton with approximate linear solve	Local, explicit rate under majorant cond.
Distributed optimization, MPC	AMA/FAMA with inexact local updates	Sublinear/linear, error-certified updates
Sparse/regularized signal recovery	Inexact proximal-gradient/descent method	Local linear, quadratic under 2nd order CG
Isogeometric/domain decomposition	Multigrid, fast diagonalization, FD	Poly-logarithmic (in h), robust in p, H
Constrained smooth/nonsmooth eqns	Newton/LM with inexact projections	Linear–quadratic; preserves feasibility

In summary, inexact local solvers have become indispensable tools in numerical optimization, PDE simulation, and scientific computing, providing a principled approach to balance local computational effort with global convergence and accuracy, especially as scale, structure, and resource constraints increasingly dominate practical considerations.