---
title: Feasibility-Preserving Newton Algorithms
url: https://www.emergentmind.com/topics/feasibility-preserving-newton-type-algorithms
type: topic
---

# Feasibility-Preserving Newton Algorithms

Feasibility-preserving Newton-type algorithms are iterative numerical methods that leverage Newton’s approach to optimization and nonlinear equation solving while maintaining strict feasibility with respect to problem constraints. These methods incorporate projection-free techniques, self-concordant barrier regularization, adaptive Hessian inversion strategies, and inexact subproblem solutions. They are foundational in convex and nonconvex constrained optimization, online convex optimization (OCO), inverse optimization, and nonlinear PDE analysis.

## 1. Principles of Feasibility Preservation

Feasibility preservation requires that each iterate generated by the algorithm remains in the feasible region defined by the problem’s constraints. Classical Newton or quasi-Newton methods do not guarantee this property, necessitating either projection steps (which may be computationally intensive) or constraint-aware mechanisms. Feasibility-preserving Newton-type algorithms circumvent explicit projections by:

- Employing barrier functions that diverge at the boundary of the feasible set, thus restricting iterates to the interior by construction.
- Structuring Newton or quasi-Newton steps so that feasibility is retained, either via functional analytic estimates, truncation, or adaptive update strategies.
- Utilizing operator splitting, active-set partitioning, or tailored line searches coupled with projections only onto simple sets (such as the nonnegative orthant) [2306.11121, 2405.12401, 2005.07838].

## 2. Self-Concordant Barriers and Projection-Free Newton Methods

In online convex optimization over a compact convex set $\mathcal{K}\subset\mathbb{R}^d$, projection-free Newton-type methods use a twice-differentiable, $M_\phi$-self-concordant barrier function $\phi:\operatorname{int}\mathcal{K}\to\mathbb{R}$ with parameter $\nu$ as a regularizer. Such barriers, satisfying
- $\phi(x)\to+\infty$ as $x\to\partial \mathcal{K}$,
- third derivative control
  $$
  |\nabla^3\phi(x)[u,u,u]| \le 2 M_\phi \|u\|_{\nabla^2\phi(x)}^3,
  $$
- and $\nabla\phi(x)^\top[\nabla^2\phi(x)]^{-1}\nabla\phi(x) \le \nu$,
induce spectral stability on the Hessian and control the Newton decrement.

For composite objectives $\Phi_t(x)=\phi(x)+\sum_{s=1}^{t-1} g_s^\top x$, the Newton-step becomes
$$
x_{t+1} = x_t - [\nabla^2\phi(x_t)]^{-1} (\nabla\phi(x_t)+g_t),
$$
which remains in $\mathcal{K}$ due to the barrier’s behavior. No explicit projections onto $\mathcal{K}$ are needed [2306.11121].

Self-concordance ensures that for small moves in the local norm, Hessians remain spectrally similar, which is critical to enabling amortization of the matrix-inversion cost (see Section 4).

## 3. Inexact Newton Methods: Feasibility and Convergence

Feasibility-preserving inexact Newton methods extend the classical Newton iteration by allowing additive errors $r_k$ in each step,
$$
x_{k+1} = x_k - [Df(x_k)]^{-1}f(x_k) + r_k,
$$
with $\|r_k\|$ controlled. Under a set of minor hypotheses—specifically, invertibility of $Df(x_0)$, local Lipschitz continuity of the Jacobian, and a bound on the error term $d<1/K$—one can guarantee (semi-)local feasibility: all iterates remain within a prescribed ball around $x_0$, and $Df(x_k)$ remains invertible. A majorant equation $g_d(t)$ determines the maximal radius of this ball and contraction guarantees; quadratic convergence is achieved if $\|r_k\|=O(\|x^*-x_k\|^2)$, while linear convergence is retained for merely vanishing $\|r_k\|$ [2005.07838].

This approach undergirds computer-assisted existence proofs for nonlinear boundary value problems, as the machinery provides both rigorous enclosures on iterates and feasibility within the function space defined by the constraints.

## 4. Hessian-Inverse Reuse and Amortized Complexity

A significant computational cost in Newton-type methods is the formation and inversion of the Hessian matrix $[\nabla^2\phi(x)]$. In high-dimensional settings, inverting the Hessian at every iteration is often prohibitive ($O(d^3)$ per inversion). Feasibility-preserving Newton-type OCO algorithms exploit the spectral stability guaranteed by the self-concordance of $\phi$:

- Define a “landmark” point $u$ and reuse $H_u^{-1}$ for all iterates $x$ close enough according to the local Hessian norm, $\|x-u\|_{H_u}<\delta$ for $\delta=O(1/M_\phi)$.
- When movement exceeds this threshold, recompute $H_u^{-1}$ at the new landmark.
- The total number of full inversions over $T$ rounds is bounded by $O(M_\phi T\eta)$, which is $o(T)$ for suitable step-sizes $\eta$ [2306.11121].

For large-scale optimization, this amortization is critical, reducing effective per-iteration complexity to that of a gradient step plus rare matrix inversions.

## 5. Adaptive and Inexact Newton Subsolvers

In feasibility-preserving Newton-type frameworks for nonnegativity-constrained or more general set-constrained optimization (e.g., nonnegative least squares or nonnegative matrix factorization), the Hessian block corresponding to strictly positive variables is generally indefinite and possibly ill-conditioned.

Methods utilize Krylov-subspace solvers such as MINRES to find Newton directions inexactly:
- For indices in the inactive set $I_k$ (where $x_k^i>\delta_k$), solve $H_k^I s^I = -g_k^I$ via MINRES up to a residual norm or detect nonpositive curvature via $r^\top H_k^I r \le 0$, allowing rapid escape from saddle regions.
- For active coordinates, employ scaled projected gradient updates.

Projection onto nonnegative orthant is trivial, enforcing feasibility exactly even under inexact directions. Two-metric Armijo-type line search ensures decrease in the objective function [2405.12401].

This approach readily extends to more general bound-constrained or simple set-constrained optima by choosing the projection $P$ accordingly.

## 6. Newton-Type Algorithms for Discrete and Inverse Optimization

In inverse optimization with combinatorial feasible sets, Newton-type algorithms can be constructed to iteratively adjust cost functions so as to make a given candidate solution $F^*$ optimal, while minimizing deviation according to measures such as weighted span. For the unit-weight case, the feasible deviation vector has a special form and Newton steps update the deviation via explicit combinatorial calculations involving the current minimal violator $F_i$.

A line search ensures that feasibility with respect to box constraints $\ell \le p \le u$ is preserved at every step. The overall number of Newton steps is $O(n^2)$, each requiring a single call to an $\mathcal{F}$-oracle, yielding a strongly polynomial-time algorithm. For general rational weights, the complexity becomes pseudo-polynomial in the worst case, and the existence of a general combinatorial, strongly polynomial scheme remains open [2302.13414].

## 7. Convergence Guarantees and Applications

Feasibility-preserving Newton-type algorithms yield theoretical guarantees competitive with the best known projection-based and first-order methods:
- In OCO with self-concordant barriers, regret is $O(RG\sqrt{T\log T})$ for losses with bounded gradients, comparable to Euclidean projection-based algorithms but without explicit projection operations [2306.11121].
- Inexact Newton methods for Banach-space nonlinear equations guarantee global feasibility and convergence under verifiable analytic conditions and are applicable to rigorous numerics in PDE and boundary value problems [2005.07838].
- Nonnegativity-constrained, nonconvex optimization with two-metric projection Newton-MR methods achieves optimality in $O(\varepsilon_g^{-3/2})$ iterations under Lipschitz-Hessian assumptions, showing robust and rapid convergence in large-scale empirical tests [2405.12401].
- Discrete inverse optimization Newton-type schemes are strongly polynomial for unit weights, readily extend to multi-cost functions, and provide a systematic prescription to maintain feasibility through coordinate-wise truncation [2302.13414].

Feasibility-preserving Newton-type algorithms thus combine advanced analytic control, adaptive computational routines, and compliance with constraints to address a broad class of problems across convex, nonconvex, continuous, and combinatorial optimization.

Source: https://www.emergentmind.com/topics/feasibility-preserving-newton-type-algorithms