cNKZ Projection: Nonlinear & Constrained Kaczmarz

Updated 4 March 2026

Constrained Nonlinear Kaczmarz (cNKZ) projection is an iterative method for solving high-dimensional nonlinear equations with convex constraints.
It generalizes the classical Kaczmarz method by integrating row-action updates with metric and Bregman projections and diverse sampling rules.
Empirical studies demonstrate that cNKZ variants achieve faster convergence and computational efficiency in inverse problems, robotics, and large-scale systems.

The constrained nonlinear Kaczmarz (cNKZ) projection encompasses a class of iterative projection methods for solving high-dimensional systems of nonlinear equations $F(x) = 0$ with side constraints $x \in C$ , where $C$ is a (possibly composite) closed convex set. These methods generalize the classical Kaczmarz projection from linear systems to nonlinear and constrained contexts, employing row-action updates derived from residuals and one-row Jacobians, and enforcing constraint feasibility via (metric or Bregman) projections. Modern cNKZ variants admit stochastic sampling, greedy selection, Bregman geometry, and manifold-based formulations, and are supported by sharp convergence theory under local tangential cone, convexity, or full-rank Jacobian assumptions (Zhang et al., 2022, Gower et al., 2023, Zeng et al., 2020, Agrawal et al., 2024, Leitao et al., 2020, Xiao et al., 2024).

1. Problem Setting and General Framework

The prototypical cNKZ problem is to compute $x^*\in C$ such that $F(x^*) = 0$ for a differentiable $F: D(F) \subset \mathbb{R}^n \to \mathbb{R}^m$ and a nonempty closed convex $C\subset\mathbb{R}^n$ . $C$ can be a Cartesian product of sets, an $\ell_1$ -ball, the probability simplex, or more complex polyhedral/semialgebraic constraints. The solution set $S=\{x\in C:F(x)=0\}$ is assumed nonempty.

Standard cNKZ methods operate via sequential projections onto the tangent hyperplanes of $F_i(x)=0$ (possibly linearized at the current iterate), interleaved with (metric or Bregman) projections onto $C$ or its components. Variants include cyclic, random, greedy, and sampling-based selection of the equation index; projection types depend on the geometry of $C$ and the chosen divergence.

Main update equation (Euclidean metric):

At iteration $k$ ,

$x_{k+1} = P_C\left( x_k - \frac{F_{i_k}(x_k)}{\|\nabla F_{i_k}(x_k)\|^2} \nabla F_{i_k}(x_k) \right)$

where $i_k$ is chosen by a sampling, residual maximization, or cyclic rule. For general Bregman divergences $D_\phi$ , the step is

$x_{k+1} = \operatorname{argmin}_{z\in C} D_\phi(x_k, z) \quad \text{s.t.} \; \langle \nabla F_{i_k}(x_k), z - x_k \rangle = -F_{i_k}(x_k)$

which is equivalently solved by a dual update involving the convex conjugate $\phi^*$ (Gower et al., 2023, Xiao et al., 2024).

2. Algorithmic Variants and Bregman Extensions

Metric vs. Bregman Projections

Metric (Euclidean) projection: Suitable when $C$ is simple (e.g., box, ball, simplex). In this case, projections onto $C$ are $O(n)$ and projection onto linearized equation hyperplanes admits closed form; standard Kaczmarz steps result.
Bregman projection: Used for structured $C$ (e.g., $\ell_1$ ball, sparsity, simplex, entropic constraints) or non-Euclidean geometry. Each update projects onto a tangent affine hyperplane wrt. the Bregman distance induced by a strictly convex distance-generating function $\phi$ . The resulting problem is a one-dimensional convex program in the dual space and is solved by a globalized Newton step per iteration (Gower et al., 2023, Xiao et al., 2024).

Algorithmic sampling/selection rules

Cyclic: Step through $i=1,\ldots,m$ in order.
Randomized: Sample $i_k$ uniformly or with nonuniform weights.
Greedy-residual: Choose $i_k = \operatorname{argmax}_i |F_i(x_k)|/\|\nabla F_i(x_k)\|$ to maximize expected decrease per step (Zeng et al., 2020, Xiao et al., 2024).
Greedy-distance: Maximize the step length to the projected constraint.

Accelerated and Motzkin Variants

Projected Sampling Kaczmarz–Motzkin (PSKM): At each iteration, sample a batch of row indices, select the one with maximal residual, perform one-row Kaczmarz update, and project onto a randomly chosen component constraint. The accelerated variant (APSKM) uses sequential projections plus extrapolation to further reduce distance to the feasible set (Zhang et al., 2022).

3. Analytical Assumptions and Convergence Theory

Key assumptions enabling convergence include:

Local tangential cone condition (TCC): Controls the local nonlinearity of each $F_i$ (Zhang et al., 2022, Leitao et al., 2020, Gower et al., 2023, Xiao et al., 2024).
Row boundedness below: Bounds $\|\nabla F_i(x)\| \geq \epsilon > 0$ (Zhang et al., 2022).
Full-rank Jacobian / convexity: Ensures global linear convergence; else, monotonic decrease (Zhang et al., 2022, Gower et al., 2023, Zeng et al., 2020).

Central convergence statements:

Linear convergence in expectation: Under TCC with constant $\eta<1/2$ and full-rank Jacobian, for both metric and Bregman cNKZ,

$\mathbb{E}[\|x_k-x^*\|^2] \leq (1-\rho) \mathbb{E}[\|x_{k-1}-x^*\|^2]$

for explicit rate $\rho$ dependent on $\sigma_{\min}(F'(x))$ , $\eta$ , $m$ , and norm structure (Zhang et al., 2022, Gower et al., 2023, Xiao et al., 2024).

Monotonic decrease: If each $F_i$ is convex nonnegative, the error norm strictly decreases per update until convergence (Zhang et al., 2022).
Bregman descent property: Each Bregman projection step strictly reduces $D_\phi(x_k,\hat x)$ up to a term proportional to squared residual (Gower et al., 2023, Xiao et al., 2024).
Regularization and noise tolerance: Landweber-Kaczmarz/cNKZ admits robust regularization in inverse problems with noisy data and converges weakly/strongly to the solution set, halting when residual falls below noise floor (Leitao et al., 2020).

4. Computational Structure and Practical Complexity

Each cNKZ iteration for a single constraint has per-step cost determined by:

Computing $F_i(x)$ and $\nabla F_i(x)$ : $O(n)$ or per-component cost $G$ .
(Bregman) projection step: $O(n)$ plus a one-dimensional Newton solve for Bregman variants.
Projection onto $C$ (if nontrivial): $O(n)$ or a prox operator.
APSKM/accelerated steps increase per-iteration cost marginally due to additional projections or extrapolation steps.

The overall complexity per iteration is $O(G + n)$ for single-row methods, and $O(\beta G + n)$ for batch sampling of size $\beta$ (Zhang et al., 2022, Gower et al., 2023). Empirical studies show that batch sampling ( $10\lesssim\beta\lesssim 100$ ) balances convergence speed and work per update.

5. Applications and Comparative Empirical Performance

Recent cNKZ implementations have proven especially effective for:

Large-scale nonlinear equation systems (e.g. $m \sim 10^4$ , $n \sim 10^3$ ), including equations with strong nonlinearity or sparsity constraints (Zhang et al., 2022, Gower et al., 2023, Xiao et al., 2024).
Inverse problems under noisy data: Demonstrated robust performance and fewer forward/adjoint solves compared to Landweber–Kaczmarz with line search (Leitao et al., 2020).
Constrained motion planning and coordinated multi-robot manipulation: Enabling rapid, robust projection onto the intersection of up to 80 nonlinear manifolds, with order-of-magnitude speed-ups in feasibility projection and success rates exceeding 90% in complex settings (Agrawal et al., 2024).
Graph realization and quadratic systems: Locally linear convergence established for systems of quadratic (or sphere) constraints (Zeng et al., 2020).

Empirically, cNKZ and its variants (especially those using greedy-residual sampling or Bregman geometry) outperform classical randomized Kaczmarz, SGD, and projected gradient methods by factors of 2–10 $\times$ in iterations and 10–50 $\times$ in wall-clock time for large systems under convex or nearly convex constraints (Zhang et al., 2022, Xiao et al., 2024, Gower et al., 2023).

6. Parameter and Implementation Guidelines

Tuning critical parameters:

Step size $\alpha_k$ : Basic Kaczmarz step uses $\alpha_k=1$ ; $\alpha_k\in(0,2)$ can enforce descent; for Bregman steps, the scalar $t_k$ is adaptively solved per-iteration (Zhang et al., 2022, Gower et al., 2023, Xiao et al., 2024).
Batch/sample size $\beta$ : Empirically, intermediate batch sizes ( $10\lesssim\beta\lesssim 100$ ) yield favorable trade-offs (Zhang et al., 2022).
Bregman generator $\phi$ : Must be strictly convex with $C$ as its domain; entropy, $\ell_1+\ell_2$ , and block-simplex penalties are common (Gower et al., 2023, Xiao et al., 2024).
Projection thresholds/tolerances: Set according to application specifics, e.g., $10^{-8}$ for equality, or application-specific physical tolerances (Agrawal et al., 2024).
Randomization vs. greedy selection: Greedy-residual or distance selection provides significant iteration savings in practice; uniform random selection is simple and effective for homogeneous systems (Zeng et al., 2020, Xiao et al., 2024).

7. Extensions, Limitations, and Current Research Directions

cNKZ methods can be further generalized to:

Handle general nonlinear manifold intersection via manifold-based projection formulation, as in complex robotic systems (Agrawal et al., 2024).
Incorporate acceleration (e.g., Motzkin or APSKM extrapolation steps) to improve convergence in ill-conditioned or nearly inconsistent systems (Zhang et al., 2022).
Integrate relaxed/adaptive projection step sizes (mirror-SPS, adaptive Bregman steps) for further empirical gains (Gower et al., 2023, Xiao et al., 2024).
Support additional side constraints via Bregman-generated domains (sparsity, simplex, entropy).
Employ scalable parallel implementations and composite constraint handling for high-dimensional applications.

Observed limitations include sensitivity of convergence to initializations in nonconvex settings, slowdown if projections onto $C$ are expensive, and the need for empirical tuning of per-constraint tolerances and step sizes in complex applications (e.g., multi-robot systems). Extensions to force/torque constraints, adaptive projection rules, and coupling with trajectory optimization are active research areas (Agrawal et al., 2024).

Key references:

(Zhang et al., 2022): Introduction of sampling Kaczmarz–Motzkin methods and their constrained variants (PSKM/APSKM)
(Gower et al., 2023): Bregman–Kaczmarz cNKZ with adaptive steps and strong convergence guarantees
(Zeng et al., 2020): Successive projection and greedy/greedy-residual cNKZ for nonlinear/inequality systems
(Agrawal et al., 2024): Manifold-based cNKZ for multi-robot manipulation
(Leitao et al., 2020): Projective Landweber–Kaczmarz with regularization for ill-posed constrained systems
(Xiao et al., 2024): Greedy randomized Bregman–Kaczmarz with sharp convergence for large-scale, structured, constraint-rich nonlinear systems