Adaptive Momentum via Minimal Dual Function for Accelerating Randomized Sparse Kaczmarz

Abstract: Recently, the randomized sparse Kaczmarz method has been accelerated by designing heavy ball momentum adaptively via a minimal-error principle. In this paper, we develop a new adaptive momentum method based on the minimal dual function principle to go beyond the exact measurement restriction of the minimal-error principle. Moreover, by integrating the new adaptive momentum method with the quantile-based sampling, we introduce a general algorithmic framework, called quantile-based randomized sparse Kaczmarz with minimal dual function momentum, which provides a unified approach to exact, noisy, or corrupted linear systems. In addition, we utilize the discrepancy principle and monotone error as stopping rules for the proposed algorithm. Theoretically, we establish linear convergence in expectation of Bregman distance up to a finite horizon related to the contaminated level. At last, we provide numerical illustrations on simulated and real-world data to demonstrate the effectiveness of our proposed method.

• The paper introduces Quantile-RaSK-MM, a novel approach that uses a minimal dual function for adaptive momentum to accelerate sparse recovery.
• It integrates quantile-based row selection to filter out outlier equations, ensuring robust performance even with significant data corruption.
• Empirical results on synthetic and real datasets demonstrate improved convergence speed and accuracy in both noisy and highly corrupted regimes.

Adaptive Momentum via Minimal Dual Function for Accelerating Randomized Sparse Kaczmarz

Problem Setting and Motivation

The paper addresses robust and accelerated algorithms for sparse recovery in linear inverse problems of the form $Ax = b$, with particular focus on systems contaminated by arbitrary, possibly large-magnitude corruptions in the measurement vector. Classical randomized Kaczmarz (RK) and its sparse adaptations (RaSK) provide randomized iterative projections for such systems but face significant slowdowns and even failure for corrupted data, primarily because standard projection selection and momentum strategies either assume exact or small additive Gaussian noise, or require knowledge of the true solution or noise/corruption structure. This work proposes a generalization of adaptive momentum updates via a minimal dual function approach, enabling accelerated convergence even in the presence of unknown corruptions and allowing the integration of quantile-based row selection for outlier-robustness.

Algorithmic Contributions

The principal methodological innovation is the Quantile-based Randomized Sparse Kaczmarz with Minimal Dual function Momentum (Quantile-RaSK-MM). Rather than minimizing the Bregman distance to the (unknown) solution for momentum parameter selection, as in the minimal-error principle [lorenz2023minimal], the authors minimize a perturbed dual objective whose gradient can be computed at each iteration using only observed (potentially corrupted) data. This update is compatible with both quantile-based acceptance of "uncorrupted" equations—adaptively filtering out likely outlier equations based on current residuals—and with heavy-ball-type momentum for accelerated convergence.

After each iteration, the algorithm:

1. Computes residuals for all equations, retaining those within a quantile threshold as likely uncorrupted.
2. Samples a row from the retained equations, and solves a small two-dimensional minimization (via solving a linear 2x2 system) for momentum parameters $\alpha_k, w_k$ that minimize the perturbed dual function.
3. Updates the dual and primal variables accordingly, projecting in Bregman geometry suited for sparsity regularization.

Figure 1: Relative error after 2000 iterations of Quantile-RaSK-MM for a range of $\beta$ (corruption fraction) and $q$ (quantile threshold) underscores the effectiveness and parameter sensitivity of quantile selection.

The approach is mathematically formalized with convergence guarantees for arbitrary contamination level, including explicit bounds for exact, noisy, and corrupted regimes, and extends naturally to broader convex regularizers.

Theoretical Analysis

The analysis demonstrates that, under appropriate parameterization (notably, quantile parameter $q > \beta$, with $\beta$ the fraction of corrupted equations), Quantile-RaSK-MM enjoys linear convergence in expectation, up to a horizon determined by the contamination level $\delta$. The rate is explicit in terms of matrix geometry (minimal singular values of quantile-submatrices), proportion of accepted equations, and the convexity and regularity of the objective.

A notable technical point is the replacement of the true residual by one computable from only the observed, possibly contaminated data, allowing meaningful parameter updates even with adversarial or unstructured corruptions. The algorithm gracefully degrades to previous RaSK and BK-EM methods when $q=1$ and $\beta=0$, highlighting its generality.

The paper also provides the first formalizations and analysis of stopping criteria (discrepancy principle and monotone error) for quantile-based Kaczmarz methods, including extension to Bregman distances for non-quadratic regularization.

Empirical Evaluation

Extensive simulations on synthetic Gaussian and real-world matrices demonstrate empirical improvements in both convergence speed and final accuracy across a spectrum of contamination regimes, for both underdetermined and overdetermined systems. In highly corrupted cases, the algorithm robustly ignores outlier equations, converging to solutions with lower error than classical or recent quantile-based approaches lacking acceleration.

Figure 2: The performance of the minimal error (ME) rule and the discrepancy principle (DP) on Quantile-RaSK-MM, highlighting robust termination under both noise and corruption.

Figure 3: The performance of different methods on real matrices in the corrupted case, indicating the superior speed and accuracy of Quantile-RaSK-MM for high sparsity and high corruption levels.

In imaging (CT reconstruction) settings, the algorithm yields visually faithful reconstructions under both noise and heavy corruption, validating the theoretical predictions regarding robustness.

Figure 4: CT reconstructions by Quantile-RaSK-MM with ME stopping rule: left—ground truth; middle—recovered from corrupted system; right—recovered from noisy system, for $N=20$ and $\alpha_k, w_k$0.

Implications and Future Perspectives

The minimal dual function momentum principle obviates the need for knowledge of the true solution and admits straightforward integration with robust subsampling strategies. This approach retains the efficiency of randomized Kaczmarz algorithms, complements existing momentum acceleration paradigms, and extends to inverse problems with general convex data terms and regularizers.

Practically, the method provides a robust, scalable, and easily implementable alternative for large-scale sparse recovery under non-ideal measurement conditions, with immediate applications in tomography, machine learning, and compressed sensing. Theoretically, the main limitation—a residual error floor determined by corruption/noise—parallels unavoidable bias in robust estimation. Future research is likely to focus on strategies for removing or reducing this error horizon, possibly through additional regularization, robust estimation frameworks, or improved quantile statistics, as well as extensions to infinite-dimensional inverse problems with adaptive sampling in function spaces.

Conclusion

The Quantile-RaSK-MM algorithm provides an effective and theoretically principled framework for sparse recovery under arbitrary measurement corruption. The minimal dual function approach to momentum selection is widely applicable and paves the way for robust, accelerated randomized projection methods in a range of ill-posed problems. The synthesis of quantile-based sampling, adaptive dual momentum, and rigorous stopping criteria positions this methodology as a leading candidate for future robust large-scale solvers.