Relaxed regularization for linear inverse problems

Abstract: We consider regularized least-squares problems of the form $\min_{x} \frac{1}{2}\Vert Ax - b\Vert_2^2 + \mathcal{R}(Lx)$. Recently, Zheng et al., 2019, proposed an algorithm called Sparse Relaxed Regularized Regression (SR3) that employs a splitting strategy by introducing an auxiliary variable $y$ and solves $\min_{x,y} \frac{1}{2}\Vert Ax - b\Vert_2^2 + \frac{\kappa}{2}\Vert Lx - y\Vert_2^2 + \mathcal{R}(x)$. By minimizing out the variable $x$ we obtain an equivalent system $\min_{y} \frac{1}{2} \Vert F_{\kappa}y - g_{\kappa}\Vert_2^2+\mathcal{R}(y)$. In our work we view the SR3 method as a way to approximately solve the regularized problem. We analyze the conditioning of the relaxed problem in general and give an expression for the SVD of $F_{\kappa}$ as a function of $\kappa$. Furthermore, we relate the Pareto curve of the original problem to the relaxed problem and we quantify the error incurred by relaxation in terms of $\kappa$. Finally, we propose an efficient iterative method for solving the relaxed problem with inexact inner iterations. Numerical examples illustrate the approach.