Convergence rates for Tikhonov regularization on compact sets: application to neural networks

Abstract: In this work, we consider ill-posed inverse problems in which the forward operator is continuous and weakly closed, and the sought solution belongs to a weakly closed constraint set. We propose a regularization method based on minimizing the Tikhonov functional on a sequence of compact sets which is dense in the intersection between the domain of the forward operator and the constraint set. The index of the compact sets can be interpreted as an additional regularization parameter. We prove that the proposed method is a regularization, achieving the same convergence rates as classical Tikhonov regularization and attaining the optimal convergence rate when the forward operator is linear. Moreover, we show that our methodology applies to the case where the constrained solution space is parametrized by means of neural networks (NNs), and the constraint is obtained by composing the last layer of the NN with a suitable activation function. In this case the dense compact sets are defined by taking a family of bounded weight NNs with increasing weight bound. Finally, we present some numerical experiments in the case of Computerized Tomography to compare the theoretical behavior of the reconstruction error with that obtained in a finite dimensional and non-asymptotic setting. The numerical tests also show that our NN-based regularization method is able to provide piece-wise constant solutions and to preserve the sharpness of edges, thus achieving lower reconstruction errors compared to the classical Tikhonov approach for the same level of noise in the data.