--- title: End-to-End Iterative Regularization url: https://www.emergentmind.com/topics/end-to-end-training-with-iterative-regularization type: topic --- # End-to-End Iterative Regularization End-to-end training with iterative regularization denotes a family of learning procedures in which the reconstruction operator, the iterative solver, or the optimization trajectory is treated as part of a single training pipeline, while regularization is supplied by explicit penalties, learned priors, the update structure, the stopping rule, or weighted averaging of iterates. In inverse problems, the central pattern is to combine a physical forward model with learned regularization inside an unrolled architecture and, in some cases, to use the learned output as a warm start for a variational refinement. In statistical learning, closely related analyses show that the number of passes over the data or the averaging scheme over optimization iterates can itself act as the effective regularization parameter [2203.02166] [2106.03538] [1405.0042] [2008.06736]. ## 1. Conceptual scope and problem classes The cited literature covers two closely connected settings. The first is ill-posed inverse problems, where reconstruction must be stabilized because the system may be underdetermined or unstable due to poor conditioning. The second is empirical risk minimization, where repeated optimization steps can fit sample noise unless the trajectory is controlled. In both settings, the key question is not only which objective is minimized, but also how the iterative procedure itself shapes the final estimator [2203.02166] [1405.0042]. In the inverse-problem setting, one representative formulation assumes an image that is sparse with respect to convolutional filters and considers an objective of the form $$ \underset{x}{\min}\ \frac{1}{2}\|x-y\|_2^2 + \alpha \sum_{k=1}^K \|h_k * x\|_1. $$ The regularizer is explicit, but its practical effect depends on the iterative reconstruction rule that uses it. In the statistical-learning setting, the canonical objective is expected square loss, $$ \inf_{w\in \mathcal H} \mathcal E(w), \qquad \mathcal E(w)=\int_{\mathcal H\times\mathbb R}(\langle w,x\rangle-y)^2\,d\rho(x,y), $$ with empirical counterpart $$ \widehat{\mathcal E}(w)=\frac1n\sum_{i=1}^n(\langle w,x_i\rangle-y_i)^2. $$ There, iterative regularization arises because the optimization dynamics over epochs alter the bias-variance tradeoff [2203.02166] [1405.0042]. A useful cross-sectional comparison is the following. | Study | Iterative object | Regularization mechanism | |---|---|---| | [2203.02166] | Unrolled iterative neural network | Learned convolutional sparsifying filters with data consistency | | [2106.03538] | Unrolled reconstruction operator plus refinement | Jointly learned variational regularizer and warm-started iterative solve | | [1405.0042] | Incremental gradient over epochs | Number of passes over the data acts as a regularization parameter | | [2008.06736] | Averaged optimization trajectory | Weighted iterate averaging yields adjustable regularization | Taken together, these works place “iterative regularization” in a broader sense than classical explicit penalization alone. The iteration can carry the regularizing effect through architecture, solver coupling, stopping time, or post hoc averaging. ## 2. Physics-informed unrolling and convolutional analysis operator learning In "Convolutional Analysis Operator Learning by End-To-End Training of Iterative Neural Networks" [2203.02166], convolutional analysis operator learning (CAOL) is reformulated so that the learned filters are optimized for the reconstruction algorithm that actually uses them. Classical CAOL trains filters separately on ground-truth images by minimizing a transform-learning objective, but that decoupled procedure ignores the subsequently employed reconstruction algorithm as well as the physical model responsible for image formation. The paper’s central claim is that iterative neural networks can overcome this limitation because they contain the physical model [2203.02166]. The variational model introduces auxiliary variables \(s_k\) and rewrites the nonsmooth problem as $$ \underset{x,\{s_k\}_k}{\min}\ \frac{1}{2}\|x-y\|_2^2 + \frac{\lambda}{2}\sum_{k=1}^K \|h_k * x - s_k\|_2^2 + \alpha \sum_{k=1}^K \|s_k\|_1, $$ with \(\lambda>0\). The unrolled network alternates between a regularization step and a data-consistency step: $$ z_j = \sum_{k=1}^K h_k^\ast * \mathcal{S}_{\alpha/\lambda}(h_k * x_j), $$ $$ x_{j+1} = \arg\min_x \frac{1}{2}\|Ax-y\|_2^2 + \frac{\lambda}{2}\|x-z_j\|_2^2, $$ initialized by $$ x_0 := A^\sharp y. $$ The second step is equivalent to solving $$ \big(A^\ast A + \lambda I\big)x = A^\ast y + \lambda z_j. $$ For non-Cartesian MRI with density compensation, \(A^\ast A\) and \(A^\ast y\) are replaced by preconditioned versions using \(A^\sharp\) [2203.02166]. This architecture is model-based rather than a generic CNN. The reconstruction is constrained by the measurement operator \(A\), the learnable part is the regularizer or filter bank, and the architecture mirrors a proximal splitting method. Under an orthonormal-basis assumption, the iteration corresponds to a backward-backward splitting algorithm for the regularized problem. The trainable parameter set is \(\Theta = \bigcup_k \{h_k\}\), and the network can also learn \(\lambda\) and \(\alpha\); to enforce positivity, the paper applies a Soft-Plus nonlinearity to them. Because soft-thresholding is not differentiable with respect to its threshold, the paper uses the smooth approximation $$ \tilde{\mathcal{S}}_t(z) = z + \frac{1}{2} \left( \sqrt{(z-t)^2 + b} - \sqrt{(z+t)^2 + b} \right), $$ with \(b=0.001\), and trains all methods with ADAM [2203.02166]. The principal significance of this formulation is that the learned filters are optimized through the full unrolled reconstruction pipeline. They are therefore tuned to the finite number of unrolled iterations, the exact measurement operator, and the specific data-consistency step, rather than to a standalone sparsity objective. ## 3. Jointly learned regularizers and warm-started variational refinement A different but closely related formulation appears in "End-to-end reconstruction meets data-driven regularization for inverse problems" [2106.03538], which proposes Unrolled Adversarial Regularization (UAR). The method combines an unrolled reconstruction operator \(G_\phi:Y\to X\) with a parametrized regularizer \(R_\theta:X\to\mathbb{R}\), and the two are trained jointly. The training objective is a min-max problem, $$ \underset{\phi}{\inf} \,\underset{R\in \mathbb{L}_1}{\sup}\,J_1\left(G_{\phi},R|\lambda,\pi_{y^\delta}\right), $$ which is equivalent, via Kantorovich-Rubinstein duality, to $$ \min_{\phi} \mathbb{E}_{\pi_{y^\delta}}\|y^{\delta}-{A}(G_{\phi}(y^{\delta}))\|_2^2 + \lambda W_1\left((G_{\phi})_{\#}\pi_{y^\delta},\pi_x\right). $$ The first term enforces measurement consistency, while the second matches the distribution of reconstructions to the ground-truth distribution through the Wasserstein-1 distance [2106.03538]. The regularizer acts adversarially. The alternating updates are $$ \phi_{k}\in \underset{\phi}{\arg\min}\,J_k^{(1)}(\phi), $$ with $$ J_k^{(1)}(\phi):=\mathbb{E}_{\pi_{y^\delta}}\left[\left\|{A}(G_{\phi}(y^{\delta}))-y^{\delta} \right\|_2^2 + \lambda R_{\theta_k}(G_{\phi}(y^{\delta}))\right], $$ and $$ \theta_{k+1}\in\underset{\theta: R_{\theta}\in \mathbb{L}_1}{\arg\max}\,J_k^{(2)}(\theta), $$ with $$ J_k^{(2)}(\theta):=\mathbb{E}_{\pi_{y^\delta}}\left[R_{\theta}(G_{\phi_k}(y^{\delta}))\right] - \mathbb{E}_{\pi_x}\left[R_{\theta}(x)\right]. $$ The paper also uses a gradient penalty, $$ \lambda_{\text{gp}}\left(\left\|\nabla R_{\theta}(x^{(\epsilon)})\right\|_2-1\right)^2, $$ to enforce the 1-Lipschitz condition [2106.03538]. The reconstruction network itself is iterative and unrolled. Inspired by a primal-dual-style algorithm, it replaces proximal operators in the image and measurement spaces with trainable CNNs: $$ h^{(\ell+1)}=\Gamma_{\phi_{\text{d}}^{(\ell)}}\left(h^{(\ell)},\sigma^{(\ell)}A(x^{\ell}),y^{\delta}\right), $$ $$ x^{(\ell+1)}=\Lambda_{\phi_{\text{p}}^{(\ell)}}\left(x^{(\ell)},\tau^{(\ell)}A^*(h^{\ell+1})\right), \quad 0\leq\ell\leq L-1, $$ initialized by $$ x^{(0)} = A^\dagger y^\delta,\qquad h^{(0)} = 0. $$ The learnable components include primal CNN blocks, dual CNN blocks, and step sizes \(\sigma^{(\ell)}\), \(\tau^{(\ell)}\), initialized at \(0.01\), with \(L=20\) unrolled layers [2106.03538]. After training, the learned reconstruction \(G_{\phi^*}(y^\delta)\) is used to initialize a separate variational refinement, $$ \min_{x \in \mathbb{X}} \|{A}(x) - y^{\delta}\|_2 + \lambda'\left( R_{\theta^*}(x) + \sigma \|x\|^2_2\right), $$ solved by gradient descent from $$ x^{(0)} = G_{\phi^*}(y^\delta). $$ The paper states that the end-to-end reconstruction gives an excellent initial point and that it takes significantly fewer iterations for gradient-descent to recover the optimal solution. It also claims well-posedness under compactness and continuity assumptions, noise stability in the sense that \(G_{\phi_n}\rightarrow G_{\phi^*}\) up to subsequences, and a principled tradeoff controlled by \(\lambda\): small \(\lambda\) prioritizes measurement consistency, while large \(\lambda\) prioritizes matching the image distribution [2106.03538]. ## 4. Epoch count and early stopping as iterative regularization "Learning with incremental iterative regularization" [1405.0042] develops a statistical theory in which the number of epochs acts as the effective regularization parameter. The setting is least-squares learning in a separable Hilbert space \(\mathcal H\), with bounded inputs and outputs. The algorithm is an incremental gradient method over the empirical risk, run one point at a time and repeated for multiple epochs: $$ \hat u_t^0=\hat w_t,\qquad \hat u_t^i=\hat u_t^{i-1}-\frac{\gamma}{n}\big(\langle \hat u_t^{i-1},x_i\rangle-y_i\big)x_i,\quad i=1,\dots,n, $$ followed by $$ \hat w_{t+1}=\hat u_t^n. $$ Here \(t\) indexes epochs, not individual sample updates [1405.0042]. The paper’s central principle is that the step size \(\gamma\) is fixed a priori and the number of epochs \(t\) is the only tuning parameter. Small \(t\) means that the algorithm has not fit the sample too closely; large \(t\) reduces approximation error but increases estimation error. In that sense, epochs control the bias-variance tradeoff and act exactly like a regularization parameter. The analysis compares the empirical iterate \(\hat w_t\) with a population iterate \(w_t\), decomposing the error into sample error and approximation error through inequalities such as $$ \mathcal E(\hat w_t)-\inf \mathcal E \le 2\kappa \|\hat w_t-w_t\|^2 + 2\big(\mathcal E(w_t)-\inf \mathcal E\big). $$ When the best linear solution exists, $$ \|\hat w_t-w^\dagger\|\le \|\hat w_t-w_t\|+\|w_t-w^\dagger\|. $$ These identities formalize the claim that optimization dynamics and statistical regularization are inseparable in multi-epoch training [1405.0042]. The strong universal consistency theorem states that if the stopping rule \(t^*(n)\) satisfies $$ t^*(n)\to\infty,\qquad \frac{t^*(n)^3\log n}{n}\to 0, $$ then $$ \mathcal E(\hat w_{t^*(n)})-\inf_w \mathcal E(w)\to 0 \quad\text{almost surely}. $$ If \(\mathcal O\neq\varnothing\), then also $$ \|\hat w_{t^*(n)}-w^\dagger\|_{\mathcal H}\to 0 \quad\text{almost surely}. $$ Finite-sample bounds make the same point quantitatively. For any \(r>0\), $$ \mathcal E(w_t)-\inf \mathcal E \le \left(\frac{r}{\gamma t}\right)^{2r}\|g\|_\rho^2. $$ In the attainable case \(r>1/2\), the excess risk obeys $$ \mathcal E(\hat w_t)-\inf \mathcal E \le C_3\frac{t^2}{n} + C_4 t^{-2r}, $$ with recommended stopping rule $$ t^*(n)=\left\lceil n^{\frac{1}{2(r+1)}}\right\rceil, $$ yielding $$ \mathcal E(\hat w_{t^*(n)})-\inf \mathcal E =O\!\left(n^{-r/(r+1)}\right) \quad\text{with high probability}. $$ In the non-attainable case \(00, \qquad \gamma := \frac{\eta}{1+\lambda \eta}, $$ and $$ P_k := 1-\prod_{i=0}^k \frac{\gamma_i}{\eta_i}. $$ The main theorem gives the exact relation in expectation $$ P_k\,\mathbb E[\tilde w_k] = \mathbb E[\hat w_k] - (1-P_k)\,\mathbb E[w_k], $$ together with $$ \|\mathbb E[\hat w_k]-\mathbb E[\tilde w_k]\|_2 \le \mathcal O\big((1-\lambda\gamma)^k\big). $$ In this sense, iterate averaging implements \(\ell_2\)-regularization. The practical interpretation is explicit: one can train only the unregularized model, choose the averaging weights accordingly, and obtain the effect of the regularized solution without retraining for each \(\lambda\) [2008.06736]. The result extends beyond vanilla SGD. For preconditioned SGD with positive definite preconditioner \(Q\), the induced regularizer becomes $$ R(w)=\frac12 w^\top Q w = \frac12 \|w\|_Q^2, $$ and the same structural relation holds. For Nesterov’s accelerated SGD, the paper defines a modified weighting scheme $$ P_k = 1 - \frac{\gamma}{\eta} \left( \frac{1-\sqrt{\gamma(\alpha+\lambda)}}{1-\sqrt{\eta\alpha}} \right)^{k-1}, $$ and proves analogous convergence, thereby answering the open question posed by Neu and Rosasco on whether an averaging scheme exists for NSGD that yields adjustable \(\ell_2\)-regularization [2008.06736]. The paper also treats general strongly convex and smooth objectives. There the result becomes approximate rather than exact: the averaged solution lies between two regularized solutions with tunable parameters $$ \lambda_1 = \frac{1}{\gamma}-\frac{1}{\eta}+\beta-\alpha, \qquad \lambda_2 = \frac{1}{\gamma}-\frac{1}{\eta}+\alpha-\beta. $$ When the objective is quadratic, \(\alpha=\beta\), and the approximate statement becomes exact regularization. This establishes iterate averaging as a form of iterative regularization rather than merely a variance-reduction device [2008.06736]. ## 6. Empirical demonstrations and methodological trade-offs The empirical studies span MRI reconstruction, CT reconstruction, and deep networks. In accelerated radial cardiac cine MRI, the CAOL-unrolling study evaluates a multi-coil, non-Cartesian encoding model with \(N_c = 12\) coil sensitivity maps, golden-angle radial sampling, density compensation preconditioning, and TorchKbNufft for the NUFFT implementation. The dataset consists of 15 healthy volunteers and 4 patients, with 216 cine MR images total, image size \(320 \times 320 \times 30\), split by subject into 12/3/4 for train/validation/test, retrospective undersampling at approximately \(R \approx 18\), and added Gaussian noise with \(\sigma=0.02\). Models were trained with \(K=8,16,24\), filter sizes \(k_f=3,5,7\), network depth \(T=4\), and \(n_{\mathrm{CG}}=4\) conjugate-gradient iterations, using ADAM with learning rate \(10^{-4}\). The proposed network was trained for 75 epochs (\(\sim 9\) hours), while the DnCn3D baseline was trained for 500 epochs (\(\sim 4\) days). Compared against decoupled CAOL and DnCn3D, the proposed method produced visibly better reconstructions than CAOL, clearly outperformed CAOL quantitatively, and achieved performance comparable to DnCn3D while remaining more interpretable and using fewer trainable parameters [2203.02166]. In sparse-view CT on Mayo Clinic low-dose CT data, the UAR study uses 2250 slices from 9 patients for training and 128 slices from 1 patient for testing, with parallel-beam geometry, 200 projection angles, and Gaussian noise with \(\sigma_e = 2.0\). Reported quantitative results include about \(33.84\) dB PSNR and \(0.86\) SSIM for AR, about \(31.55\) dB PSNR and \(0.85\) SSIM for ACR, about \(34.35\) dB PSNR and \(0.88\) SSIM for UAR with \(\lambda=0.1\), and about \(34.77\) dB PSNR and \(0.90\) SSIM for UAR with refinement. Reconstruction times are around \(22\) seconds per image for AR, around \(110\) seconds for ACR, around \(0.25\) seconds for UAR end-to-end, and around \(5.9\) seconds for UAR with refinement. The paper also reports an interpretable dependence on \(\lambda\): \(\lambda=0.001\) and \(\lambda=0.01\) are too weak, \(\lambda=0.1\) gives the best tradeoff, and \(\lambda=1.0\) is over-regularized [2106.03538]. In deep-network experiments, the iterate-averaging study trains VGG-16 on CIFAR-10, ResNet-18 on CIFAR-10, and ResNet-18 on CIFAR-100 with SGD for 300 epochs, averages checkpoints from epochs 61 to 300, uses geometric weighting \(p\in\{0.9999,0.999,0.99,0.9\}\), and recomputes batchnorm statistics after averaging. Reported results improve test accuracy from \(92.54\%\) to \(93.18\%\) for VGG-16 on CIFAR-10, from \(94.54\%\) to \(94.72\%\) for ResNet-18 on CIFAR-10, and from \(75.62\%\) to \(76.24\%\) for ResNet-18 on CIFAR-100, with very small additional computation time [2008.06736]. These experiments highlight a recurring trade-off. End-to-end unrolling can deliver fast inference and competitive quality, but the strongest formulations in this literature retain explicit data consistency, variational structure, or physically grounded initialization rather than replacing them with unconstrained feedforward mappings [2203.02166] [2106.03538]. ## 7. Interpretation, misconceptions, and scope A common misconception is that end-to-end training with iterative regularization is equivalent to training an arbitrary CNN on paired input-output examples. The inverse-problem studies considered here do not support that equation. In one case, the architecture alternates a learned thresholding-based regularization step with a data-consistency solve that explicitly uses the forward model \(A\); in the other, the unrolled operator is coupled to a learned 1-Lipschitz regularizer and may initialize a separate variational optimization. Both are model-based constructions rather than generic feedforward surrogates [2203.02166] [2106.03538]. A second misconception is that regularization must always be an explicit penalty coefficient in the loss. The least-squares and iterate-averaging analyses show otherwise. With fixed step size, the number of passes over the data acts as a regularization parameter, and weighted averaging of an unregularized trajectory can reproduce or approximate a regularized solution path. In these settings, stopping time and averaging weights are not ancillary implementation details; they are part of the estimator definition [1405.0042] [2008.06736]. A third misconception is that interpretability and performance are necessarily opposed. The MRI study reports performance comparable to DnCn3D while emphasizing a more transparent regularization structure and fewer trainable parameters, and the CT study combines end-to-end speed with the well-posedness and noise-stability guarantees of the variational setting. This does not imply that interpretability always improves performance, but it does show that the two objectives are not treated as mutually exclusive in the cited work [2203.02166] [2106.03538]. Taken together, the literature suggests a unifying principle: iterative procedures can be made part of the learning problem itself. In some formulations, the regularizer is learned through the full unrolled solver; in others, the unrolled solver initializes a convergent variational refinement; in still others, the optimization trajectory is regularized by early stopping or transformed into a regularized solution by iterate averaging. A plausible implication is that “end-to-end” in this domain is best understood not as the elimination of iterative structure, but as its incorporation into training, model selection, and inference. Source: https://www.emergentmind.com/topics/end-to-end-training-with-iterative-regularization