Discrepancy Principle in Inverse Problems

• Discrepancy Principle is an a posteriori method that selects regularization parameters by calibrating the residual norm with an estimated noise level.
• It employs iterative schemes like bisection and geometric search to adjust parameters, ensuring the residual stays within a prescribed interval even in nonlinear or discretized settings.
• The principle guarantees order-optimal convergence and prevents overfitting by balancing data fidelity against regularization bias, with broad applications from inverse problems to statistical learning.

The discrepancy principle is a foundational a posteriori strategy for selecting regularization parameters and discretization levels in the solution of ill-posed inverse problems. It prescribes choosing these quantities based on the magnitude of the residual between the observed, noisy data and the output of a regularized or discretized forward model, calibrated explicitly to the estimated noise level. This paradigm originated in deterministic inverse problems and has undergone significant generalization to nonlinear, statistical, and computationally intensive regimes, with mathematically rigorous guarantees on existence and optimality under broad conditions.

1. Classical Formulation and Generalizations

The classical Morozov discrepancy principle, originally formulated for ill-posed operator equations in Hilbert spaces, states that the regularization parameter $\alpha$ should be chosen so that the norm of the residual $\|F(x_\alpha) - y^\delta\|$ matches a prescribed multiple of the noise level $\delta$: $$\|F(x_\alpha) - y^\delta\| = \tau\,\delta \qquad (\tau>1)$$ where $F:X\to Y$ is the forward operator, $x_\alpha$ is the minimizer of a regularized Tikhonov functional

$$J_\alpha(x) = \|F(x) - y^\delta\|_Y^2 + \alpha\,\|x-x_0\|_X^2$$

and $y^\delta$ is the observed data with additive noise. The principle directly encodes the balance between data fidelity and regularization: $\alpha$ is decreased until the residual is commensurate with—but not below—the intrinsic noise floor, thus thwarting overfitting.

A commonly adopted generalization, particularly important in nonlinear or discretized settings where residuals rarely hit the target exactly, is the two-sided or relaxed discrepancy principle, which prescribes

$$\tau_1\,\delta \leq \|F(x_\alpha) - y^\delta\| \leq \tau_2\,\delta, \qquad 1 < \tau_1 \leq \tau_2$$

This accommodates irregular, non-monotonic behavior of $\alpha\mapsto \|F(x_\alpha) - y^\delta\|$ and ensures practical feasibility in iterative or finite-dimensional regimes (Albani et al., 2014, Ding et al., 13 Jun 2025).

2. Existence, Algorithmic Realization, and Extensions

Existence of a regularization parameter satisfying the discrepancy principle is not trivial, especially for nonlinear $F$ in Banach or Hilbert spaces. Under general assumptions—convexity, weak-lower-semicontinuity, and tangential cone conditions for $F$, and suitable coercivity for the penalty—one can guarantee existence provided the upper threshold $\tau_2$ is sufficiently larger than the lower, specifically, $\tau_2 \geq (3+2\gamma)\tau_1$ where $\gamma$ is the constant in the tangential cone condition (Ding et al., 13 Jun 2025). The same existence claims extend when joint selection of discretization level $m$ and regularization $\alpha$ is required, given dense discretization subspaces and weak compactness assumptions (Albani et al., 2014).

Effective algorithms typically proceed by iteratively refining $\alpha$ (and, if needed, the discretization level $m$) via bisection or geometric search over a grid. For each candidate, one solves the regularized minimization and evaluates the residual against the prescribed interval. For multidimensional parameter selection, an outer loop increments discretization fidelity until the relaxed discrepancy principle becomes feasible (Albani et al., 2014, Ding et al., 13 Jun 2025).

Pseudocode for practical implementation includes:

• Initializing $\alpha$ large enough so that the residual exceeds $\tau_2\delta$;
• Decrementing $\alpha$ (e.g., $\alpha \leftarrow q \alpha$ for $q<1$) until the residual drops below $\tau_1\delta$;
• Performing bisection between last above- and below-threshold values, stopping upon satisfaction of the interval condition;
• For discretization, increment $m$ if the interval condition cannot be met for any $\alpha$ at fixed $m$.

3. Regularization Theory: Convergence and Saturation

The discrepancy principle ensures that, as the noise level $\delta\to0$, the sequence of regularized solutions converges (weakly) to the minimum-norm solution of the noiseless problem under minimal assumptions. When both regularization and discretization are involved (with discretization error $\gamma_m=o(\delta)$), the principle yields

$$D_{\xi^\dagger}(x_\alpha^m, x^\dagger) = O(\delta), \qquad \|x_\alpha^m - x^\dagger\| = O(\sqrt{\delta})$$

under Bregman distance for general convex penalty or in norm for quadratic penalty (Albani et al., 2014). The convergence rates remain order-optimal for a broad class of source conditions, including low-order logarithmic types (Klinkhammer et al., 2022).

A sharp saturation phenomenon—first identified by Groetsch for linear problems and extended to nonlinear settings—demonstrates that

$$\sup_{\|F(x_\alpha^\delta)-y\|\leq\delta} \|x_\alpha^\delta - x^\dagger\| = o(\delta^{1/2}) \implies x^\dagger = x^*$$

i.e., it is impossible to achieve better than $O(\delta^{1/2})$ worst-case rates under the standard discrepancy principle; this bound is known as the saturation index (Jin, 2024).

4. Statistical, Adaptive, and Computational Extensions

The discrepancy principle generalizes naturally to a range of problems beyond deterministic, linear operators:

• Statistical linear inverse problems: For white-noise models where $\|y^\delta-y\|^2$ is infinite almost surely, a truncated or discretized residual is used, with modification to maintain probabilistic oracle inequalities achieving minimax rates (Jahn, 2022, Jahn, 2021). The modified principle adapts to both polynomial and exponential ill-posedness and requires only a single hyperparameter.
• Nonparametric Regression and Learning: In k-NN regression and kernelized machine learning, the minimum discrepancy principle provides a data-driven, computationally efficient mechanism for model selection and early stopping that achieves minimax optimality, outperforming traditional hold-out or cross-validation in typical regimes (Averyanov et al., 2020, Celisse et al., 2020).
• High-dimensional and econometric models: In conditional moment problems and modern regularized instrumental variables, a discrepancy-principle-based selection of regularization parameters achieves adaptivity to unknown smoothness and optimal doubly-robust inference without prior specification of source conditions (Tan et al., 2 Mar 2026).
• Kernel density estimation: Bandwidth selection via the discrepancy principle, comparing empirical and smoothed distribution functions via Kolmogorov or Kuiper distances, gives provable $L_1$-consistency under suitable Hölder conditions and explicit asymptotic rates under additional smoothness (Mildenberger, 2011).
• Stochastic optimization (SGD): The discrepancy principle as an a posteriori stopping rule for stochastic gradient descent delivers finite-iteration termination and convergence in probability to the minimum-norm solution as noise vanishes (Jahn et al., 2020).

5. Duality, Oracle Inequalities, and Connections to Other Principles

Under quadratic Tikhonov regularization in the linear case, the constraint $\|A f_\alpha - g\| = \tau$ can be rigorously imposed via Lagrange duality. The dual function is strictly concave, and its maximizer yields the exact parameter meeting the discrepancy principle, providing robust, globally convergent algorithms for parameter selection (Bonnefond et al., 2014).

The discrepancy principle is closely related to classical model selection criteria:

• Its adaptation to statistical problems connects it to Lepski's method and the balancing principle. In certain cases (e.g., $K=\text{Id}$), the stopping indices coincide pointwise (Jahn, 2022).
• Modified (smoothed) discrepancy principles can extend statistical adaptivity into regimes of higher smoothness, outperforming classical DP in situations with fast eigenvalue decay (Celisse et al., 2020).

A summary comparison of key adaptive rules:

Principle	Main Regime	Typical Guarantee
Discrepancy (DP)	Deterministic/white noise	Minimax rate, order-optimal under source cond.
Lepski Balancing	Direct/Indirect	Minimax or near-minimax, log-factor losses
Smoothed DP	High smoothness	Removes $n^{-1/2}$ gap, minimax in inner range

6. Practical Implementation and Numerical Evidence

Numerical studies across inverse problems (local volatility calibration, compressive sensing, geophysical inversion, kernel regression, k-NN regression, and kernel density estimation) substantiate the practical viability of the discrepancy principle:

• The residual is monitored during optimization or iteration; parameters are selected when it enters the prescribed discrepancy interval.
• For regression and learning, early stopping induced by the discrepancy principle is both statistically and computationally efficient, often achieving or exceeding the performance of cross-validation or information criteria with reduced overhead (Averyanov et al., 2020).
• In the context of density estimation and nonparametric regression, performance on real and synthetic testbeds demonstrates robust consistency across wide classes of smoothness and sample sizes (Celisse et al., 2020, Mildenberger, 2011).

Crucially, over-refinement (too small $\alpha$ or too fine a discretization) leads to overfitting noise, while too much regularization incurs excessive bias; the discrepancy principle intrinsically balances this trade-off.

7. Limitations, Recent Advances, and Outlook

Limitations arise when the noise level $\delta$ is unknown but estimable; classical DP can be suboptimal, requiring modification via rescaling operators or estimation of component variances to restore optimality (Jahn, 2021). Saturation phenomena imply unbreakable worst-case convergence rate barriers ($O(\delta^{1/2})$ in norm, or worse for low-regularity solutions) unless additional prior information or alternative methodologies are employed (Jin, 2024, Klinkhammer et al., 2022).

Recent research has extended existence results and regularization guarantees to broader classes of nonlinear and nonconvex settings, with precise control on convergence in Bregman or strong norms under natural source conditions (Ding et al., 13 Jun 2025, Klinkhammer et al., 2022). Ongoing developments involve adaptive, fully data-driven implementations, especially in high-dimensional statistical and machine learning contexts, and further unification with other adaptive regularization and model selection paradigms.

The discrepancy principle remains a central concept in regularization theory with persisting significance and broad applicability across modern computational, statistical, and learning-theoretic frameworks (Albani et al., 2014, Ding et al., 13 Jun 2025, Jin, 2024, Celisse et al., 2020, Tan et al., 2 Mar 2026).