Normal Consistency Regularization

• Normal consistency regularization is a method that enforces function stability by balancing empirical risk with a graph-based total variation term.
• It distinguishes between overfitting, consistency, and underfitting through careful tuning of regularization parameters relative to the data’s geometric scale.
• The approach leverages TL¹ convergence, Γ–convergence, and optimal transport methods to rigorously connect discrete models with continuum limits and enhance theoretical insights.

Normal consistency regularization is a form of regularization that enforces certain compactness or invariance properties on the solution space, often by penalizing oscillatory, unstable, or otherwise degenerate behaviors in learned functions. Within regularized empirical risk minimization, particularly for classification on finite samples, normal consistency regularization precisely modulates the balance between fidelity to data labels and smoothness of the solution. This is accomplished by tuning the strength of regularization terms—such as graph-based total variation—relative to the intrinsic geometric scale of the data graph. The concept provides a mathematically rigorous framework for distinguishing regimes of underfitting, overfitting, and consistency, linking regularization to convergence in suitable metrics and to notions of compactness in function spaces (Trillos et al., 2016). The following sections provide an in-depth exploration of the principle, its characterizations, and implications.

1. Mathematical Formulation of Consistency-Regularized Empirical Risk

In the setting of binary classification on a data cloud $\{x_1,\dots,x_n\}$ with labels $y_i \in \{0,1\}$, the normal consistency regularized empirical risk functional is: $$R_{n,\lambda}(u_n) = \lambda \cdot \mathrm{GTV}_{n,\varepsilon}(u_n) + R_n(u_n)$$ where:

• $$R_n(u_n) = \frac{1}{n} \sum_{i=1}^n |u_n(x_i) - y_i|$$ is the empirical risk,
• $$\mathrm{GTV}_{n,\varepsilon}(u_n) = \frac{1}{n^2 \varepsilon^{d+1}} \sum_{i,j} \eta\left(\frac{x_i-x_j}{\varepsilon}\right) |u_n(x_i)-u_n(x_j)|$$ is a graph total variation (GTV) regularizer on the data-driven neighborhood graph, with $\varepsilon$ the connectivity scale and $\eta$ a symmetric kernel.

In the continuum, the analogous energy is

$$R_\lambda(u) = \lambda \sigma_\eta \cdot \mathrm{TV}(u) + R(u)$$

where $\mathrm{TV}(u)$ is a (possibly weighted) total variation and $\sigma_\eta$ is a kernel-dependent normalization.

The minimizer $u_n^\star$ of this energy depends on the balance between $R_n$ and the GTV term, as modulated by the regularization parameter $\lambda$ and the graph scale $\varepsilon$.

2. Regimes: Overfitting, Underfitting, and Consistency

The paper identifies three distinct scaling regimes for $\lambda_n$ relative to $\varepsilon_n$ as $n\to\infty$ (Trillos et al., 2016):

Regime	Scaling	Limiting Behavior of Minimizer $u_n^\star$
Overfitting	$\lambda_n \ll \varepsilon_n$	$u_n^\star \to$ empirical label fn $l_n$ (oscillatory, non-compact in $L^1$)
Consistency	$\varepsilon_n \ll \lambda_n \ll 1$	$u_n^\star \to$ Bayes classifier $u_B$ (in $TL^1$)
Underfitting	$\lambda_n \to \textrm{const}>0$ or $\infty$	$u_n^\star \to$ overly smoothed fn (e.g., median)

Consistency (Compactness): $\varepsilon_n \ll \lambda_n \ll 1$

In this regime, the regularization is strong enough to suppress label noise-driven oscillations but weak enough to avoid over-smoothing, leading the discrete minimizer $u_n^\star$ to converge in the transport-$L^1$ (TL$^1$) metric to the Bayes classifier $u_B$.

Overfitting: Loss of Compactness

If $\lambda_n$ is too small, the solution memorizes the data label assignment, resulting in a highly oscillatory function $l_n$ that lacks compactness in $L^1$ (the function-valued limit does not exist; only a generalized, Young measure-type limit exists).

Underfitting: Excessive Smoothing

Too large $\lambda_n$ forces solutions toward excessive regularity, so $u_n^\star$ converges to a constant or smoothed function that discards meaningful label structure (approaching the data median).

3. Role of Transport–$L^1$ and Young Measures

A critical analytical tool is the $TL^1$ metric, which enables meaningful comparison between functions defined on the empirical data cloud and those defined with respect to the population measure. The $TL^1$ metric uses optimal transportation maps $T_n$ between the empirical and underlying measures and quantifies convergence/failure of compactness as follows:

• $u_n \to_{TL^1} u$ means that $u_n \circ T_n \to u$ in $L^1$.
• In the overfitting regime, $l_n$ fails to converge in $L^1$ but admits a generalized limit as a Young measure: a measurable family of probability measures that "describes" the oscillatory limit of the sequence.

This interpretation rigorously connects classical overfitting to loss of compactness in functional spaces commonly used in analysis and PDEs.

4. Γ–Convergence, Discrete-to-Continuum Limits, and Optimal Transport

The mechanism for establishing consistency rigorously is via Γ–convergence:

• The paper proves that, under appropriate scaling of $\varepsilon_n$ and $\lambda_n$, the discrete GTV functional converges (in the Γ–sense) to the continuum total variation.
• This convergence, together with compactness in $TL^1$, ensures that discrete minimizers converge to minimizers of the continuum energy, i.e., the Bayes classifier.

The construction and quantitative control of transportation plans between discrete samples and the underlying distribution (including error estimates) are fundamental to this argument.

5. Choice of Regularization Parameters

A central deliverable of the framework is a guide for regularization parameter selection:

• $\varepsilon_n \gg (\log n)^\alpha/n^{1/d}$ ensures graph connectivity,
• $\lambda_n$ must satisfy $\varepsilon_n \ll \lambda_n \ll 1$ for consistency,
• $\lambda_n \ll \varepsilon_n$ leads to overfitting, and $\lambda_n \to \infty$ to underfitting.

This gives a non-asymptotic, data-dependent prescription for robust regularization in high-dimensional, nonparametric settings.

6. Modern Analytical Tools: Compactness, Convex Analysis, and Concentration

Key analytical tools employed:

• Compactness theory in metric measure spaces (via $TL^1$ and pushforward measures).
• Convex duality and subdifferential analysis for discrete TV regularizers; Fenchel duality is used to characterize minimizers' behavior in different regimes.
• Concentration inequalities (e.g., Hoeffding's inequality) are used to control fluctuations of empirical risk terms and to quantify convergence rates.

This synthesis links machine learning phenomena (overfitting, underfitting, generalization) with deep results from the calculus of variations and functional analysis.

7. Implications and General Significance

Normal consistency regularization, as mathematically formulated in this framework, provides:

• A precise, rigorous bridge between empirical risk minimization, regularization, and geometric properties of data-driven function spaces.
• A transparent understanding of overfitting as a loss of compactness: solutions can fail to converge to classical functions, necessitating regularization-induced compactness.
• Strong differentiability between “undesirable” minimizers (empirical label functions with oscillatory, non-convergent limits) and “consistent” minimizers (sequences converging to the population Bayes classifier as measured in $TL^1$).

This analysis establishes theoretical guarantees for regularization-based machine learning algorithms and informs principled selection of regularization parameters in practice (Trillos et al., 2016). It also exposes deep connections between machine learning, analysis, and partial differential equations, making it applicable to a wide range of high-dimensional nonparametric estimation problems where sample-driven geometry and regularization interact non-trivially.