Implicit Regularization for Linearity in Deep Networks

• The paper demonstrates that implicit regularization induces near-linear behavior between samples using the dynamics of stochastic gradient descent.
• It introduces quantitative measures like gradient gap deviation and deflection to capture local and global curvature in over-parameterized ReLU networks.
• Empirical and theoretical evidence shows that despite extreme expressiveness, networks maintain low complexity, enhancing generalization.

Implicit regularization for linearity refers to the set of phenomena by which optimization algorithms, particularly stochastic gradient descent (SGD) and its variants, bias over-parameterized neural networks toward linear or nearly-linear solutions between data samples—despite the absence of explicit regularization terms and the extreme expressiveness of the model class. This effect is now recognized as a central mechanism underlying the generalization ability of deep networks. In the context of ReLU (rectified linear unit) neural networks and other piecewise linear models, implicit regularization ensures that, although the network can in principle fit highly nonsmooth functions, the fit between samples is characterized by low complexity and, most notably, by near-linearity along input-space paths connecting training samples.

1. Mechanisms of Implicit Regularization Toward Linearity

The primary mechanism promoting linearity between samples in over-parameterized deep ReLU networks is the statistical effect of random initialization in conjunction with the dynamics of (stochastic) gradient descent. Along linear interpolations in input space—i.e., paths of the form $X_t = (1-t)X_0 + tX_1$—the network's output is piecewise linear, with "breakpoints" where the activation pattern of ReLUs changes. The gradient of the output with respect to the input, along such a path, is piecewise constant but can change at breakpoints. The sequence of these gradients can be modeled statistically as a random walk bridge, i.e., a martingale with endpoints pinned to the gradients at $X_0$ and $X_1$.

This framework describes how, despite the combinatorial potential for highly nonlinear interpolation (due to the exponential number of activation patterns), the actual solution learned by SGD exhibits minimal curvature and closely tracks the linear interpolation between data points. The underlying reasons are:

• The number of breakpoints along the input path grows only linearly with network width (not exponentially).
• The typical size (standard deviation) of the gradient jump at a breakpoint decreases as network width ($m$) and input dimension ($d$) increase.
• SGD solutions remain close to initialization, preserving the smallness of the gradient jumps.

2. Quantifying Linearity: Gradient Gap Deviation and Deflection

To measure linearity and local/global curvature, two key statistics are introduced:

Gradient Gap Deviation:

Defines the standard deviation of deviation in the network gradient, beyond that expected of a straight-line interpolation, over adjacent regions: $$\text{Gap Deviation} = \left\{\mathbb{E}[T_k^2]\right\}^{1/2}, \quad T_k = (S_k - S_0) - \frac{k}{\mathcal{K}}(S_{\mathcal{K}}-S_0)$$ where $S_k$ is the sequence of input-gradient projections at each segment (random walk steps), $k$ enumerates the breakpoints, and $\mathcal{K}$ is the total number of segments.

Gradient Deflection:

Measures the global $\ell_2$ deviation between the network's projected gradient and the straight-line interpolant at position $t$: $$\text{Deflection} = \left\{\mathbb{E}_{X_0 \neq X_1}[D(t; X_0, X_1)^2]\right\}^{1/2}$$

$$D(t; X_0, X_1) = \nabla_v u(t; X_0, X_1) - [(1-t)\nabla_v u(0; X_0, X_1) + t\nabla_v u(1; X_0, X_1)]$$

where $u$ is the network output and $\nabla_v$ the directional derivative along the interpolation.

Both quantities serve as proxies for curvature and (in the ReLU context, where the Hessian is undefined at breakpoints) encode the effective smoothness of the learned mapping. The deviation is proven analytically to be small at initialization and remains small after SGD training.

3. Statistical and Dynamical Control of Curvature

The statistical structure of breakpoints and gradient jumps is determined primarily by the architecture and random initialization:

• The number of breakpoints for a two-layer ReLU network with $m$ units is binomially distributed, $\mathcal{K} \sim \mathrm{Binomial}(m, 1/2)$.
• The typical size of a gradient jump per breakpoint is approximately $\mathcal{N}(0, 4/(md))$ at the first layer, and smaller in deeper layers.

Stochastic gradient descent preserves the smallness of weights and their proximity to initialization (the "lazy regime"), ensuring that the intrinsic statistical control over gradient gaps is maintained throughout training. As a consequence, the interpolating function between data points is piecewise linear with only small and infrequent departures from global straightness.

4. Observational and Theoretical Evidence

Empirical studies on multilayer perceptrons, residual networks, and VGG-like networks confirm that gradient gap deviation and gradient deflection are uniformly small, both at initialization and after standard training with SGD in the over-parameterized regime. This is in contrast to the network's combinatorial expressiveness, and is a direct manifestation of implicit regularization.

Analytically, the gradient gap deviation in the random walk bridge model peaks at the midpoint between endpoints and is bounded by the variance of the step size: $$\text{Gap Deviation} = \sigma \sqrt{k \left(1-\frac{k}{\mathcal{K}}\right)}$$ for $k$th segment, maximal at $k = \mathcal{K}/2$.

5. Implications for Generalization and Complexity

Linearity enforced by implicit regularization is directly related to the complexity of the learned function. It suppresses the network's capacity to form wild oscillations or high-curvature "bumpy" functions in the regions between (possibly distant) samples, even in absence of any explicit norm or curvature regularization.

This effect is reminiscent of the classical regularization provided by minimum-norm or minimum-curvature (spline-like, RKHS norm) penalties, yet it is not imposed via the loss but arises spontaneously from the combination of random initialization, over-parameterization, and SGD. This mechanism explains the often observed, unexpectedly good generalization of modern deep neural networks without explicit regularization.

6. Broader Context, Related Methods, and Limiting Factors

The implicit promotion of linearity between samples is a robust empirical phenomenon across architectures and datasets, and complements other forms of implicit bias, such as margin maximization or low-norm solution selection. While in some settings the implicit bias can be formally related to optimization in certain norms, in deep ReLU networks the appropriate measure of function complexity turns out to be these statistical measures of curvature and deviation from linearity.

However, the power of implicit regularization is modulated by architectural choices, the size of initialization, and the training algorithm. While over-parameterization and small initialization favor nearly-linear interpolation, different regimes or aggressive training can potentially break this effect. Further work is needed to delineate these boundaries, unify curvature-based and norm-based perspectives, and extend analysis to non-piecewise-linear architectures.

---

Concept	Formula/Model	Regularization Role
Gradient gap deviation	$\left\{\mathbb{E}[T_k^2]\right\}^{1/2}$	Local curvature/Hessian proxy
Gradient deflection	$\left\{\mathbb{E}[D(t)^2]\right\}^{1/2}$	Global curvature, path linearity
Random walk bridge model	$S_k = S_{k-1} + \mathrm{gap}$, pinned ends	Dynamics of gradient along path
Smallness of gap deviation	Empirically $<\!\!<\!1$ after SGD	Implies near-linear function between samples

Implicit regularization for linearity thus emerges as a central explanatory mechanism for the low-complexity, generalizing behavior of modern neural networks. Through the lens of random walk bridges for gradients and quantitative deviation-from-linearity statistics, it provides a highly predictive and architecture-agnostic account of why over-parameterized models avoid overfitting despite their expressiveness (Kubo et al., 2019).