Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks

Abstract: Although neural networks are remarkably effective, their underlying optimization principles remain theoretically elusive, often characterized by non-convex landscapes and stochastic heuristics. In this work, we propose a paradigm shift by replacing the discrete training problem of shallow neural networks with a well-posed continuum variational surrogate. We identify a family of $λ$-convex functionals over parameter densities in weighted Sobolev spaces and prove that these variational problems are globally well-posed, stable, and exhibit unexpected almost $C<sup>3$ regularity. Unlike existing Wasserstein-based or Mean-Field approaches, which often face limited regularity and discretization challenges, our formulation provides direct access to elliptic regularity and convex analysis. This allows us to prove that the optimal parameter density can be obtained by solving a single linear system, bypassing iterative optimization entirely. We establish explicit generalization error controls at a rate of $1/α$ relative to the regularization parameter, and prove that finite-width networks of size $N$ achieve the continuum optimum at an $O(1/N)$ rate. This perspective bridges the gap between the Neural Tangent Kernel (NTK) and feature-learning regimes, providing a principled framework for understanding over-parameterization through the lens of variational calculus.

• The paper introduces a convex variational formulation for shallow neural networks, ensuring existence and uniqueness through optimization in a weighted Sobolev space.
• It establishes finite-width consistency by showing that discrete networks approximate the continuum optimum with an error scaling of O(1/N).
• The framework elucidates implicit regularization, linking parameter smoothness and statistical stability to explicit ridge regression and linear elliptic PDE solutions.

Variational Reformulation of Shallow Neural Networks: A Rigorous Functional Analytic Perspective

The paper "Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks" (2607.02003) proposes and develops a mathematically rigorous, direct variational formulation for the training of shallow (single-hidden-layer) neural networks. Instead of relying on classical non-convex parameter optimization or mean-field/Wasserstein perspectives, the paper recasts shallow network training as a convex optimization problem over parameter densities in a weighted Sobolev space. This approach enables precise analysis of existence, uniqueness, regularity, and generalization, and provides new insight into the implicit bias and regularization behavior of overparameterized models.

---

Variational Formulation and Exactness

The central technical contribution is a convex, well-posed variational formulation of the shallow neural network training problem. Given a target function $f$ and feature map specified by the activation $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$, the classical finite-$N$ model corresponds to parameterizing signed measures (atomic, with $N$ terms) over the parameter space $\Omega$. The variational approach considers, instead, parameter densities $u$ in the weighted Sobolev space $\mathcal{W}=W^{1,2}(\Omega)\cap L^2_\omega(\Omega)$ and optimizes a functional

$$\mathcal{F}^{(f)}_{\alpha,\beta}(u) = \mathcal{R}(f,u) + \alpha \|u\|^2_{L^2_\omega} + \beta \|\nabla u\|^2_{L^2},$$

with $\mathcal{R}(f,u)$ the squared $L^2$ risk and $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$0 regularization terms enforcing $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$1-norm control and Sobolev smoothness, respectively. This transformation to a continuum, convex, strongly regularized space yields several crucial properties:

Exactness (No Lavrentiev Gap): The infimum of the risk functional is unchanged whether one optimizes over finite atomic measures (finite-width networks), general measures in $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$2, $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$3 densities, or even compactly supported smooth functions. There is no gap between the finite-discrete and continuum problems (Theorem 1).

Finite-Width Consistency: Finite-$h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$4 networks achieve the continuum optimum up to $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$5 error (Proposition 2). This places the continuum variational formulation as an exact relaxation justifying analytic techniques.

---

Convexity, Regularity, and Solution Structure

Unlike mean-field/Wasserstein approaches (which typically guarantee only displacement convexity and suffer limited regularity), this $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$6-based approach yields functional objectives with global $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$7-convexity ($h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$8 in $h(\theta, x)=\sigma(\theta_0+\theta'\cdot x)$9, $N$0 in $N$1), ensuring:

• Existence and uniqueness of minimizers.
• Exponential gradient flow convergence to equilibrium in continuous time, with explicit rates governed by $N$2.
• Structural stability to data perturbations.

The minimizer $N$3 is the solution to a linear elliptic PDE (an explicit Euler–Lagrange equation), yielding strong elliptic regularity. Remarkably, the optimal parameter density is always nearly $N$4 smooth ($N$5 for any $N$6), even for merely Lipschitz activations. This regularity surpasses typical results in NTK- or Wasserstein-limits, where minimizer smoothness is closely tied to the regularity of the activation.

Figure 1: Empirical reconstruction of a noisy $N$7 target with 50 samples. The regularized variational solution closely fits the target and extrapolates with high regularity, unlike the noisier neural network baseline and oscillatory unregularized solution.

---

Statistical Stability and Generalization

The variational framework enables explicit, quantitative analysis of stability and generalization. The paper proves that the minimizer $N$8 depends Lipschitz-continuously on the target $N$9 in both $N$0 and $N$1 norms, with explicit constants involving $N$2, $N$3, and weights. In the noisy data setting, the expected error in the density scales as $N$4 with the noise level $N$5, and the generalization error is correspondingly controlled.

This analysis yields regularization parameters $N$6 with explicit interpretation: for example, $N$7 directly governs stability and robustness, and the regularization does not merely serve as a heuristic but quantitatively shapes implicit bias and error.

---

Computational Implementation

A further significant result is that, due to the convex-quadratic structure of the regularized functional, the optimum can be computed by solving a single linear system—no iterative non-convex optimization or gradient descent is required. Using basis function expansions (polynomials, trigonometric), the parameter density is projected onto a finite-dimensional subspace, and the minimization reduces to ridge regression with explicit, positive-definite matrix structure.

Numerical examples confirm the theoretical analysis: regularized solutions exhibit strong denoising, high regularity, and stability even under outliers, and match classical neural network baselines in standard regression benchmarks. The method is also robust to the choice of basis and scales well for moderate $N$8 via exploitation of sparsity in the feature matrix.

Figure 2: Visualization for a two-dimensional regression problem ($N$9). The variational method reconstructs the smooth, radially-symmetric structure of the true function with high fidelity and regularity.

---

Theoretical and Practical Implications

Theoretical implications:

This variational approach provides a rigorous explanation for the empirical non-overfitting and implicit bias toward smooth parameterizations observed in overparameterized shallow networks. The absence of a Lavrentiev gap rules out non-physical minimizers and establishes a mathematically precise connection between continuum and finite networks. The derivation of near-$\Omega$0 regular parameter densities—without explicit dependence on the smoothness of the activation—suggests that overparameterization systematically drives solutions toward low-dimensional, regular manifolds in parameter space.

Practical implications:

The framework provides a principled basis for new regularized training algorithms (ridge- or kernel-based, with explicit convexity and stability) in shallow architectures, without recourse to the infinite-width/lazy NTK regime or computationally intensive Wasserstein flows. The explicit error rates with respect to network width and regularization enable fine-grained algorithmic tuning.

Limitations and Future Directions:

The scope is currently restricted to single-layer networks, as deeper architectures introduce non-convexity and coupled nonlinear PDEs in the variational formulation. However, the analytical structure established—in particular, the convexity and regularity toolkit—suggest a viable route toward multilayer extensions, possibly via implicit/variational time discretization of gradient flows or other convex-analytic relaxations. The convex variational lens could guide future convergence analyses in deep learning and provide new insight into the geometry of implicit regularization.

---

Conclusion

This work establishes a new variational perspective for shallow neural network optimization, unifying convex analysis, elliptic PDE theory, and statistical learning bounds. The resulting theory delivers global convexity and near-$\Omega$1 regularity for the continuum minimizers, exact correspondence with finite-width models, and principled regularization guarantees. These results clarify the implicit bias under overparameterization and open concrete avenues for future analytic and algorithmic development in the training and understanding of neural networks.

---

Reference:

Born Discrete, Made Smooth: Variational Formulation of Shallow Neural Networks (2607.02003)