Enhanced Feature Learning via Regularisation: Integrating Neural Networks and Kernel Methods (2407.17280v1)

Abstract: We propose a new method for feature learning and function estimation in supervised learning via regularised empirical risk minimisation. Our approach considers functions as expectations of Sobolev functions over all possible one-dimensional projections of the data. This framework is similar to kernel ridge regression, where the kernel is $\mathbb{E}w ( k^{(B)}(w^\top x,w^\top x^\prime))$, with $k^{(B)}(a,b) := \min(|a|, |b|)1{ab>0}$ the Brownian kernel, and the distribution of the projections $w$ is learnt. This can also be viewed as an infinite-width one-hidden layer neural network, optimising the first layer's weights through gradient descent and explicitly adjusting the non-linearity and weights of the second layer. We introduce an efficient computation method for the estimator, called Brownian Kernel Neural Network (BKerNN), using particles to approximate the expectation. The optimisation is principled due to the positive homogeneity of the Brownian kernel. Using Rademacher complexity, we show that BKerNN's expected risk converges to the minimal risk with explicit high-probability rates of $O( \min((d/n)^{1/2}, n^{-1/6}))$ (up to logarithmic factors). Numerical experiments confirm our optimisation intuitions, and BKerNN outperforms kernel ridge regression, and favourably compares to a one-hidden layer neural network with ReLU activations in various settings and real data sets.

• The paper presents the Brownian Kernel Neural Network (BKerNN), which unifies neural nets and kernel methods under a joint regularised risk minimisation framework with provable risk bounds.
• It introduces a novel function space inspired by infinite-width neural networks, expanding the set of learnable functions beyond traditional ReLU activations through Sobolev norms.
• It achieves computational efficiency using particle-based approximations and gradient descent with proximal steps, validated by experimental results on synthetic and real datasets.

Enhanced Feature Learning via Regularisation: Integrating Neural Networks and Kernel Methods

The article "Enhanced Feature Learning via Regularisation: Integrating Neural Networks and Kernel Methods" by Bertille Follain and Francis Bach introduces a novel approach to feature learning and function estimation in supervised learning tasks. The authors propose a method that combines neural networks and kernel methods using regularised empirical risk minimisation, positioning their framework to leverage the strengths of both learning paradigms.

Summary of Contributions

The primary contribution of this paper is the development and analysis of the Brownian Kernel Neural Network (BKerNN). The method is based on regularised empirical risk minimisation, where functions are considered as expectations of Sobolev functions over one-dimensional projections of the data. Key innovations include:

1. Function Space Definition: The paper introduces a custom space of functions inspired by infinite-width single hidden layer neural networks. Functions are represented as

$$f(x) = c + \int_{\mathcal{S}^{d-1}} g_w(w^\top x) \, \mathrm{d}\mu(w),$$

with $g_w$ in a Sobolev space $\mathcal{H}$. This space is shown to subsume larger function spaces compared to neural networks with ReLU activations.

1. Optimization Framework: The authors formulate an optimisation problem that jointly learns features and the function by regularising both the empirical risk and the complexity of the function, expressed in terms of the Sobolev norm.
2. Efficient Computation: To compute the estimator efficiently, the method uses particles to approximate the expectation, allowing a tractable optimization process. The methods for optimisation include gradient descent with proximal steps, leveraging the homogeneity properties of the Brownian kernel.
3. Theoretical Analysis: Using Rademacher complexity, the paper provides bounds on the expected risk of the BKerNN estimator, demonstrating convergence to the minimal risk. The convergence rates are $O(\min((d/n)^{1/2}, n^{-1/6}))$ up to logarithmic factors, showing robustness in high-dimensional settings but also highlighting scenarios where the curse of dimensionality is mitigated.
4. Experimental Validation: Numerical experiments confirm the efficiency and effectiveness of BKerNN, indicating superior performance over traditional kernel ridge regression and one-hidden layer neural networks with ReLU activations on various synthetic and real-world datasets.

Implications of Research

Practical Implications

1. Improved Feature Learning: BKerNN provides a more robust framework for feature learning, capable of performing well even when the underlying data includes high-dimensional manifolds or latent structures. This has practical applications in fields such as genomics, image processing, and natural language processing where feature selection and dimensionality reduction are paramount.
2. Smooth Transition Between Kernels and Neural Networks: By integrating kernel methods with neural network architectures, the paper offers a hybrid model that can be tailored to specific problem characteristics, leading to potential improvements in generalization performance across a variety of tasks.
3. Computational Efficiency: The use of particles to approximate the function expectation helps maintain computational tractability, making it feasible to apply BKerNN to large-scale datasets without prohibitive computational overhead.

Theoretical Implications

1. Extension of Function Spaces: The paper extends the class of functions that can be efficiently learned, adding to the theoretical understanding of how kernels and neural networks can be unified within a single framework. The comparison with traditional RKHS and neural network spaces underscores the flexibility and power of the proposed methods.
2. Risk Convergence and Complexity Bounds: Providing explicit bounds on the risk and showing the conditions under which BKerNN achieves these bounds contribute valuable insights into the behavior of regularised empirical risk minimisation techniques in high-dimensional spaces.

Future Developments

Several avenues for future work can be envisioned from this research:

1. Improved Algorithms for Optimization: Developing more advanced optimization algorithms tailored to the structure of BKerNN could further enhance the convergence rates and scalability of the method. This could include stochastic gradient techniques, adaptive learning rates, or variational optimization approaches.
2. Extension to Deep Architectures: Exploring how multi-layer architectures can be built following similar principles to BKerNN could lead to further improvements in learning complex hierarchical representations.
3. Application to Unsupervised and Semi-Supervised Learning: Extending the regularisation techniques to unsupervised or semi-supervised learning settings would increase their applicability, addressing scenarios where labelled data is scarce.
4. Data-Adaptive Kernels: Investigating how the choice of kernels beyond the Brownian kernel can be made data-adaptive to further tailor the model to specific datasets or learning tasks.

By bridging the methodological gap between kernel methods and neural network approaches, this paper lays a robust foundation for future enhancements in feature learning and function estimation, addressing both theoretical insights and practical needs in machine learning.