Conjecture on fundamental-solution activations ensuring representability and convergence

Establish that one-hidden-layer neural networks with fixed first-layer biases and activation functions that are fundamental solutions of second-order differential operators possess good representability and convergence properties under gradient-descent training with the L2 loss.

Background

The paper analyzes a simplified one-hidden-layer network with fixed biases, showing for ReLU that representability and convergence of gradient descent follow from the fact that ReLU is a fundamental solution of the one-dimensional Laplacian. This motivates examining activation functions tied to second-order operators.

Based on these observations, the authors introduce FReX (e^{-|x|}), which is the fundamental solution of (1/2)(-d^2/dx^2 + 1), and prove analogous convergence results for this activation in both continuous and discrete settings. These findings motivate the conjecture that the fundamental-solution property more generally underpins desirable representability and convergence behavior.