Conjecture on fundamental-solution activations ensuring representability and convergence

Establish that one-hidden-layer neural networks with fixed first-layer biases and activation functions that are fundamental solutions of second-order differential operators possess good representability and convergence properties under gradient-descent training with the L2 loss.

Background

The paper analyzes a simplified one-hidden-layer network with fixed biases, showing for ReLU that representability and convergence of gradient descent follow from the fact that ReLU is a fundamental solution of the one-dimensional Laplacian. This motivates examining activation functions tied to second-order operators.

Based on these observations, the authors introduce FReX (e{-|x|}), which is the fundamental solution of (1/2)(-d2/dx2 + 1), and prove analogous convergence results for this activation in both continuous and discrete settings. These findings motivate the conjecture that the fundamental-solution property more generally underpins desirable representability and convergence behavior.

References

These remarks, which are elaborated in Section~4, allow us to conjecture that activation functions with the property of being fundamental solutions to a second-order differential operator should give rise to networks which possess good representability and convergence properties.

— Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations  (2604.07715 - Macià et al., 9 Apr 2026) in Introduction

Why a bounded two-operand unit is at ease with a geometry a GELU unit is not, and whether that reading is the whole of it, are not settled in this paper;

— Through the Looking Glass: Directly Reading and Writing Transformers  (2609.10210 - Oskin, 9 Sep 2026) in Appendix “The set-operator model and how it is trained,” Section “The pressure”