Bias-learning with discontinuous activations (e.g., Heaviside)

Investigate the applicability of the bias-learning framework to discontinuous activation functions, specifically the Heaviside step function, by determining whether neural networks with fixed random weights and learned biases retain universal approximation guarantees when the activation is discontinuous and does not satisfy the continuity assumption in the definition of γ-bias-learning activations.

Background

The bias-learning framework requires a γ-parameter bounding activation that is continuous and has a threshold τ such that φ(x)=0 for x<τ; ReLU satisfies this, whereas Heaviside is discontinuous. The authors explicitly flag the discontinuous case as not addressed and left to future work.

Clarifying whether discontinuous activations like Heaviside can support bias-only learning under random weights would extend the theoretical scope and potentially link to classical threshold-based models in neuroscience and machine learning.