---
title: Heaviside & Sigmoid Expansions
url: https://www.emergentmind.com/topics/heaviside-and-sigmoid-expansions
type: topic
---

# Heaviside & Sigmoid Expansions

Heaviside and sigmoid expansions are families of representations in which discontinuous step functions \(H\) or smooth step-like surrogates are used as elementary building blocks for function decomposition, approximation, optimization, control, and neural computation. In current usage, the term spans exact Heaviside-based integral representations of nonlinear multivariate functions, smooth logistic and hyperbolic-tangent approximations that converge to steps under steepness limits, and composite constructions that preserve differentiability while driving solutions toward binary states [1404.4338][2102.07156][2605.11558]. Across these settings, the central issue is the same: Heaviside functions encode crisp on/off structure, but sigmoid-like expansions make that structure analytically or computationally tractable.

## 1. Definitions, conventions, and basic forms

The Heaviside step function is not used with a single universal convention at the origin. That variation is mathematically consequential in distribution theory, optimization, and learning, because formulas involving derivatives, Fourier transforms, and threshold inclusions depend on whether the threshold point is assigned to the active or inactive side.

| Source | Convention at \(0\) | Context |
|---|---:|---|
| [2605.22193], [2411.05800] | \(H(0)=1\) | tempered distributions; recursive step sequences |
| [2401.17936] | \(H(0)=\tfrac12\) | pointwise limit of logistic \(\sigma_\varepsilon\) |
| [2007.15737] | \(h(0)=0\) | Heaviside-set constrained optimization |

In the tempered-distribution treatment, the Heaviside function is defined by
\[
H(x)=
\begin{cases}
0,&x<0,\\
1,&x\ge 0,
\end{cases}
\]
with the related sign function satisfying \(H(x)=\tfrac12(1+\operatorname{sgn}(x))\) under that convention [2605.22193]. In the infinite-delay neural-network setting, the pointwise Heaviside limit of the logistic
\[
\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}
\]
is instead written as
\[
H(x)=
\begin{cases}
0,&x<0,\\
\tfrac12,&x=0,\\
1,&x>0,
\end{cases}
\]
while the actual discontinuous model is formulated by the set-valued graph
\[
\chi(s)=
\begin{cases}
\{0\},&s<0,\\
[0,1],&s=0,\\
\{1\},&s>0,
\end{cases}
\]
so that threshold ambiguity is treated rigorously as a differential inclusion [2401.17936]. In Heaviside set constrained optimization, the step is encoded through
\[
h(t)=\frac{\operatorname{sgn}(t)+1}{2}
=
\begin{cases}
1,&t>0,\\
0,&t\le 0,
\end{cases}
\]
which makes the count of positive coordinates equal to a sum of Heaviside terms [2007.15737].

Sigmoid expansions appear whenever the sharp jump is replaced by a smooth \(S\)-curve. The most common form is logistic:
\[
\sigma_\beta(x)=\frac{1}{1+e^{-\beta x}},
\]
with \(\beta\) controlling transition width. A related logistic-type “Heaviside sequence function” is reconstructed in one recursive-sequence model as \(H_N(t)\approx 1/(1+e^{-2Nt})\), with derivative
\[
\delta_N(t)=\frac{2N e^{-2Nt}}{(1+e^{-2Nt})^2},
\]
so that \(H_N\to H\) and \(\delta_N\to\delta\) as \(N\to\infty\) [2411.05800]. Another conceptual equivalence arises after the logarithmic change of variables \(s=\log x\), under which rational partial fractions
\[
\frac{1}{1+x/\epsilon_k}
\]
become translated logistics
\[
\frac{1}{1+e^{s-s_k}},\qquad s_k=\log\epsilon_k,
\]
leading to the mnemonic “sigmoid = log(ratapprox)” [2303.01967].

A frequent misconception is that the only issue in moving from Heaviside to sigmoid is differentiability. The cited literature shows that the choice of convention at \(0\), the asymmetry of the approximation, the behavior of derivatives in saturated regions, and the treatment of threshold ambiguity are all structurally important, not merely notational.

## 2. Exact Heaviside representations and transform identities

One major meaning of Heaviside expansion is exact representation of a function in terms of derivatives weighted by Heaviside steps. Starting from Dirac’s single-variable identity, a nonlinear multivariate function \(R(X_1,\dots,X_N)\) on \(X_i\ge 0\) can be decomposed as a sum over all index subsets \(S\subseteq\{1,\dots,N\}\):
\[
R(X_1,\dots,X_N)
=
\sum_{S\subseteq\{1,\dots,N\}}
\int_{[0,\infty)^{|S|}}
\frac{\partial^{|S|}R}{\prod_{i\in S}\partial u_i}
\bigl(\mathbf u_S;\mathbf 0_{S^c}\bigr)
\prod_{i\in S}H(X_i-u_i)\,d\mathbf u_S.
\]
The \(S=\emptyset\) term is \(R(0,\dots,0)\), first-order terms integrate partial derivatives along coordinate axes, and higher-order terms integrate mixed partial derivatives over hyperrectangles selected by products of Heaviside factors [1404.4338]. This representation is mathematically equivalent to the standard multivariate Dirac-delta formula and is obtained by repeated distributional integration by parts. It is therefore an exact Heaviside expansion, not merely an approximation.

The same source explicitly notes that these Heaviside products can be replaced by sigmoids with suitable parameters, yielding smooth approximations of the original function. A representative form is
\[
R(X_1,\dots,X_N)
\approx
\sum_{S\subseteq\{1,\dots,N\}}
\int_{[0,\infty)^{|S|}}
\frac{\partial^{|S|}R}{\prod_{i\in S}\partial u_i}
(\mathbf u_S;\mathbf 0_{S^c})
\prod_{i\in S}\sigma_k(X_i-u_i)\,d\mathbf u_S,
\]
with \(\sigma_k\to H\) as \(k\to\infty\) [1404.4338]. This suggests a direct bridge from exact step-function expansions to smooth neural-network-style expansions.

A second exact representation concerns frequency analysis. In tempered-distribution form, the Fourier transform of the Heaviside function is
\[
F H
=
-i\,\mathrm{p.v.}\!\left(\frac{1}{x}\right)+\pi\,\delta_0,
\]
under the convention \(F\varphi(x)=\int_{-\infty}^{+\infty} e^{-ixy}\varphi(y)\,dy\) [2605.22193]. The delta term records the DC contribution, whereas the principal-value term records the jump singularity. The same analysis yields explicit formulas such as
\[
\left\langle \mathrm{p.v.}\!\left(\frac{1}{x}\right),\varphi\right\rangle
=
\int_0^\infty \frac{\varphi(x)-\varphi(-x)}{x}\,dx
=
\int_{-\infty}^{\infty}\frac{\varphi(x)-\varphi(0)}{x}\,dx,
\]
which make the singular kernel rigorous by Taylor subtraction rather than heuristic cancellation [2605.22193].

From that transform identity follows a standard Heaviside expansion of step-discontinuous signals:
\[
f(x)=c_0+\sum_k a_k H(x-x_k),
\]
with Fourier transform
\[
Ff(\omega)
=
2\pi c_0\,\delta(\omega)
+
\sum_k a_k e^{-i\omega x_k}
\left[
\pi\delta(\omega)-i\,\mathrm{p.v.}\!\left(\frac{1}{\omega}\right)
\right].
\]
This decomposition separates constant background from jump structure in the frequency domain [2605.22193]. It also clarifies why steep sigmoid approximations inherit a transform that converges, in \(S'(\mathbb R)\), to the same delta-plus-principal-value template.

## 3. Smooth Heaviside approximations and continuation schemes

A large applied literature uses sigmoid expansions not merely to approximate a step but to embed binary decisions in gradient-based optimization. In structured pruning, ChipNet formulates channel selection through mask variables \(z_i\in\{0,1\}\) and replaces them by a differentiable two-stage map. First, an optimization parameter \(\psi_i\) is passed through a logistic projection
\[
\tilde z_i=\frac{1}{1+e^{-\beta(\psi_i-\psi_0)}},
\]
and then through a continuous Heaviside projection
\[
z_i = 1 - e^{-\gamma \tilde z_i + \tilde z_i e^{-\gamma}},
\]
followed by a logistic rounding for budget accounting,
\[
\bar z_i = \frac{1}{1+e^{-\beta(z_i-0.5)}}.
\]
The total pruning objective combines cross-entropy, a crispness loss
\[
\mathcal L_c=\|\tilde{\mathbf z}-\mathbf z\|_2^2,
\]
and a quadratic budget penalty \(\mathcal L_b=(\mathcal V(\bar{\mathbf z})-\mathcal V_0)^2\), where \(\mathcal V\) may encode channels, volume, parameters, or FLOPs [2102.07156]. The paper states that logistic projection alone does not sufficiently penalize intermediate values, whereas the continuous Heaviside plus crispness loss drives masks toward near-binary endpoints while remaining fully differentiable. It also reports stability under very low target budgets, improvements of up to \(16.1\%\) in accuracy over structured pruning baselines, and transferability of learned masks across datasets [2102.07156].

The same theme reappears in a newer activation design, the Heavy Tailed Activation Function,
\[
\mathrm{HTAF}(x\mid \alpha_0,\alpha_1)=\sigma(\alpha_1\tanh(\alpha_0 x)).
\]
Here the Heaviside limit is approached by controlling two parameters separately. The derivative at the origin is
\[
\mathrm{HTAF}'(0)=\frac{\alpha_0\alpha_1}{4},
\]
while the tail decay obeys
\[
\mathrm{HTAF}'(x)=C(\alpha_0,\alpha_1)e^{-2\alpha_0|x|}
\quad\text{for large }|x|.
\]
The paper proves uniform approximation bounds outside an \(\epsilon\)-neighborhood of the threshold and argues that the two-parameter form decouples central steepness from tail decay, thereby preserving larger gradient mass near zero while slowing gradient collapse in saturated regions [2605.11558]. This construction is then used to train spiking neural networks, binary neural networks, deep Heaviside networks, and implicit concept bottleneck models with exact Heaviside activations restored at inference time [2605.11558].

A recurrent technical pattern is continuation: start with a soft or broad sigmoid, then gradually sharpen it. In ChipNet this is expressed by annealing \(\beta\) and \(\gamma\); in HTAF it is expressed by tuning \((\alpha_0,\alpha_1)\); in recursive step-sequence models it appears as increasing \(N\) in \(H_N\) and \(\delta_N\). This suggests a general principle: useful Heaviside approximations are rarely defined only by pointwise convergence to a step. Their derivative profile, continuation schedule, and compatibility with downstream constraints are equally central.

## 4. Constrained optimization and control

Heaviside expansions also arise as exact formulations of discrete or cardinality-type constraints. In Heaviside set constrained optimization, the feasible set is
\[
S=\{y\in\mathbb R^m:\|y_+\|_0\le s\},
\]
and the optimization problem is
\[
\min_{x\in\mathbb R^n} f(x)\qquad\text{s.t.}\qquad \|(Ax-b)_+\|_0\le s.
\]
Because \(\|z_+\|_0\) equals a sum of Heaviside terms, the constraint counts how many affine components are positive [2007.15737]. The paper computes the Bouligand tangent cone \(T_S(y)\), the Fréchet normal cone \(\widehat N_S(y)\), projection formulas \(\Pi_S(z)\), KKT conditions, and projection-based \(\tau\)-stationarity conditions. It then derives a Newton-type method based on the system
\[
\nabla f(x)+A^\top\lambda=0,\qquad y=Ax-b\in \Pi_S(y+\tau\lambda),
\]
and proves local quadratic convergence under full-row-rank and positive-definite-Hessian assumptions [2007.15737]. The central methodological point is that the Heaviside discontinuity is handled directly through set geometry, not by replacing it with a smooth sigmoid surrogate.

A closely related framework studies affine Heaviside composite optimization problems of the form
\[
\max_{x\in P}\;
c(x)+\sum_{j=1}^{J_0}\psi_{0j}\,1_{[0,\infty)}(\phi_{0j}(x))
\]
subject to affine combinations of Heaviside composites in the constraints. Here the coefficients \(\psi_{ij}\) may have mixed signs, the inner functions \(\phi_{ij}\) are dc piecewise affine, and the resulting objective need not be upper semicontinuous while the feasible set need not be closed [2605.06020]. The proposed remedy is an \(\varepsilon\)-approximation that leaves positive-coefficient terms as \(1_{[0,\infty)}\) but replaces negative-coefficient terms by \(1_{(-\varepsilon,\infty)}\), restoring upper semicontinuity and closedness sufficiently for analysis. The resulting problems are solved by a progressive integer programming method with successive decomposition, and convergence to local optimizers of the original Heaviside problem is established [2605.06020].

In time-optimal control, the role of a Heaviside expansion is to encode the discrete switching time. For a discrete index \(i\) and a real switching variable \(N^*\), the approximation
\[
h(i,N^*)=0.5+0.5\,\tanh\big(k(i-N^*)\big)
\]
is used together with
\[
w(i,N^*)=h(i,N^*)(i-N^*+1)^k.
\]
The added polynomial factor is introduced specifically to avoid vanishing gradients with respect to \(N^*\), because the derivative of the tanh sigmoid alone becomes negligible away from the switching region [2303.04516]. Embedded in a nonlinear hierarchical least-squares program with piecewise-constant discretization of states and controls, this weighting yields a comparatively implementable approximation of discrete time-optimal control, and the paper proves recovery of the discrete time-optimal solution in the limit for resting goal points [2303.04516].

A common misconception is that replacing a Heaviside by a sigmoid automatically resolves optimization difficulties. The cited work shows three distinct possibilities: exact nonsmooth treatment through tangent and normal cones [2007.15737], discontinuous but regularized interval shifts [2605.06020], and smooth sigmoid surrogates augmented to preserve useful gradients [2303.04516]. Each addresses a different failure mode.

## 5. Neural networks, binary activations, and dynamical systems

In neural-network dynamics, the Heaviside–sigmoid relation often takes the form of a singular limit. One nonautonomous neural network with infinite delay uses the logistic activation
\[
\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}
\]
in a functional differential equation and studies the limit \(\varepsilon\to 0\), where the model becomes a Heaviside differential inclusion built from the set-valued graph \(\chi\) and Aumann delay integrals [2401.17936]. Under coercivity, continuity, and delay-integrability assumptions, the paper proves global existence for the sigmoidal systems, existence of solutions for the Heaviside inclusion, uniform convergence of sigmoidal solutions to Heaviside-inclusion solutions on finite intervals, existence of pullback attractors in both settings, and convergence of the sigmoidal attractors to the Heaviside attractor [2401.17936]. The result is not merely pointwise convergence of activations; it is convergence of long-time nonautonomous dynamics in an infinite-dimensional phase space.

At the level of representation capacity, deep Heaviside networks are formally defined by hidden activations
\[
\sigma_0(x)=\mathbb I(x\ge 0)
\]
and therefore produce piecewise-constant functions. For plain DHNs, restriction to any line segment yields at most \(p_1+1\) constant pieces, where \(p_1\) is the first hidden-layer width; depth alone does not increase that one-dimensional complexity [2505.00110]. The paper then shows that skip connections or linear neurons fundamentally change the picture. Skip-DHNs achieve linewise complexity growing like \((p_1+1)\prod_{\ell=2}^L(s_\ell+1)\), have VC dimension of order \(Lp^2\) up to logarithmic factors in rectangular architectures, and attain approximation rates
\[
\left(\frac{\log^3(Lp)}{Lp^2}\right)^{\beta/d}
\]
for \(\beta\)-Hölder targets. Lin-DHNs reach VC dimension of order \(L^2ps\vee Lp^2\) and approximation rates
\[
\left(\frac{\log^3(Lp)}{L^2ps\vee Lp^2}\right)^{\beta/d},
\]
which match lower bounds up to logarithmic factors and yield minimax-optimal nonparametric regression rates after suitable scaling of architecture size [2505.00110]. The paper’s main conclusion is that plain DHNs are expressively limited, but those limits can be overcome without abandoning Heaviside activations entirely.

A separate line of work uses recursive compositions of logistic-type Heaviside sequence functions to model layered state transitions. One proposed recursive form is
\[
u(t)=1-H_{0.1}\bigl(-t+T_1 H_1(-t+T_2 H_2(-t+T_3 H_5(-t+T_4)))\bigr),
\]
with event times \(T_1\le T_2\le T_3\le T_4\). In that framework, \(H_N\) is a smooth logistic approximation of Heaviside, \(\delta_N=H_N'\) is a delta sequence, and the recursive compositions are interpreted as smooth approximations of piecewise-constant mental-state potentials whose \(N\to\infty\) limits are exact solutions of a multidimensional inviscid advection equation [2411.05800]. The application is unconventional, but the mathematical structure is characteristic: nested Heaviside or sigmoid expansions generate multistage switching behavior, while derivatives of those expansions generate spike-like activation events.

Taken together, these results support a broader interpretation. Heaviside-based architectures are not just limiting cases of smooth networks; they are a separate representation class whose deficiencies and advantages depend strongly on whether the model includes skip pathways, analog subchannels, or set-valued threshold formulations.

## 6. Multiscale and algebraic theories of sigmoid expansions

A different viewpoint treats sigmoid expansions as transformed versions of rational or orthogonal expansions. For functions with singularities at \(x=0\), rational approximants of the form
\[
r_n(x)=a_0+\sum_{k=1}^n \frac{a_k}{1+x/\epsilon_k}
\]
achieve root-exponential error
\[
\|f-r_n\|_{[0,1]}=O\bigl(e^{-C\sqrt n}\bigr).
\]
After the logarithmic change of variables \(s=\log x\), the elementary factors become translated logistics:
\[
\frac{1}{1+x/\epsilon_k}=\frac{1}{1+e^{s-s_k}},\qquad s_k=\log\epsilon_k,
\]
so that \(r_n(e^s)\) is a linear combination of sigmoids plus a constant [2303.01967]. Exponential clustering of poles near the singularity becomes approximately uniform spacing of sigmoid centers in \(s\), and the paper interprets this as a multiscale resolution principle shared by rational approximation, sigmoidal approximation, and \(hp\)-mesh refinement for PDEs. It emphasizes the balancing law behind the root-exponential rate: grid extent \(S_{\min}=O(\sqrt n)\) versus spacing \(\Delta s=O(1/\sqrt n)\) [2303.01967].

An algebraic theory places sigmoids inside the derivative subgroup of exponential Riordan arrays. For an analytic sigmoid \(f\), the array
\[
[f'(x),f(x)]
\]
belongs to that subgroup, with multiplication corresponding to composition by the chain rule [1702.04778]. Examples include
\[
[\operatorname{sech}^2(x),\tanh(x)],\qquad
\left[\frac{1}{1+x^2},\tan^{-1}(x)\right],\qquad
[e^{-x^2},\sqrt\pi\,\operatorname{erf}(x)],
\]
as well as Gudermannian and Gompertz variants [1702.04778]. The associated production matrix determines whether the inverse array is the coefficient array of an orthogonal polynomial family: tri-diagonal production matrices yield orthogonal systems and moment interpretations for the sigmoid derivative. In this sense, a sigmoid expansion can be studied either as a neural-style superposition of smoothed steps or as a structured generating-function object linked to recurrence relations, moment sequences, and Hankel transforms [1702.04778].

This algebraic and multiscale literature suggests a unifying interpretation. Heaviside expansions emphasize discrete region selection; sigmoid expansions regularize that selection; rational, orthogonal, and mesh-based theories explain where the placement and scaling of those transitions should come from. The modern applied constructions in pruning, control, and binary deep learning can therefore be read as specialized descendants of a broader approximation-theoretic idea: represent complexity by accumulating thresholded transitions across location, scale, or hierarchy.

Source: https://www.emergentmind.com/topics/heaviside-and-sigmoid-expansions