Heaviside & Sigmoid Expansions

• Heaviside and Sigmoid Expansions are representation families that use discontinuous step functions or smooth surrogates to decompose complex functions, incorporating analytic tractability and binary encoding.
• They provide exact integral representations and transform identities, linking sharp on/off behavior with smooth approximations and ensuring rigorous treatments in optimization and control.
• Applications span neural network pruning, deep binary activations, and control systems where continuation schemes and threshold handling preserve differentiability while enforcing crisp decisions.

Heaviside and sigmoid expansions are families of representations in which discontinuous step functions $H$ or smooth step-like surrogates are used as elementary building blocks for function decomposition, approximation, optimization, control, and neural computation. In current usage, the term spans exact Heaviside-based integral representations of nonlinear multivariate functions, smooth logistic and hyperbolic-tangent approximations that converge to steps under steepness limits, and composite constructions that preserve differentiability while driving solutions toward binary states (Chikayama, 2014, Tiwari et al., 2021, Park et al., 12 May 2026). Across these settings, the central issue is the same: Heaviside functions encode crisp on/off structure, but sigmoid-like expansions make that structure analytically or computationally tractable.

1. Definitions, conventions, and basic forms

The Heaviside step function is not used with a single universal convention at the origin. That variation is mathematically consequential in distribution theory, optimization, and learning, because formulas involving derivatives, Fourier transforms, and threshold inclusions depend on whether the threshold point is assigned to the active or inactive side.

Source	Convention at $0$	Context
(Dalang, 21 May 2026, Shin, 2024)	$H(0)=1$	tempered distributions; recursive step sequences
(Kloeden et al., 2024)	$H(0)=\tfrac12$	pointwise limit of logistic $\sigma_\varepsilon$
(Zhou et al., 2020)	$h(0)=0$	Heaviside-set constrained optimization

In the tempered-distribution treatment, the Heaviside function is defined by

$$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$

with the related sign function satisfying $H(x)=\tfrac12(1+\operatorname{sgn}(x))$ under that convention (Dalang, 21 May 2026). In the infinite-delay neural-network setting, the pointwise Heaviside limit of the logistic

$$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$

is instead written as

$$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$

while the actual discontinuous model is formulated by the set-valued graph

$0$0

so that threshold ambiguity is treated rigorously as a differential inclusion (Kloeden et al., 2024). In Heaviside set constrained optimization, the step is encoded through

$0$1

which makes the count of positive coordinates equal to a sum of Heaviside terms (Zhou et al., 2020).

Sigmoid expansions appear whenever the sharp jump is replaced by a smooth $0$2-curve. The most common form is logistic: $0$3 with $0$4 controlling transition width. A related logistic-type “Heaviside sequence function” is reconstructed in one recursive-sequence model as $0$5, with derivative

$0$6

so that $0$7 and $0$8 as $0$9 (Shin, 2024). Another conceptual equivalence arises after the logarithmic change of variables $H(0)=1$0, under which rational partial fractions

$H(0)=1$1

become translated logistics

$H(0)=1$2

leading to the mnemonic “sigmoid = log(ratapprox)” (Huybrechs et al., 2023).

A frequent misconception is that the only issue in moving from Heaviside to sigmoid is differentiability. The cited literature shows that the choice of convention at $H(0)=1$3, the asymmetry of the approximation, the behavior of derivatives in saturated regions, and the treatment of threshold ambiguity are all structurally important, not merely notational.

2. Exact Heaviside representations and transform identities

One major meaning of Heaviside expansion is exact representation of a function in terms of derivatives weighted by Heaviside steps. Starting from Dirac’s single-variable identity, a nonlinear multivariate function $H(0)=1$4 on $H(0)=1$5 can be decomposed as a sum over all index subsets $H(0)=1$6: $H(0)=1$7 The $H(0)=1$8 term is $H(0)=1$9, first-order terms integrate partial derivatives along coordinate axes, and higher-order terms integrate mixed partial derivatives over hyperrectangles selected by products of Heaviside factors (Chikayama, 2014). This representation is mathematically equivalent to the standard multivariate Dirac-delta formula and is obtained by repeated distributional integration by parts. It is therefore an exact Heaviside expansion, not merely an approximation.

The same source explicitly notes that these Heaviside products can be replaced by sigmoids with suitable parameters, yielding smooth approximations of the original function. A representative form is

$H(0)=\tfrac12$0

with $H(0)=\tfrac12$1 as $H(0)=\tfrac12$2 (Chikayama, 2014). This suggests a direct bridge from exact step-function expansions to smooth neural-network-style expansions.

A second exact representation concerns frequency analysis. In tempered-distribution form, the Fourier transform of the Heaviside function is

$H(0)=\tfrac12$3

under the convention $H(0)=\tfrac12$4 (Dalang, 21 May 2026). The delta term records the DC contribution, whereas the principal-value term records the jump singularity. The same analysis yields explicit formulas such as

$H(0)=\tfrac12$5

which make the singular kernel rigorous by Taylor subtraction rather than heuristic cancellation (Dalang, 21 May 2026).

From that transform identity follows a standard Heaviside expansion of step-discontinuous signals: $H(0)=\tfrac12$6 with Fourier transform

$H(0)=\tfrac12$7

This decomposition separates constant background from jump structure in the frequency domain (Dalang, 21 May 2026). It also clarifies why steep sigmoid approximations inherit a transform that converges, in $H(0)=\tfrac12$8, to the same delta-plus-principal-value template.

3. Smooth Heaviside approximations and continuation schemes

A large applied literature uses sigmoid expansions not merely to approximate a step but to embed binary decisions in gradient-based optimization. In structured pruning, ChipNet formulates channel selection through mask variables $H(0)=\tfrac12$9 and replaces them by a differentiable two-stage map. First, an optimization parameter $\sigma_\varepsilon$0 is passed through a logistic projection

$\sigma_\varepsilon$1

and then through a continuous Heaviside projection

$\sigma_\varepsilon$2

followed by a logistic rounding for budget accounting,

$\sigma_\varepsilon$3

The total pruning objective combines cross-entropy, a crispness loss

$\sigma_\varepsilon$4

and a quadratic budget penalty $\sigma_\varepsilon$5, where $\sigma_\varepsilon$6 may encode channels, volume, parameters, or FLOPs (Tiwari et al., 2021). The paper states that logistic projection alone does not sufficiently penalize intermediate values, whereas the continuous Heaviside plus crispness loss drives masks toward near-binary endpoints while remaining fully differentiable. It also reports stability under very low target budgets, improvements of up to $\sigma_\varepsilon$7 in accuracy over structured pruning baselines, and transferability of learned masks across datasets (Tiwari et al., 2021).

The same theme reappears in a newer activation design, the Heavy Tailed Activation Function,

$\sigma_\varepsilon$8

Here the Heaviside limit is approached by controlling two parameters separately. The derivative at the origin is

$\sigma_\varepsilon$9

while the tail decay obeys

$h(0)=0$0

The paper proves uniform approximation bounds outside an $h(0)=0$1-neighborhood of the threshold and argues that the two-parameter form decouples central steepness from tail decay, thereby preserving larger gradient mass near zero while slowing gradient collapse in saturated regions (Park et al., 12 May 2026). This construction is then used to train spiking neural networks, binary neural networks, deep Heaviside networks, and implicit concept bottleneck models with exact Heaviside activations restored at inference time (Park et al., 12 May 2026).

A recurrent technical pattern is continuation: start with a soft or broad sigmoid, then gradually sharpen it. In ChipNet this is expressed by annealing $h(0)=0$2 and $h(0)=0$3; in HTAF it is expressed by tuning $h(0)=0$4; in recursive step-sequence models it appears as increasing $h(0)=0$5 in $h(0)=0$6 and $h(0)=0$7. This suggests a general principle: useful Heaviside approximations are rarely defined only by pointwise convergence to a step. Their derivative profile, continuation schedule, and compatibility with downstream constraints are equally central.

4. Constrained optimization and control

Heaviside expansions also arise as exact formulations of discrete or cardinality-type constraints. In Heaviside set constrained optimization, the feasible set is

$h(0)=0$8

and the optimization problem is

$h(0)=0$9

Because $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$0 equals a sum of Heaviside terms, the constraint counts how many affine components are positive (Zhou et al., 2020). The paper computes the Bouligand tangent cone $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$1, the Fréchet normal cone $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$2, projection formulas $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$3, KKT conditions, and projection-based $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$4-stationarity conditions. It then derives a Newton-type method based on the system

$$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$5

and proves local quadratic convergence under full-row-rank and positive-definite-Hessian assumptions (Zhou et al., 2020). The central methodological point is that the Heaviside discontinuity is handled directly through set geometry, not by replacing it with a smooth sigmoid surrogate.

A closely related framework studies affine Heaviside composite optimization problems of the form

$$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$6

subject to affine combinations of Heaviside composites in the constraints. Here the coefficients $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$7 may have mixed signs, the inner functions $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$8 are dc piecewise affine, and the resulting objective need not be upper semicontinuous while the feasible set need not be closed (Zheng et al., 7 May 2026). The proposed remedy is an $$H(x)= \begin{cases} 0,&x<0,\ 1,&x\ge 0, \end{cases}$$9-approximation that leaves positive-coefficient terms as $H(x)=\tfrac12(1+\operatorname{sgn}(x))$0 but replaces negative-coefficient terms by $H(x)=\tfrac12(1+\operatorname{sgn}(x))$1, restoring upper semicontinuity and closedness sufficiently for analysis. The resulting problems are solved by a progressive integer programming method with successive decomposition, and convergence to local optimizers of the original Heaviside problem is established (Zheng et al., 7 May 2026).

In time-optimal control, the role of a Heaviside expansion is to encode the discrete switching time. For a discrete index $H(x)=\tfrac12(1+\operatorname{sgn}(x))$2 and a real switching variable $H(x)=\tfrac12(1+\operatorname{sgn}(x))$3, the approximation

$H(x)=\tfrac12(1+\operatorname{sgn}(x))$4

is used together with

$H(x)=\tfrac12(1+\operatorname{sgn}(x))$5

The added polynomial factor is introduced specifically to avoid vanishing gradients with respect to $H(x)=\tfrac12(1+\operatorname{sgn}(x))$6, because the derivative of the tanh sigmoid alone becomes negligible away from the switching region (Pfeiffer et al., 2023). Embedded in a nonlinear hierarchical least-squares program with piecewise-constant discretization of states and controls, this weighting yields a comparatively implementable approximation of discrete time-optimal control, and the paper proves recovery of the discrete time-optimal solution in the limit for resting goal points (Pfeiffer et al., 2023).

A common misconception is that replacing a Heaviside by a sigmoid automatically resolves optimization difficulties. The cited work shows three distinct possibilities: exact nonsmooth treatment through tangent and normal cones (Zhou et al., 2020), discontinuous but regularized interval shifts (Zheng et al., 7 May 2026), and smooth sigmoid surrogates augmented to preserve useful gradients (Pfeiffer et al., 2023). Each addresses a different failure mode.

5. Neural networks, binary activations, and dynamical systems

In neural-network dynamics, the Heaviside–sigmoid relation often takes the form of a singular limit. One nonautonomous neural network with infinite delay uses the logistic activation

$H(x)=\tfrac12(1+\operatorname{sgn}(x))$7

in a functional differential equation and studies the limit $H(x)=\tfrac12(1+\operatorname{sgn}(x))$8, where the model becomes a Heaviside differential inclusion built from the set-valued graph $H(x)=\tfrac12(1+\operatorname{sgn}(x))$9 and Aumann delay integrals (Kloeden et al., 2024). Under coercivity, continuity, and delay-integrability assumptions, the paper proves global existence for the sigmoidal systems, existence of solutions for the Heaviside inclusion, uniform convergence of sigmoidal solutions to Heaviside-inclusion solutions on finite intervals, existence of pullback attractors in both settings, and convergence of the sigmoidal attractors to the Heaviside attractor (Kloeden et al., 2024). The result is not merely pointwise convergence of activations; it is convergence of long-time nonautonomous dynamics in an infinite-dimensional phase space.

At the level of representation capacity, deep Heaviside networks are formally defined by hidden activations

$$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$0

and therefore produce piecewise-constant functions. For plain DHNs, restriction to any line segment yields at most $$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$1 constant pieces, where $$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$2 is the first hidden-layer width; depth alone does not increase that one-dimensional complexity (Kong et al., 30 Apr 2025). The paper then shows that skip connections or linear neurons fundamentally change the picture. Skip-DHNs achieve linewise complexity growing like $$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$3, have VC dimension of order $$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$4 up to logarithmic factors in rectangular architectures, and attain approximation rates

$$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$5

for $$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$6-Hölder targets. Lin-DHNs reach VC dimension of order $$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$7 and approximation rates

$$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$8

which match lower bounds up to logarithmic factors and yield minimax-optimal nonparametric regression rates after suitable scaling of architecture size (Kong et al., 30 Apr 2025). The paper’s main conclusion is that plain DHNs are expressively limited, but those limits can be overcome without abandoning Heaviside activations entirely.

A separate line of work uses recursive compositions of logistic-type Heaviside sequence functions to model layered state transitions. One proposed recursive form is

$$\sigma_\varepsilon(x)=\frac{1}{1+e^{-x/\varepsilon}}$$9

with event times $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$0. In that framework, $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$1 is a smooth logistic approximation of Heaviside, $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$2 is a delta sequence, and the recursive compositions are interpreted as smooth approximations of piecewise-constant mental-state potentials whose $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$3 limits are exact solutions of a multidimensional inviscid advection equation (Shin, 2024). The application is unconventional, but the mathematical structure is characteristic: nested Heaviside or sigmoid expansions generate multistage switching behavior, while derivatives of those expansions generate spike-like activation events.

Taken together, these results support a broader interpretation. Heaviside-based architectures are not just limiting cases of smooth networks; they are a separate representation class whose deficiencies and advantages depend strongly on whether the model includes skip pathways, analog subchannels, or set-valued threshold formulations.

6. Multiscale and algebraic theories of sigmoid expansions

A different viewpoint treats sigmoid expansions as transformed versions of rational or orthogonal expansions. For functions with singularities at $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$4, rational approximants of the form

$$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$5

achieve root-exponential error

$$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$6

After the logarithmic change of variables $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$7, the elementary factors become translated logistics: $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$8 so that $$H(x)= \begin{cases} 0,&x<0,\ \tfrac12,&x=0,\ 1,&x>0, \end{cases}$$9 is a linear combination of sigmoids plus a constant (Huybrechs et al., 2023). Exponential clustering of poles near the singularity becomes approximately uniform spacing of sigmoid centers in $0$00, and the paper interprets this as a multiscale resolution principle shared by rational approximation, sigmoidal approximation, and $0$01-mesh refinement for PDEs. It emphasizes the balancing law behind the root-exponential rate: grid extent $0$02 versus spacing $0$03 (Huybrechs et al., 2023).

An algebraic theory places sigmoids inside the derivative subgroup of exponential Riordan arrays. For an analytic sigmoid $0$04, the array

$0$05

belongs to that subgroup, with multiplication corresponding to composition by the chain rule (Barry, 2017). Examples include

$0$06

as well as Gudermannian and Gompertz variants (Barry, 2017). The associated production matrix determines whether the inverse array is the coefficient array of an orthogonal polynomial family: tri-diagonal production matrices yield orthogonal systems and moment interpretations for the sigmoid derivative. In this sense, a sigmoid expansion can be studied either as a neural-style superposition of smoothed steps or as a structured generating-function object linked to recurrence relations, moment sequences, and Hankel transforms (Barry, 2017).

This algebraic and multiscale literature suggests a unifying interpretation. Heaviside expansions emphasize discrete region selection; sigmoid expansions regularize that selection; rational, orthogonal, and mesh-based theories explain where the placement and scaling of those transitions should come from. The modern applied constructions in pruning, control, and binary deep learning can therefore be read as specialized descendants of a broader approximation-theoretic idea: represent complexity by accumulating thresholded transitions across location, scale, or hierarchy.