Stochastic Activations in Neural Networks

• Stochastic activations are probabilistic mechanisms that replace fixed nonlinearities with random sampling to yield outputs with inherent uncertainty.
• They are applied in feedforward, recurrent, and hardware-based systems to improve robustness, sparse routing, and energy efficiency.
• Advanced constructions like competition-based units, random-threshold gates, and Gaussian perturbations offer versatile methods for optimizing training and inference.

Stochastic activations are activation mechanisms in which the forward nonlinearity, routing decision, or state update is random rather than fixed for a given pre-activation. In the neural-network literature, the term encompasses Bernoulli-sampled binary hidden units, Gaussian-perturbed activations, stochastic local winner-takes-all competition, random-threshold gates, Brownian and $q$-derivative constructions, stochastic state preservation in recurrent networks, and hardware realizations in stochastic computing; related work also studies activation patterns themselves as stochastic processes and, in a different scientific usage, treats “stochastic activation” as a fluctuation-induced rare transition in a genetic switch (Raiko et al., 2014, Shridhar et al., 2019, Panousis et al., 2021, Rossi, 4 Jul 2026, Krueger et al., 2016, Li et al., 2017, Lehmler et al., 2023, Hertz et al., 2018).

1. Conceptual scope and terminology

The most basic contrast is with deterministic activations such as sigmoid, $\tanh$, ReLU, Leaky ReLU, PReLU, ELU, SELU, and Swish, whose input-output relationship is fixed. Stochastic activations replace that fixed mapping with a conditional distribution, a sampled gate, a sampled winner, or a sampled transformation. In stochastic feedforward networks, for example, each hidden unit is a Bernoulli random variable,

$$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$

so the hidden layer is a random vector $h\in\{0,1\}^H$ and the model output is a mixture over hidden “particles” (Raiko et al., 2014).

Several papers use a broader operational notion. ProbAct defines

$$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$

so the activation output is Gaussian-distributed around a mean. LWTA networks group units into blocks and sample a one-hot winner rather than applying an elementwise nonlinearity independently. Zoneout stochastically preserves the previous hidden state instead of updating it. In stochastic computing, a scalar value is encoded as the probability of $1$s in a bit stream, so even a deterministic software nonlinearity becomes a probability-coded hardware operation (Shridhar et al., 2019, Panousis et al., 2021, Krueger et al., 2016, Li et al., 2017).

The phrase is also potentially misleading. “Stochastic” in stochastic gradient descent refers to minibatch sampling, not to random activations. A separate line of theory on recurrent state equations analyzes SGD with nonlinear activations under assumptions such as

$$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$

and proves linear convergence for learning $h_{t+1}=\phi(Ah_t+Bu_t)$; this concerns optimization with nonlinear activations, not stochastic activations in the forward pass (Oymak, 2018). Outside machine learning, “stochastic activation” can denote rare noise-driven switching, with transition rate

$\Gamma=\eta e^{-S_0},$

in an autoregulatory genetic switch (Hertz et al., 2018).

2. Principal construction families

The literature contains several distinct ways to introduce stochasticity into activation behavior.

Family	Core mechanism	Representative papers
Sampled-output units	Sample from Bernoulli or Gaussian output law around an activation mean	(Raiko et al., 2014, Shridhar et al., 2019, Ravanbakhsh et al., 2016)
Competition-based activations	Sample a one-hot winner inside a local block of competing units	(Panousis et al., 2021, Panousis et al., 2022)
Randomized activation transforms	Randomize the activation itself via $q$-derivatives, Brownian motion, or random thresholds	(Nielsen et al., 2018, Nagaraj et al., 2024, Awiakye-Marfo et al., 23 Jan 2026, Rossi, 4 Jul 2026)
Stochastic state preservation	Randomly keep the previous recurrent state instead of applying the full update	(Krueger et al., 2016)
Stochastic-computing activations	Encode values as bit-stream probabilities and implement nonlinearities with FSM-based hardware	(Li et al., 2017)
Learnable stochastic activation functions	Place a Gaussian process prior over each neuron’s scalar nonlinearity	(Urban et al., 2017)

In sampled-output models, the activation law itself is the central object. Exp-RBM starts from a smooth monotonic mean activation $\tanh$0 and derives a stochastic unit whose conditional is locally approximated by

$\tanh$1

The variance is therefore tied to the derivative of the base nonlinearity: rapidly changing regions induce larger uncertainty, whereas saturated regions induce smaller variance (Ravanbakhsh et al., 2016). ProbAct uses a simpler reparameterized Gaussian perturbation, with fixed or trainable $\tanh$2, and interprets this as “augmentation-by-activation” (Shridhar et al., 2019).

Competition-based activations replace pointwise nonlinearities with sparse routing. In dense LWTA layers, weights are organized as $\tanh$3, each block $\tanh$4 contains $\tanh$5 competing linear units, and the winner variable is sampled from a categorical distribution. The layer output is

$\tanh$6

so only one competitor per block survives and all others are forced to zero (Panousis et al., 2021). A related representation-learning formulation augments winner sampling with binary utility indicators $\tanh$7 and an Indian Buffet Process stick-breaking prior to infer which blocks or connections are essential (Panousis et al., 2022).

Other constructions randomize the activation transform more directly. $\tanh$8-neurons define

$\tanh$9

with stochastic $$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$0 sampled around $$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$1, producing a structured finite-difference activation linked to the curvature of $$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$2 (Nielsen et al., 2018). BrowNNe perturbs ReLU with Brownian motion,

$$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$3

and computes backward signals through a nonlocal directional derivative rather than an ordinary derivative (Nagaraj et al., 2024). BrownianReLU for LSTMs uses the piecewise form

$$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$4

with Monte Carlo averaging over Brownian samples (Awiakye-Marfo et al., 23 Jan 2026). A structurally different interpretation arises in random-threshold activations: if $$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$5 has CDF $$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$6, then the transmitted signal is

$$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$7

ReLU, GELU, SiLU/Swish, and hard swish appear as special cases under degenerate, Gaussian, logistic, and uniform threshold laws, respectively (Rossi, 4 Jul 2026).

A separate probabilistic route makes the activation function itself learnable. Gaussian process neurons assign each neuron its own scalar activation $$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$8 with prior

$$P(h_i=1\mid x)=\sigma(a_i), \qquad a_i=W_{i:}x+b_i,$$9

so the nonlinearity is not fixed in advance but inferred jointly with weights, kernel lengthscales, and noise parameters (Urban et al., 2017).

3. Training and inference methodologies

Training stochastic activations is usually harder than training deterministic nonlinearities because gradients must pass through discrete samples, latent routing variables, or function-valued uncertainty. In stochastic feedforward networks, the objective

$h\in\{0,1\}^H$0

is intractable, and the paper studies a Monte Carlo criterion

$h\in\{0,1\}^H$1

A key result is that $h\in\{0,1\}^H$2 is qualitatively different: maximizing the expected $h\in\{0,1\}^H$3 makes hidden units prefer deterministic behavior, whereas $h\in\{0,1\}^H$4 is needed if meaningful stochasticity is to be learned. The paper compares five estimators $h\in\{0,1\}^H$5 and reports that $h\in\{0,1\}^H$6 and $h\in\{0,1\}^H$7 perform best overall among the fully stochastic methods (Raiko et al., 2014).

Variational Bayes is common when the activation mechanism introduces latent random variables. LWTA models optimize an ELBO with categorical cross-entropy and KL regularization, use a symmetric categorical prior over winners, and rely on a Gumbel-Softmax / Concrete relaxation for differentiable winner sampling. At prediction time, they average over multiple sampled winner configurations, so the same input may traverse different subnetworks (Panousis et al., 2021). The diversified-representation variant additionally uses Bernoulli utility variables and a stick-breaking IBP prior, with Monte Carlo variational inference and single reparameterized samples for each latent variable (Panousis et al., 2022).

Other methods preserve backpropagation by using continuous reparameterizations or deterministic surrogates. ProbAct trains $h\in\{0,1\}^H$8 by standard backpropagation because $h\in\{0,1\}^H$9, and bounds the variance through

$$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$0

which constrains $$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$1 (Shridhar et al., 2019). Exp-RBM uses contrastive divergence, sampling either from exact conditionals or from the Gaussian approximation $$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$2 (Ravanbakhsh et al., 2016). Gaussian process neurons employ variational Bayesian inference and, using the central limit theorem, derive a fully deterministic loss that propagates means and covariances layer by layer and can be optimized with mini-batch gradient descent (Urban et al., 2017).

In recurrent settings, the stochasticity may act on state preservation rather than on a pointwise nonlinearity. Zoneout samples Bernoulli masks at each timestep and mixes the learned transition with the identity operator,

$$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$3

so masked units keep their previous values instead of being zeroed as in dropout (Krueger et al., 2016). In LLMs, “stochastic activations” have also been used to randomly choose between ReLU and SiLU in the feed-forward layer during training. One reported strategy combines such mixing with late ReLU fine-tuning (“Swi+FT”) so that training benefits from SiLU-like gradient flow while inference uses sparse ReLU activations (Lomeli et al., 26 Sep 2025).

4. Theoretical interpretations and analytical frameworks

Several papers reinterpret stochastic activations in ways that decouple forward signal transmission from more conventional nonlinear-function lore. The threshold-transmission formulation is especially explicit. For a random threshold $$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$4 with CDF $$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$5, the activation is $$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$6, whereas the complementary first-order loss is

$$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$7

For a Gaussian threshold, this yields

$$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$8

and the paper argues that the $$f(\mathbf{x})=\mu(\mathbf{x})+\sigma\epsilon,\qquad \epsilon\sim\mathcal N(0,1),$$9 term is the forward signal-transmission component, while the remaining truncated-moment term belongs to loss accounting rather than to the activation itself (Rossi, 4 Jul 2026).

Brownian activations motivated a different mathematical apparatus. BrowNNe introduces the nonlocal directional derivative

$1$0

interprets it probabilistically as the expectation of a random difference quotient, and shows that for convex objectives it behaves as an $1$1-subgradient for sufficiently large kernel index. For Brownian motion and the moving-rectangle kernel $1$2, the nonlocal derivative process is Gaussian with mean zero and variance

$1$3

This provides a calculus for activations that are Hölder continuous but nowhere classically differentiable (Nagaraj et al., 2024).

A different analytical lens treats observed activation patterns as stochastic processes after thresholding. The Poisson-process model discards activation magnitudes, keeps only spike/no-spike events, and summarizes the last hidden layer through firing rates, Mean Firing Rate,

$1$4

Fano factors,

$1$5

their layerwise means, and coefficients of variation. In that analysis, memorizing networks tend to have lower MFR and higher MF than generalizing networks, and MFR exhibits clear change points when training switches from correct to shuffled labels (Lehmler et al., 2023).

Theory on learnability with nonlinear activations also clarifies what stochasticity is not. For recurrent state equations, SGD can be proved to converge linearly when $1$6 is $1$7-increasing, including leaky ReLU but excluding vanilla ReLU because its derivative vanishes on negative inputs. This suggests that some benefits often attributed informally to stochastic activations may instead depend on monotonicity, non-flat derivatives, or conditioning of the state statistics (Oymak, 2018).

5. Empirical regimes and application domains

The empirical literature is heterogeneous because the activation mechanism, objective, and evaluation regime differ substantially across application areas. In hardware-oriented stochastic computing for DCNNs, the core problem is accurate implementation of nonlinearities under bit-stream arithmetic. Using SC-based tanh, logistic, and ReLU neurons in LeNet-5 on MNIST, the reported DCNN achieves up to $1$8, $1$9, and $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$0 improvement in area, power, and energy relative to a binary ASIC DCNN, at the cost of small precision degradation. Relative to CPU and GPU baselines, the same paper reports up to $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$1 and $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$2 area reduction, $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$3 and $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$4 power reduction, and $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$5 and $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$6 energy reduction, while the error is increased by less than $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$7. It concludes that ReLU is the preferred activation for future SC-based DCNNs because it maintains accuracy better than SC-tanh when bit-stream length is reduced (Li et al., 2017).

For adversarial robustness, stochastic competition-based activations have been especially prominent. Replacing ReLU with stochastic LWTA blocks in WideResNet-34 on CIFAR-10 under PGD adversarial training yields robust accuracies under PGD-20 with $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$8 of $$\phi(0)=0,\qquad 1\ge \phi'(x)\ge \beta \quad \forall x\in\mathbb R,$$9, $h_{t+1}=\phi(Ah_t+Bu_t)$0, and $h_{t+1}=\phi(Ah_t+Bu_t)$1 for widen factors $h_{t+1}=\phi(Ah_t+Bu_t)$2, $h_{t+1}=\phi(Ah_t+Bu_t)$3, and $h_{t+1}=\phi(Ah_t+Bu_t)$4, compared with $h_{t+1}=\phi(Ah_t+Bu_t)$5, $h_{t+1}=\phi(Ah_t+Bu_t)$6, and $h_{t+1}=\phi(Ah_t+Bu_t)$7 for the ReLU baselines. Under AutoAttack, the same models report $h_{t+1}=\phi(Ah_t+Bu_t)$8, $h_{t+1}=\phi(Ah_t+Bu_t)$9, and $\Gamma=\eta e^{-S_0},$0 robust accuracy, and robustness remains high under AA + EoT with 20 gradient averages (Panousis et al., 2021). In diversified representation learning, stochastic LWTA combined with utility sampling and ICP is reported to yield substantial model compression via pruning of unnecessary components and improved linear separability of learned representations (Panousis et al., 2022).

Generalization-oriented stochastic perturbations have been studied on both vision and text tasks. ProbAct reports gains of $\Gamma=\eta e^{-S_0},$1 on CIFAR-10, $\Gamma=\eta e^{-S_0},$2 on CIFAR-100, $\Gamma=\eta e^{-S_0},$3 on STL-10, and $\Gamma=\eta e^{-S_0},$4 on IMDB relative to ReLU for its best trainable-$\Gamma=\eta e^{-S_0},$5 variant; it also reports about $\Gamma=\eta e^{-S_0},$6 average improvement when data is halved and about $\Gamma=\eta e^{-S_0},$7 improvement when reduced to $\Gamma=\eta e^{-S_0},$8 (Shridhar et al., 2019). $\Gamma=\eta e^{-S_0},$9-neurons usually improve both training and testing loss relative to their deterministic counterparts, with the best reported MNIST result at $q$0 error ($q$1 accuracy) using $q$2-ELU and $q$3-$q$4, and the best CIFAR-10 CNN result at $q$5 accuracy with $q$6-ELU (Nielsen et al., 2018). Brownian activations in low-data regimes improve E-MNIST accuracy in both standard and deliberately overfitting-prone MLPs, with much larger gains in the latter case; BrownianReLU in LSTMs reports lower MSE and higher $q$7 on Apple, GCB, and S&P 500 forecasting, for example MSE $q$8 and test $q$9 on Apple (Nagaraj et al., 2024, Awiakye-Marfo et al., 23 Jan 2026).

Sequential and LLMs provide yet another operating point. Zoneout improves over unregularized LSTMs and recurrent dropout on Char-PTB, Text8, and pMNIST; the reported best zoneout Char-PTB configuration reaches $\tanh$00 test BPC, and recurrent batch normalization plus zoneout reaches $\tanh$01 test error on pMNIST (Krueger et al., 2016). In decoder-only LLMs, stochastic ReLU/SiLU mixing is used to reconcile optimization and sparse inference. For gated FFNs of the form

$\tanh$02

the paper states that if a fraction $\tanh$03 of ReLU outputs are zero, the FFN cost drops from $\tanh$04 to $\tanh$05; with about $\tanh$06 sparsity, CPU inference gets about a $\tanh$07 speedup. On LM1.5B and LM3B, the hybrid methods recover much of the ReLU-to-SiLU performance gap while preserving sparse inference, and test-time stochastic activations provide a controlled alternative for diverse generation (Lomeli et al., 26 Sep 2025).

6. Limitations, misconceptions, and open questions

The most consistent limitation is training difficulty. In stochastic feedforward networks, $\tanh$08 is not merely a high-variance approximation but induces a qualitatively different objective that pushes hidden units toward determinism; this directly contradicts the common misconception that a single-particle estimator is just a cheaper version of the same learning problem (Raiko et al., 2014). More generally, stochasticity is not uniformly beneficial across activation families. $\tanh$09-ReLU does not improve over ReLU, which the paper attributes to ReLU’s piecewise linearity and the fact that $\tanh$10 almost everywhere (Nielsen et al., 2018).

Another recurring concern is whether stochastic activations improve robustness or only hide gradients. The LWTA robustness paper addresses this explicitly by evaluating AutoAttack and AA + EoT with 20 gradient averages; because robustness remains high under EoT, the reported gains are argued not to be an artifact of weak evaluation or obfuscated gradients (Panousis et al., 2021). A related practical issue is train-test mismatch. In LLMs, training with stochastic ReLU/SiLU mixtures but inferring with ReLU requires late ReLU fine-tuning; without that adaptation, the paper reports that stochastic activations alone leave a mismatch that hurts ReLU inference (Lomeli et al., 26 Sep 2025).

Hardware and stochastic-process formulations introduce different caveats. In stochastic computing, longer bit streams improve accuracy and shorter streams favor SC-ReLU over SC-tanh, so the precision-energy trade-off is central rather than incidental (Li et al., 2017). In the stochastic-process analysis of thresholded activations, many spike trains exhibit $\tanh$11, meaning variability lower than a Poisson model would predict; the authors therefore treat the Poisson process as a useful baseline rather than a definitive model (Lehmler et al., 2023).

Brownian constructions remain promising but non-universal. BrowNNe reports that gains depend on kernel family, noise scale $\tanh$12, and layer placement, and it does not claim a universal proof that Brownian activations always outperform deterministic ReLU (Nagaraj et al., 2024). BrownianReLU similarly shows strong regression results but limited classification gains: on LendingClub, ROC-AUC values remain around $\tanh$13–$\tanh$14, so accuracy must be interpreted together with recall and F1-score rather than as stand-alone evidence of a strong classifier (Awiakye-Marfo et al., 23 Jan 2026).

Taken together, the literature shows that stochastic activations are not a single method but a family of probabilistic design choices: sampled hidden units, sampled competitive routing, random thresholds, function-valued uncertainty, state-preserving masks, and hardware probability encodings. Their strongest reported benefits appear in settings where stochasticity aligns with a specific systems objective—multimodal prediction, adversarial robustness, uncertainty estimation, sparse routing, low-data generalization, or hardware efficiency—rather than as a universal replacement for deterministic nonlinearities.