Brownian ReLU(Br-ReLU): A New Activation Function for a Long-Short Term Memory (LSTM) Network

Abstract: Deep learning models are effective for sequential data modeling, yet commonly used activation functions such as ReLU, LeakyReLU, and PReLU often exhibit gradient instability when applied to noisy, non-stationary financial time series. This study introduces BrownianReLU, a stochastic activation function induced by Brownian motion that enhances gradient propagation and learning stability in Long Short-Term Memory (LSTM) networks. Using Monte Carlo simulation, BrownianReLU provides a smooth, adaptive response for negative inputs, mitigating the dying ReLU problem. The proposed activation is evaluated on financial time series from Apple, GCB, and the S&P 500, as well as LendingClub loan data for classification. Results show consistently lower Mean Squared Error and higher $R<sup>2$ values, indicating improved predictive accuracy and generalization. Although ROC-AUC metric is limited in classification tasks, activation choice significantly affects the trade-off between accuracy and sensitivity, with Brownian ReLU and the selected activation functions yielding practically meaningful performance.

• The paper introduces Brownian ReLU (Br-ReLU), a stochastic activation function that improves gradient flow and learning stability in LSTM networks, outperforming traditional activations in volatile financial time-series
• Br-ReLU replaces standard activations in the LSTM's cell state and output gate, utilizing a Monte Carlo average of reflected Brownian motion to handle negative inputs, with a tunable level of randomness.
• The function demonstrates superior performance on stock price forecasting tasks. While the highest accuracy is not uniformly decisive, Br-ReLU's ability to achieve a notable accuracy-sensitivity trade-off is promising within an imbalanced classification setting.

Motivation and contribution

This paper proposes Brownian ReLU (Br-ReLU), a stochastic activation function for Long Short-Term Memory (LSTM) networks, targeted at noisy, non-stationary financial time series. The authors motivate the work by the known pathologies of rectifier activations—non-differentiability at zero, unbounded outputs, and the "dying ReLU" problem—and argue that deterministic leaky variants (LeakyReLU, PReLU) provide only partial remedies. Their hypothesis is that injecting Brownian-motion-driven randomness into the negative-input regime improves gradient flow and learning stability in LSTM gating. The proposal is evaluated on three forecasting tasks (Apple stock, Ghana Commercial Bank (GCB) stock, and the S&P 500) and one imbalanced classification task (LendingClub loan status).

Function definition and training

For positive inputs Br-ReLU is the identity; for $x \le 0$ it outputs a scaled Monte Carlo average of reflected Brownian motion samples:

$$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$

The reflection/symmetry principle of Brownian motion extends its behavior to negative pre-activations. Two hyperparameters govern behavior: the learnable slope $\alpha$, initialized and updated as in PReLU, and the number of Monte Carlo paths $M$. As $M$ grows, the mean path converges by the law of large numbers to a smooth deterministic response, so the function interpolates between fully stochastic ($M=1$) and near-deterministic regimes. The authors derive the gradient estimator with respect to $\alpha$ as $$\partial L_{\mathcal{B}}/\partial\alpha = -\sum_{i\in\mathcal{B}}\delta_i\,\mathbf{1}_{x_i\le 0}\,\bar{B}(|x_i|)$$ and supply an explicit minibatch training algorithm. Note that the noise enters only through the forward sample $\bar{B}$; the paper treats it as fixed within a step rather than applying a reparameterization or re-sampling scheme during backpropagation of gradients to earlier layers.

Br-ReLU replaces the candidate cell state activation, $$\tilde{C}_t = \text{Br-ReLU}(W_{xc}X_t + W_{hc}h_{t-1}+b_c)$$, and the output nonlinearity, $$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$0, while sigmoid gates are left unchanged.

Forecasting results

A sensitivity analysis over $$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$1 shows dataset-dependent behavior. GCB is essentially insensitive to $$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$2 ($$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$3 ≈ 0.987 throughout); Apple shows moderate variation ($$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$4 from 0.9036 to 0.9381); S&P 500 peaks at $$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$5 ($$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$6 = 0.9896, MSE = 0.000232) but degrades at $$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$7 ($$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$8 = 0.9736). The authors select $$f(x) = \begin{cases} x & x > 0 \ -\alpha \cdot \frac{1}{M}\sum_{k=1}^{M} B^{(k)}(|x|), \quad B^{(k)}(|x|)\sim\mathcal{N}(0,|x|) & x \le 0 \end{cases}$$9 as the operating point, though this choice appears empirical rather than derived from any variance–bias analysis.

The headline comparisons against standard activations are strong:

Dataset	Activation	MSE	Test $\alpha$0
Apple	Br-ReLU	0.002035	0.9381
Apple	ReLU	0.005931	0.3164
Apple	LeakyReLU	0.160918	—
GCB	Br-ReLU	0.000275	0.9869
S&P 500	Br-ReLU	0.000242	0.9891
S&P 500	Tanh	0.000255	0.9841

On Apple, Br-ReLU achieves the lowest MSE and highest test $\alpha$1, while ReLU generalizes poorly (test $\alpha$2 = 0.3164 despite train $\alpha$3 = 0.9881) and LeakyReLU performs worst. On GCB, all activations perform similarly (test $\alpha$4 between 0.9815 and 0.9869), so Br-ReLU's advantage there is marginal—the authors acknowledge the dataset's inherent stability limits discriminative power. On S&P 500, Tanh is competitive (MSE 0.000255 vs. 0.000242), indicating the stochastic component is not uniformly decisive on high-volatility data either. Visual inspection confirms close tracking of actual price trajectories across all three series.

Classification results

On the imbalanced LendingClub task, results are considerably weaker and more nuanced. ROC–AUC values cluster between roughly 0.50 and 0.53 across all activations—marginally above random guessing. Among Br-ReLU variants, higher accuracy coincides with collapsed minority-class recall: $\alpha$5 yields the highest accuracy (0.7802) but recall of only 0.0272, whereas $\alpha$6 gives the best balance within the family (recall 0.2446, F1 0.2290). GELU achieves the highest recall among standard activations (0.2609). The paper correctly emphasizes that in credit-risk settings, recall and F1 matter more than accuracy, and concedes that no activation produces a substantial AUC improvement—the practical claim is limited to a favorable accuracy–sensitivity trade-off for well-tuned Br-ReLU configurations.

Limitations and open questions

Several caveats bear directly on the reported gains. First, the evaluation lacks statistical significance testing, confidence intervals, or multiple seeds, so the differences between Br-ReLU and competitive baselines such as Tanh or PReLU cannot be assessed rigorously. Second, the classification results hover at chance-level ROC–AUC, which raises the question of whether the activation choice—or the overall model capacity and class-imbalance handling—is the binding constraint. Third, the computational overhead of drawing $\alpha$7 Gaussian samples per negative pre-activation per step is not quantified, nor is the interaction between stochastic forward passes and gradient estimation for upstream weights analyzed theoretically. Finally, the sensitivity of results to $\alpha$8 on high-variance datasets (S&P 500) suggests a principled selection criterion for $\alpha$9 remains open, as does evaluation beyond finance on tasks where stochasticity is not naturally motivated.

Conclusion

The paper introduces Br-ReLU, a Monte Carlo–based stochastic variant of ReLU applied to LSTM cell-state and hidden-state nonlinearities, with a learnable negative slope. Empirically, it attains the lowest MSE and highest test $M$0 on three stock-price forecasting benchmarks, most notably test $M$1 of 0.9381 on Apple where standard ReLU collapses to 0.3164, while offering only marginal gains on stable data and no meaningful ROC–AUC improvement on imbalanced classification. The evidence supports Br-ReLU as a viable alternative for volatile time-series regression, contingent on replication with uncertainty quantification and cost accounting for the Monte Carlo sampling it requires.