Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions

Abstract: The choice of activation function plays a crucial role in the optimization and performance of deep neural networks. While the Rectified Linear Unit (ReLU) remains the dominant choice due to its simplicity and effectiveness, its lack of smoothness may hinder gradient-based optimization in deep architectures. In this work we propose a family of $C<sup>{2N}$-smooth activation functions whose gate follows a log-logistic CDF, achieving ReLU-like performance with purely rational arithmetic. We introduce three variants: GEM (the base family), E-GEM (an $ε$-parameterized generalization enabling arbitrary $L<sup>p$-approximation of ReLU), and SE-GEM (a piecewise variant eliminating dead neurons with $C<sup>{2N}$ junction smoothness). An $N$-ablation study establishes $N=1$ as optimal for standard-depth networks, reducing the GELU deficit on CIFAR-100 + ResNet-56 from 6.10% to 2.12%. The smoothness parameter $N$ further reveals a CNN-transformer tradeoff: $N=1$ is preferred for deep CNNs, while $N=2$ is preferred for transformers. On MNIST, E-GEM ties the best baseline (99.23%). On CIFAR-10 + ResNet-56, SE-GEM ($ε=10<sup>{-4}$) surpasses GELU (92.51% vs 92.44%) -- the first GEM-family activation to outperform GELU. On CIFAR-100 + ResNet-56, E-GEM reduces the GELU deficit from 6.10% (GEM $N=2$) to just 0.62%. On GPT-2 (124M), GEM achieves the lowest perplexity (72.57 vs 73.76 for GELU), with GEM $N=1$ also beating GELU (73.32). On BERT-small, E-GEM ($ε=10$) achieves the best validation loss (6.656) across all activations. The $ε$-parameterization reveals a scale-dependent optimum: small $ε$ ($10<sup>{-4}$--$10<sup>{-6}$) for deep CNNs and larger transformers, with the special case of small transformers (BERT-small) benefiting from large $ε$ ($ε=10$) due to its limited depth and unconstrained gradients.

• The paper introduces GEM using rational, tunable activation functions with C^(2N) smoothness, offering a compelling alternative to ReLU by mitigating gradient flow issues.
• Empirical benchmarks reveal that GEM with N=1 closely matches GELU and Swish in CNNs, while GEM with N=2 lowers perplexity in transformer models like GPT-2.
• GEM’s reliance on rational arithmetic ensures hardware-friendly computations and controlled gradient propagation, addressing the dead-neuron problem without expensive transcendental functions.

Geometric Monomial (GEM): A Rational, 2N-Differentiable Activation Family

Introduction and Motivation

The selection of an activation function is central to neural network optimization and representational efficiency. Although ReLU dominates in practice due to its simplicity and favorable optimization properties, its lack of smoothness can impede gradient propagation—especially in deep or highly regularized networks. Smooth alternatives, such as GELU, Swish, and Mish, address this at the expense of increased computational complexity via reliance on transcendental functions (e.g., $\exp$, $\tanh$, $\operatorname{erf}$), thus limiting their efficiency, particularly in hardware-constrained environments.

The "Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions" (2604.21677) introduces a rational, tunable, and highly smooth activation framework. GEM generalizes ReLU via a rational gate inspired by the log-logistic CDF and parameterized for $C^{2N}$ smoothness, offering controlled tradeoffs between sharpness, gradient flow, and computational simplicity.

Definition and Core Properties of GEM

Formally, for a positive integer $N$, the GEM activation is defined:

$$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$

Variants include:

• GEM (base): Controlled by $N$ for $C^{2N}$ smoothness at the origin; nonnegative, monotonic, asymptotically linear.
• E-GEM (epsilon-parameterized): Adds a scale parameter $\varepsilon$ to approximate ReLU arbitrarily closely, while maintaining smoothness.
• SE-GEM (smooth epsilon): A piecewise construction with $C^{2N}$ junction smoothness that eliminates dead neurons, utilizing a rational, non-monotonic negative branch.

Distinctive features:

• Smoothness: For $\tanh$0, GEM is $\tanh$1 at $\tanh$2, whereas ReLU is only $\tanh$3. The higher-order vanishing derivatives at the origin serve as implicit regularization and mitigate dead neuron effects without sacrificing monotonicity for $\tanh$4.
• Gradient Boundedness: The denominator ensures that, for large $\tanh$5, GEM behaves linearly—this bounds gradient magnitudes and avoids the exploding activation/gradient phenomena of high-degree monomials.
• Probabilistic Gating: The gate is the CDF of a log-logistic or, equivalently, an approximate symmetric Beta$\tanh$6 distribution—placing GEM in the $\tanh$7 gating family like Swish and GELU, but with only rational arithmetic.

Theoretical and Architectural Analysis

The smoothness parameter $\tanh$8 directly controls tradeoffs between sharpness and gradient propagation:

• Larger $\tanh$9: Approaches a step function at $\operatorname{erf}$0, mimicking ReLU, but severe gradient suppression near $\operatorname{erf}$1 leads to vanishing gradients and impedes optimization in deep stacks when $\operatorname{erf}$2 is large.
• Smaller $\operatorname{erf}$3: Provides a more gradual gating, preserving gradient flow across activation chains, crucial for deep convolutional architectures without strong residual connections.

GEM, by design, reduces the classical dead-neuron problem at $\operatorname{erf}$4, though for $\operatorname{erf}$5 it inherits the zero-output/zero-gradient pathology of ReLU. SE-GEM specifically addresses this by using a rational, non-monotonic negative branch, allowing negative pre-activations to recover—mimicking the dead-neuron avoidance of Swish/Mish but without transcendental computation.

GEM's efficiency is evident: only multiplications and additions are needed, and the backward pass can exploit forward-pass caches for further speedup. The FLOPs for $\operatorname{erf}$6 GEM are lower than Swish and GELU, especially in fused kernel implementations.

Empirical Benchmarks

Benchmarks encompass standard vision (MNIST, CIFAR-10/100 + ResNet-20/56) and NLP (BERT-small, GPT-2) tasks, focusing on test accuracy (or perplexity), runtime, throughput, and robustness.

CNNs: Depth-Complexity Tradeoff and $\operatorname{erf}$7-Ablation

• N-Ablation (CIFAR-10/100 + ResNet-20): Accuracy drops monotonically with increasing $\operatorname{erf}$8. For $\operatorname{erf}$9, GEM matches GELU/Swish to within 0.08% (CIFAR-10) and 0.28% (CIFAR-100).
• Deep Networks (ResNet-56): GEM with $C^{2N}$0 lags significantly behind GELU, especially on harder tasks (a 6.10% deficit on CIFAR-100/ResNet-56 for $C^{2N}$1). Reducing to $C^{2N}$2 compresses this deficit to 2.12%.
• Gradient Suppression: As $C^{2N}$3 increases, compounding vanishing gradients from many activation layers result in rapid underfitting and, at high $C^{2N}$4, optimization collapse.

Key empirical recommendation: $C^{2N}$5 for deep CNNs maximizes gradient flow and performance; higher $C^{2N}$6 is viable only for shallower models.

Transformer Architectures: Residual Connections and Optimal Smoothness

• BERT-small (4 layers): All activations, including GEM, perform similarly; insufficient depth for gradient suppression to matter.
• GPT-2 (12 layers, 124M): GEM ($C^{2N}$7) achieves the lowest perplexity (72.57) compared to GELU (73.76), a statistically significant improvement. The residual connections in transformers bypass the activation chain, nullifying gradient suppression concerns; the sharper-gated ($C^{2N}$8) GEM is optimal.

Extended GEM Family: E-GEM and SE-GEM

• E-GEM: By introducing $C^{2N}$9, one can approximate ReLU arbitrarily closely (in norm) while keeping high-order smoothness; as $N$0, E-GEM converges pointwise to ReLU.
• SE-GEM: Combines ReLU-style linearity for $N$1 with a smooth, rational negative branch for $N$2. It both preserves $N$3 smoothness at the origin and eliminates dead neurons.
• Benchmarking: SE-GEM matches or outperforms GELU in deep settings: e.g., on CIFAR-10 + ResNet-56, SE-GEM$N$4 exceeds GELU (92.51% vs 92.44%). On challenging setups like CIFAR-100 + ResNet-56, E-GEM$N$5 reduces the GELU gap from 6.10% to only 0.62%.

Noise Robustness

Across additive Gaussian corruptions, GEM and its variants match the robustness of standard activations—no activation shows pathological sensitivity or improved resilience, indicating architectural and optimization decisions dominate noise generalization.

Practical and Theoretical Implications

• Model Scale Dependencies: The optimal $N$6 or $N$7 are both architecture- and scale-dependent. Deep CNNs benefit from smaller $N$8 (wider, smoother gates); transformers can leverage sharper (higher $N$9 or lower $$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$0) gates because residual connections mitigate gradient suppression.
• Rational Arithmetic: GEM’s rational-only computation makes it well-suited for deployment in transpiler-based environments (e.g., edge TPUs, FPGAs, integer-only acceleration), or settings where transcendental functions are impractical or slow.
• Smooth GLU Extensions: The paper sketches GEM-GLU (GMGLU) for transformers, supporting further research into integrating these activations in SOTA transformer blocks.
• Regularization and Optimization: The vanishing derivative property for degree $$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$1 at the origin supplies implicit regularization, smoothing early optimization dynamics and possibly providing theoretical advantages for certain generalization regimes.
• Dead Neuron Elimination: With SE-GEM, the classical dead-neuron phenomenon is fully addressed for all $$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$2, without sacrificing computational simplicity.
• Adaptive Schedules: Future work is suggested on learned adaptive schedules for $$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$3 or $$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$4, and on large-scale runs of GEM-GLU in state-of-the-art models.

Conclusion

The GEM activation family systematically unifies rational computation, controlled smoothness, and tunable gradient flow, achieving competitive or state-of-the-art results on both vision and language modeling tasks. For deep CNNs, GEM ($$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$5) or SE-GEM provide improved or parity accuracy with GELU and Swish, but without their computational burdens. For transformers, GEM ($$B_N(x) = \max \left(0, \frac{x^{2N+1}}{1 + x^{2N}} \right).$$6) outperforms GELU in perplexity on GPT-2. The E-GEM and SE-GEM generalizations introduce additional flexibility for architecture-dependent optimization.

These findings underscore that high-order smooth rational activations are practically viable alternatives to standard transcendental activations—offering hardware efficiency, theoretical clarity, and measurable empirical performance. The scale-dependent parameterization and compatibility with modern GLU architectures position GEM-family activations as robust candidates for future deep learning system design.