Sigmoid Transformer

Updated 2 July 2026

Sigmoid Transformer is a self-attention variant that replaces softmax with an element-wise sigmoid function, eliminating the sum-to-one constraint.
It uses precise scaling and bias adjustments to maintain stability and improve sample complexity during training.
Empirical results show that sigmoid attention delivers enhanced performance, faster convergence, and higher throughput in language, vision, graph, and biological tasks.

The Sigmoid Transformer refers to a class of Transformer architectures in which the canonical row-wise softmax operation within self-attention is replaced by an element-wise sigmoid function or related sigmoid-based gating. This architectural deviation eliminates the sum-to-one normalization constraint of softmax, fundamentally altering the dynamics of attention, gradient propagation, and representation learning. Contemporary research demonstrates that this modification, properly normalized, provides advantageous regularity properties, improved sample complexity, and practical performance gains across language, vision, graph, and biological domains (Ramapuram et al., 2024, Sadashivaiah et al., 29 Apr 2026, Yan et al., 1 Feb 2025, Guo et al., 19 Apr 2026).

1. Mathematical Formulation of Sigmoid Attention

Let $X = (x_1, ..., x_n) \in \mathbb{R}^{n \times d}$ be a sequence of $n$ token embeddings. Standard softmax-based self-attention forms queries, keys, and values as $q_i = W_q x_i$ , $k_j = W_k x_j$ , $v_j = W_v x_j$ and produces, for each token $i$ :

$y^{\mathrm{softmax}}_i = \sum_{j=1}^n \mathrm{softmax}_j\Bigl( \frac{q_i \cdot k_j}{\sqrt{d}} \Bigr) v_j$

where $\mathrm{softmax}_j(z_1, ..., z_n) = \frac{e^{z_j}}{\sum_{\ell=1}^n e^{z_\ell}}$ .

In the Sigmoid Transformer, the attention mechanism replaces the row-wise softmax with an element-wise sigmoid, resulting in: $y^{\mathrm{sigmoid}}_i = \frac{1}{n^\alpha} \sum_{j=1}^n \sigma(x_i^\top A x_j) W_v x_j$ where $A = W_q^\top W_k / \sqrt{d}$ , and $n$ 0. Proper scaling is essential: the only exponent $n$ 1 that prevents vanishing or exploding outputs as $n$ 2 is $n$ 3 (Ramapuram et al., 2024). Thus, the canonical form is: $n$ 4 Other implementation variants add a bias $n$ 5 to the pre-sigmoid logits, yielding similar mean scales as the softmax (Sadashivaiah et al., 29 Apr 2026, Yan et al., 1 Feb 2025).

2. Theoretical Properties and Scaling Laws

Sigmoid attention exhibits distinct theoretical properties relative to softmax:

Bounded Derivatives and Jacobian Structure: The derivative of $n$ 6 is globally bounded by 0.25. Consequently, the Jacobian of the sigmoid attention nonlinearity is diagonal with all entries in $n$ 7. In contrast, the softmax Jacobian is dense and its operator norm can grow exponentially with the input scale, leading to potential gradient instabilities.

Scaling with Sequence Length: The necessity of $n$ 8 normalization in sigmoid attention arises from two arguments (Ramapuram et al., 2024):

Mean-convergence: $n$ 9 converges to a finite expectation as $q_i = W_q x_i$ 0; any other scaling leads to vanishing or divergent outputs.
Sequence-doubling invariance: Duplicating every token should not change the attention output; this property uniquely fixes the scaling exponent at $q_i = W_q x_i$ 1.

Statistical Sample Complexity: The mixture-of-experts formalism demonstrates that sigmoid attention enjoys polynomial sample complexity for function classes where softmax gating requires exponential data—particularly for polynomial expert networks in the dense regime (Yan et al., 1 Feb 2025). For instance, achieving $q_i = W_q x_i$ 2 error $q_i = W_q x_i$ 3 with polynomial experts requires $q_i = W_q x_i$ 4 samples for sigmoid gating versus $q_i = W_q x_i$ 5 for softmax.

3. Empirical Results and Training Stability

Empirical evaluations corroborate the theoretical findings across several domains.

Single-Cell Foundation Models: On six held-out single-cell RNA-seq datasets, sigmoid attention achieves approximately $q_i = W_q x_i$ 6 higher cell-type separation (Maximum Mean Discrepancy), dominates on cohesion metrics (Leiden NMI, ARI), and yields systematically lower validation loss. Training is up to $q_i = W_q x_i$ 7 faster, with large-scale models (160M–1.4B parameters) demonstrating enhanced convergence and robustness (no catastrophic divergence observed in contrast to softmax attention) (Sadashivaiah et al., 29 Apr 2026).

Gradient Stability: Stress tests (training at 8K token context, no gradient clipping) show that softmax attention models experience catastrophic divergence (gradient norms increase by $q_i = W_q x_i$ 8, attention logits reach $q_i = W_q x_i$ 9), while sigmoid attention maintains stable gradients (range $k_j = W_k x_j$ 0– $k_j = W_k x_j$ 1) and bounded attention scores (≤ 5).

Computational Efficiency: Sigmoid attention obviates the need for row-wise normalization, enabling fully parallel element-wise operations. Throughput gains of $k_j = W_k x_j$ 2– $k_j = W_k x_j$ 3 on GPUs have been observed compared to softmax-based attention (Yan et al., 1 Feb 2025, Sadashivaiah et al., 29 Apr 2026).

4. Applications: Biological, Graph, and LLMs

Biological Foundation Models: The adoption of sigmoid attention in biological foundation models has enabled stable training on large, variable-length sequences (e.g., single-cell transcriptomics), with kernel implementations (TritonSigmoid) achieving $k_j = W_k x_j$ 4 TFLOPS on H100 GPUs and demonstrating superior speed and padding support over FlashAttention-2 (Sadashivaiah et al., 29 Apr 2026).

Graph Transformers and Over-Smoothing: In graph domains, sigmoid-gated attention (SigGate-GT) breaks the sum-to-one constraint of softmax, addressing over-smoothing and attention entropy degeneration. Element-wise learned gating with sigmoid allows each attention head to selectively silence uninformative connections, raising the effective rank of outputs and maintaining higher embedding diversity. On molecular benchmarks (ZINC, ogbg-molhiv), SigGate-GT achieves state-of-the-art results, with 30% reduction in over-smoothing and robust training across a $k_j = W_k x_j$ 5 learning rate range (Guo et al., 19 Apr 2026).

Natural Language and General Sequence Modeling: Sigmoid attention matches or outperforms softmax attention across varied NLP tasks, achieving sample-efficiency gains and removing token competition bias. Synthetic and real-world tasks (ARC, HellaSwag, etc.) confirm competitive or improved generalization with faster computation (Yan et al., 1 Feb 2025).

5. Implementation Practices and Hardware Considerations

GPU Kernels: The TritonSigmoid kernel leverages block-sparse execution and fused sigmoid calculations for efficiency. It natively supports arbitrary padding patterns essential for variable-length inputs, delivering 7.15 $k_j = W_k x_j$ 6 forward speedup over PyTorch baselines and maintaining performance when up to 25% of the sequence is padded (Sadashivaiah et al., 29 Apr 2026). Compared to FlashAttention-2, TritonSigmoid achieves 10–20% higher throughput, with negligible overhead from padding due to block skipping.

Transformer Block Integration: Replacing softmax attention in existing Transformer blocks involves substituting the normalization with element-wise sigmoid, introducing a bias term $k_j = W_k x_j$ 7 to center the mean scale, and maintaining standard query/key scaling by $k_j = W_k x_j$ 8. Standard initialization (e.g., Xavier uniform) and learning rate schedules remain effective. Owing to the bounded derivative of sigmoid, higher learning rates or omitting gradient clipping are often tolerable.

$v_j = W_v x_j$ 3 This direct code adaptation supports efficient, numerically stable sigmoid attention in production settings (Sadashivaiah et al., 29 Apr 2026).

6. Sigmoid Gating Variants in Transformer Architectures

Beyond direct softmax-to-sigmoid substitution, sigmoid gating has been incorporated into attention as an auxiliary mechanism, particularly in graph domains (Guo et al., 19 Apr 2026). In SigGate-GT, for each attention head $k_j = W_k x_j$ 9, a learned gate $v_j = W_v x_j$ 0 is computed per token, and the traditional softmax attention output $v_j = W_v x_j$ 1 is post-multiplied element-wise by $v_j = W_v x_j$ 2. This gating enables heads to “say nothing” when no informative connections are present, breaking mandatory attention sinks and improving representational diversity, entropy, and depth stability. Ablation studies confirm that post-attention per-head gating is most effective, with only a 1% parameter overhead.

7. Modeling Implications, Limitations, and Open Questions

Sigmoid-based attention mechanisms fundamentally alter self-attention’s inductive bias by removing inter-token competition and enabling independent contribution scaling. This allows for higher sample efficiency, stability (via bounded derivatives and decoupled gradients), and improved representational richness (especially in molecular and graph domains). Computationally, element-wise sigmoid is more amenable to parallelism and less memory-bound than softmax.

Known limitations include the current theoretical bounds focusing primarily on single-head attention (the fully general multi-head MoE setting remains open) and the dependency of empirical results on correct scaling and bias calibration (Ramapuram et al., 2024, Yan et al., 1 Feb 2025). Open research directions include hybrid gating schemes, further multi-head theoretical analysis, and domain-specific optimizations.

Key references: