---
title: Sigmoid Transformer
url: https://www.emergentmind.com/topics/sigmoid-transformer
type: topic
---

# Sigmoid Transformer

The Sigmoid Transformer refers to a class of Transformer architectures in which the canonical row-wise softmax operation within self-attention is replaced by an element-wise sigmoid function or related sigmoid-based gating. This architectural deviation eliminates the sum-to-one normalization constraint of softmax, fundamentally altering the dynamics of attention, gradient propagation, and representation learning. Contemporary research demonstrates that this modification, properly normalized, provides advantageous regularity properties, improved sample complexity, and practical performance gains across language, vision, graph, and biological domains [2409.04431][2604.27124][2502.00281][2604.17324].

## 1. Mathematical Formulation of Sigmoid Attention

Let $X = (x_1, ..., x_n) \in \mathbb{R}^{n \times d}$ be a sequence of $n$ token embeddings. Standard softmax-based self-attention forms queries, keys, and values as $q_i = W_q x_i$, $k_j = W_k x_j$, $v_j = W_v x_j$ and produces, for each token $i$:

\[
y^{\mathrm{softmax}}_i = \sum_{j=1}^n \mathrm{softmax}_j\Bigl( \frac{q_i \cdot k_j}{\sqrt{d}} \Bigr) v_j
\]
where $\mathrm{softmax}_j(z_1, ..., z_n) = \frac{e^{z_j}}{\sum_{\ell=1}^n e^{z_\ell}}$.

In the Sigmoid Transformer, the attention mechanism replaces the row-wise softmax with an element-wise sigmoid, resulting in:
\[
y^{\mathrm{sigmoid}}_i = \frac{1}{n^\alpha} \sum_{j=1}^n \sigma(x_i^\top A x_j) W_v x_j
\]
where $A = W_q^\top W_k / \sqrt{d}$, and $\sigma(t) = 1/(1+e^{-t})$. Proper scaling is essential: the only exponent $\alpha$ that prevents vanishing or exploding outputs as $n \rightarrow \infty$ is $\alpha = 1$ [2409.04431]. Thus, the canonical form is:
\[
y^{\mathrm{sigmoid}}_i = \frac{1}{n} \sum_{j=1}^n \sigma(x_i^\top A x_j) W_v x_j
\]
Other implementation variants add a bias $b = -\log n$ to the pre-sigmoid logits, yielding similar mean scales as the softmax [2604.27124][2502.00281].

## 2. Theoretical Properties and Scaling Laws

Sigmoid attention exhibits distinct theoretical properties relative to softmax:

**Bounded Derivatives and Jacobian Structure:** The derivative of $\sigma(z)$ is globally bounded by 0.25. Consequently, the Jacobian of the sigmoid attention nonlinearity is diagonal with all entries in $[0, 0.25]$. In contrast, the softmax Jacobian is dense and its operator norm can grow exponentially with the input scale, leading to potential gradient instabilities.

**Scaling with Sequence Length:** The necessity of $1/n$ normalization in sigmoid attention arises from two arguments [2409.04431]:
- **Mean-convergence:** $\frac{1}{n} \sum_j \ldots$ converges to a finite expectation as $n \rightarrow \infty$; any other scaling leads to vanishing or divergent outputs.
- **Sequence-doubling invariance:** Duplicating every token should not change the attention output; this property uniquely fixes the scaling exponent at $\alpha=1$.

**Statistical Sample Complexity:** The mixture-of-experts formalism demonstrates that sigmoid attention enjoys polynomial sample complexity for function classes where softmax gating requires exponential data—particularly for polynomial expert networks in the dense regime [2502.00281]. For instance, achieving $L_2$ error $\epsilon$ with polynomial experts requires $n = \tilde{O}(\epsilon^{-2})$ samples for sigmoid gating versus $n = \exp(\tilde{O}(\epsilon^{-1/\tau}))$ for softmax.

## 3. Empirical Results and Training Stability

Empirical evaluations corroborate the theoretical findings across several domains.

**Single-Cell Foundation Models:** On six held-out single-cell RNA-seq datasets, sigmoid attention achieves approximately $25\%$ higher cell-type separation (Maximum Mean Discrepancy), dominates on cohesion metrics (Leiden NMI, ARI), and yields systematically lower validation loss. Training is up to $10\%$ faster, with large-scale models (160M–1.4B parameters) demonstrating enhanced convergence and robustness (no catastrophic divergence observed in contrast to softmax attention) [2604.27124].

**Gradient Stability:** Stress tests (training at 8K token context, no gradient clipping) show that softmax attention models experience catastrophic divergence (gradient norms increase by $10^4\times$, attention logits reach $2 \times 10^8$), while sigmoid attention maintains stable gradients (range $10$–$100$) and bounded attention scores (≤ 5).

**Computational Efficiency:** Sigmoid attention obviates the need for row-wise normalization, enabling fully parallel element-wise operations. Throughput gains of $10$–$15\%$ on GPUs have been observed compared to softmax-based attention [2502.00281][2604.27124].

## 4. Applications: Biological, Graph, and Language Models

**Biological Foundation Models:** The adoption of sigmoid attention in biological foundation models has enabled stable training on large, variable-length sequences (e.g., single-cell transcriptomics), with kernel implementations (TritonSigmoid) achieving $515$ TFLOPS on H100 GPUs and demonstrating superior speed and padding support over FlashAttention-2 [2604.27124].

**Graph Transformers and Over-Smoothing:** In graph domains, sigmoid-gated attention (SigGate-GT) breaks the sum-to-one constraint of softmax, addressing over-smoothing and attention entropy degeneration. Element-wise learned gating with sigmoid allows each attention head to selectively silence uninformative connections, raising the effective rank of outputs and maintaining higher embedding diversity. On molecular benchmarks (ZINC, ogbg-molhiv), SigGate-GT achieves state-of-the-art results, with 30% reduction in over-smoothing and robust training across a $10\times$ learning rate range [2604.17324].

**Natural Language and General Sequence Modeling:** Sigmoid attention matches or outperforms softmax attention across varied NLP tasks, achieving sample-efficiency gains and removing token competition bias. Synthetic and real-world tasks (ARC, HellaSwag, etc.) confirm competitive or improved generalization with faster computation [2502.00281].

## 5. Implementation Practices and Hardware Considerations

**GPU Kernels:** The TritonSigmoid kernel leverages block-sparse execution and fused sigmoid calculations for efficiency. It natively supports arbitrary padding patterns essential for variable-length inputs, delivering 7.15$\times$ forward speedup over PyTorch baselines and maintaining performance when up to 25% of the sequence is padded [2604.27124]. Compared to FlashAttention-2, TritonSigmoid achieves 10–20% higher throughput, with negligible overhead from padding due to block skipping.

**Transformer Block Integration:** Replacing softmax attention in existing Transformer blocks involves substituting the normalization with element-wise sigmoid, introducing a bias term $b = -\log n$ to center the mean scale, and maintaining standard query/key scaling by $1/\sqrt{d_h}$. Standard initialization (e.g., Xavier uniform) and learning rate schedules remain effective. Owing to the bounded derivative of sigmoid, higher learning rates or omitting gradient clipping are often tolerable.

```python
import torch
import math

class SigmoidSelfAttention(torch.nn.Module):
    def __init__(self, dim, n_heads, max_sequence_length):
        super().__init__()
        self.nh = n_heads
        self.dk = dim // n_heads
        self.Wq = torch.nn.Linear(dim, dim, bias=False)
        self.Wk = torch.nn.Linear(dim, dim, bias=False)
        self.Wv = torch.nn.Linear(dim, dim, bias=False)
        self.bias = -torch.log(torch.tensor(max_sequence_length, dtype=torch.float))
        self.scale = 1.0 / math.sqrt(self.dk)

    def forward(self, x, attn_mask=None):
        b, n, d = x.shape
        q = self.Wq(x).view(b, n, self.nh, self.dk)
        k = self.Wk(x).view(b, n, self.nh, self.dk)
        v = self.Wv(x).view(b, n, self.nh, self.dk)
        scores = torch.einsum('bqhd,bkhd->bhqk', q, k)
        scores = scores * self.scale + self.bias
        if attn_mask is not None:
            scores = scores.masked_fill(attn_mask[:, None, None, :] == 0, float('-inf'))
        attn_weights = torch.sigmoid(scores)
        out = torch.einsum('bhqk,bkhd->bqhd', attn_weights, v)
        return out.reshape(b, n, d)
```
This direct code adaptation supports efficient, numerically stable sigmoid attention in production settings [2604.27124].

## 6. Sigmoid Gating Variants in Transformer Architectures

Beyond direct softmax-to-sigmoid substitution, sigmoid gating has been incorporated into attention as an auxiliary mechanism, particularly in graph domains [2604.17324]. In SigGate-GT, for each attention head $h$, a learned gate $g^{(h)} = \sigma(H W_g^{(h)} + b_g^{(h)})$ is computed per token, and the traditional softmax attention output $Y^{(h)}$ is post-multiplied element-wise by $g^{(h)}$. This gating enables heads to “say nothing” when no informative connections are present, breaking mandatory attention sinks and improving representational diversity, entropy, and depth stability. Ablation studies confirm that post-attention per-head gating is most effective, with only a 1% parameter overhead.

## 7. Modeling Implications, Limitations, and Open Questions

Sigmoid-based attention mechanisms fundamentally alter self-attention’s inductive bias by removing inter-token competition and enabling independent contribution scaling. This allows for higher sample efficiency, stability (via bounded derivatives and decoupled gradients), and improved representational richness (especially in molecular and graph domains). Computationally, element-wise sigmoid is more amenable to parallelism and less memory-bound than softmax.

Known limitations include the current theoretical bounds focusing primarily on single-head attention (the fully general multi-head MoE setting remains open) and the dependency of empirical results on correct scaling and bias calibration [2409.04431][2502.00281]. Open research directions include hybrid gating schemes, further multi-head theoretical analysis, and domain-specific optimizations.

---

**Key references:**  
- [2409.04431]  
- [2604.27124]  
- [2502.00281]  
- [2604.17324]

Source: https://www.emergentmind.com/topics/sigmoid-transformer