SigGate-GT: Sigmoid-Gated Graph Transformer

• SigGate-GT is a graph transformer architecture that integrates learned, per-head sigmoid gates into the global attention module to selectively suppress non-informative signals.
• It applies an element-wise sigmoid gate after softmax attention to break the compulsory sum-to-one constraint, thereby increasing the effective rank of head outputs and alleviating entropy degeneration.
• Benchmark evaluations show that SigGate-GT outperforms GraphGPS on metrics like MAE and ROC-AUC across multiple datasets, all while incurring minimal parameter and runtime overhead.

SigGate-GT is a graph transformer architecture that modifies the global attention component of GraphGPS by applying learned, per-head sigmoid gates to the attention output, with the stated objective of taming over-smoothing and attention entropy degeneration in deep graph transformers (Guo et al., 19 Apr 2026). The method is motivated by the observation that these pathologies share a root cause with attention sinks in LLMs: softmax attention imposes a sum-to-one constraint, so every node must attend somewhere even when no informative signal exists. Within that framing, SigGate-GT preserves the GraphGPS recipe—local message passing, global self-attention, and positional or structural encodings—while augmenting each head with an input-dependent mechanism that can suppress activations toward zero.

1. Placement within the GraphGPS architecture

GraphGPS is presented as a modular recipe in which each layer $\ell$ combines three ingredients: a local message-passing block, typically a GatedGCN; a global self-attention block, implemented by standard multi-head self-attention; and a position- and structure-encoding pipeline, exemplified by LapPE + RWSE. For node embedding $h_i^{(\ell-1)}$, a GPS layer computes

$$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$

SigGate-GT is defined as a minimal modification of the $\mathrm{GlobalAttn}$ block. After each head computes its standard softmax-attention output $A_kV_k$, SigGate-GT applies an element-wise, input-dependent sigmoid gate before the final projection. The surrounding GraphGPS flow—MPNN branch, attention branch, residual pathway, and MLP—remains unchanged. In the formulation given for the method, the modification is therefore local to the attention output rather than a redesign of the full graph transformer stack (Guo et al., 19 Apr 2026).

This placement is significant because the method is explicitly framed as an intervention on a failure mode of softmax-based global attention rather than on local message passing or positional encoding. A plausible implication is that the reported effects on over-smoothing and training stability are meant to be interpreted as consequences of altered attention dynamics inside an otherwise standard GraphGPS instantiation.

2. Sigmoid-gated attention mechanism

For head $k$, standard self-attention is written as

$$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$

$$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$

SigGate-GT adds a learned sigmoid gate $g_k$ of shape $(n \times d_k)$:

$h_i^{(\ell-1)}$0

where $h_i^{(\ell-1)}$1, $h_i^{(\ell-1)}$2, and $h_i^{(\ell-1)}$3 is applied element-wise. The gated head output is then

$h_i^{(\ell-1)}$4

followed by the usual concatenation and output projection:

$h_i^{(\ell-1)}$5

The exposition also gives an equivalent element-wise view. If

$h_i^{(\ell-1)}$6

then

$h_i^{(\ell-1)}$7

with

$h_i^{(\ell-1)}$8

The residual update at node $h_i^{(\ell-1)}$9 is correspondingly written as

$$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$0

Initialization is specified: $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$1 is set to $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$2, so gates start near $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$3, “ensuring neither full suppression nor saturation at start” (Guo et al., 19 Apr 2026). This detail matters because the method attributes part of its training behavior to the gate’s ability to attenuate outputs without beginning in a collapsed regime.

3. Proposed explanation for over-smoothing, entropy degeneration, and attention sinks

The central claim is that over-smoothing and attention entropy degeneration in graph transformers share a root cause with attention sinks in LLMs: softmax attention forces every head to distribute probability mass somewhere. Under this view, the problem is not only that attention may become overly concentrated, but that the normalization itself enforces nonzero allocation even when informative signal is absent (Guo et al., 19 Apr 2026).

The method’s first rationale is therefore “breaking the must-attend-somewhere constraint.” Because softmax weights are non-negative and sum to one, every head must attend somewhere. SigGate-GT preserves softmax in the computation of $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$4, but then applies a sigmoid gate to $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$5, allowing $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$6 along certain dimensions and thereby letting heads selectively silence unhelpful outputs. The model is thus not described as replacing softmax normalization; it is described as adding a post-attention suppression mechanism.

A second rationale concerns the effective rank of per-head outputs. The exposition states that the matrix $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$7 has stable rank bounded by $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$8, and that under sharp softmax this can be very low. Since $$h_i^{(\ell)} = \mathrm{MLP}^{(\ell)}\!\left( h_i^{(\ell-1)} + \mathrm{MPNN}^{(\ell)}(h^{(\ell-1)}, E) + \mathrm{GlobalAttn}^{(\ell)}(h^{(\ell-1)}) \right).$$9 can no longer be written as a row-stochastic reweighting of $\mathrm{GlobalAttn}$0, the rank bound is lifted. In a calibrated synthetic study with $\mathrm{GlobalAttn}$1, $\mathrm{GlobalAttn}$2, and $\mathrm{GlobalAttn}$3 heads, stable rank increases by approximately $\mathrm{GlobalAttn}$4 on average, from $\mathrm{GlobalAttn}$5 to $\mathrm{GlobalAttn}$6.

A third rationale is the connection to over-smoothing. Repeated attention is described as acting like Laplacian smoothing, driving node embeddings toward a common vector. By gating certain dimensions to zero, SigGate-GT is said to provide a learned per-dimension “skip” path around smoothing, preserving diversity. The abstract summarizes the empirical effect as a $\mathrm{GlobalAttn}$7 reduction in over-smoothing, measured as mean relative MAD gain across $\mathrm{GlobalAttn}$8–$\mathrm{GlobalAttn}$9 layers.

The fourth rationale is training stability. The account given is that softmax outputs can become very large or very small, producing exploding or vanishing gradients, whereas the sigmoid derivative

$A_kV_k$0

lies in $A_kV_k$1, which caps gradient magnitudes and regularizes flow. This suggests that the gating mechanism is intended to function simultaneously as a representational and optimization-level intervention.

4. Benchmark performance

The reported experiments use five standard benchmarks, with all results given as mean $A_kV_k$2 standard deviation over $A_kV_k$3 seeds and paired $A_kV_k$4-tests versus the GraphGPS baseline (Guo et al., 19 Apr 2026).

Dataset and metric	GraphGPS	SigGate-GT
ZINC (MAE $A_kV_k$5)	$A_kV_k$6	$A_kV_k$7
ogbg-molhiv (ROC-AUC $A_kV_k$8)	$A_kV_k$9	$k$0
ogbg-molpcba (AP $k$1)	$k$2	$k$3
Pep-func (AP $k$4)	$k$5	$k$6
Pep-struct (MAE $k$7)	$k$8	$k$9

On ZINC, SigGate-GT matches the prior best result of GRIT at $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$0, while improving over GraphGPS with $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$1. Exphormer is reported at $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$2. On ogbg-molhiv, the method sets a new state of the art at $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$3 ROC-AUC, compared with $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$4 for GraphGPS, with $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$5. On ogbg-molpcba, it improves AP from $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$6 to $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$7, with $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$8.

For the Long-Range Graph Benchmark, the results are differentiated by task. On Pep-struct, SigGate-GT achieves the best reported number, $$Q_k = HW_k^Q,\qquad K_k = HW_k^K,\qquad V_k = HW_k^V,$$9 MAE, compared with $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$0 for GraphGPS and $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$1 for GRIT. On Pep-func, however, GRIT remains best at $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$2, while SigGate-GT reaches $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$3 and GraphGPS $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$4.

The statistical significance summary is explicit: across all five datasets, SigGate-GT improves over GraphGPS with $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$5; against the next-best non-GPS baseline, it is significant on ZINC, molhiv, molpcba, and Pep-struct, but not on Pep-func, where GRIT holds the lead. This is important for delimiting the method’s empirical scope: the reported evidence supports consistent gains over GraphGPS, but not uniform dominance over every competing architecture.

5. Ablations and diagnostic evidence

The ablation study on gate placement and sharing distinguishes several variants. G1, defined as post-SDPA output gating with per-head gates, is reported as best, achieving $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$6 MAE on ZINC and $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$7 AUC on molhiv. G2, value gating, is the next best, with $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$8 MAE and $$A_k = \mathrm{softmax}\!\left(\frac{Q_kK_k^\top}{\sqrt{d_k}}\right), \qquad \mathrm{head}_k = A_kV_k.$$9. G3, pre-softmax logit gating, is described as worse than no gate. Shared gating, in which one gate is used for all heads, degrades performance relative to per-head gating (Guo et al., 19 Apr 2026).

Over-smoothing is measured by MAD on ZINC across depth:

Depth	GraphGPS MAD / MAE	SigGate-GT MAD / MAE
4	$g_k$0 / $g_k$1	$g_k$2 / $g_k$3
8	$g_k$4 / $g_k$5	$g_k$6 / $g_k$7
10	$g_k$8 / $g_k$9	$(n \times d_k)$0 / $(n \times d_k)$1
12	$(n \times d_k)$2 / $(n \times d_k)$3	$(n \times d_k)$4 / $(n \times d_k)$5
16	$(n \times d_k)$6 / $(n \times d_k)$7	$(n \times d_k)$8 / $(n \times d_k)$9

These values are used to support the claim that gating mitigates over-smoothing. The abstract condenses this as a $h_i^{(\ell-1)}$00 reduction in over-smoothing, measured by mean relative MAD gain across $h_i^{(\ell-1)}$01–$h_i^{(\ell-1)}$02 layers. The depth sweep also shows that the performance gap widens at larger depth, which suggests that the mechanism is particularly targeted at depth-dependent representation collapse rather than shallow-layer optimization alone.

Attention entropy is reported on ZINC as per-row average entropy $h_i^{(\ell-1)}$03 across layers $h_i^{(\ell-1)}$04, $h_i^{(\ell-1)}$05, and $h_i^{(\ell-1)}$06:

Layer	GraphGPS $h_i^{(\ell-1)}$07	SigGate-GT $h_i^{(\ell-1)}$08
1	$h_i^{(\ell-1)}$09	$h_i^{(\ell-1)}$10
5	$h_i^{(\ell-1)}$11	$h_i^{(\ell-1)}$12
10	$h_i^{(\ell-1)}$13	$h_i^{(\ell-1)}$14

The higher entropy values are presented as evidence that the gating mechanism counteracts attention entropy degeneration. Because the gate is applied after softmax attention rather than before, a plausible implication is that the method changes the effective information flow without requiring a different normalization rule for $h_i^{(\ell-1)}$15 itself.

Training stability is evaluated with a $h_i^{(\ell-1)}$16 learning-rate sweep on ZINC, with test MAE reported for $h_i^{(\ell-1)}$17. GraphGPS yields $h_i^{(\ell-1)}$18, $h_i^{(\ell-1)}$19, $h_i^{(\ell-1)}$20, $h_i^{(\ell-1)}$21, and $h_i^{(\ell-1)}$22, for a range of $h_i^{(\ell-1)}$23. SigGate-GT yields $h_i^{(\ell-1)}$24, $h_i^{(\ell-1)}$25, $h_i^{(\ell-1)}$26, $h_i^{(\ell-1)}$27, and $h_i^{(\ell-1)}$28, for a range of $h_i^{(\ell-1)}$29. The text summarizes this by stating that SigGate-GT’s performance range is $h_i^{(\ell-1)}$30 smaller, demonstrating far less sensitivity to learning-rate choice.

6. Hyperparameters, overhead, and interpretive boundaries

The key training configurations are stated per benchmark family. For ZINC, the setup is $h_i^{(\ell-1)}$31 layers, $h_i^{(\ell-1)}$32, $h_i^{(\ell-1)}$33, $h_i^{(\ell-1)}$34 epochs, batch size $h_i^{(\ell-1)}$35, learning rate $h_i^{(\ell-1)}$36, and weight decay $h_i^{(\ell-1)}$37. For OGB, it is $h_i^{(\ell-1)}$38 layers, $h_i^{(\ell-1)}$39, $h_i^{(\ell-1)}$40, $h_i^{(\ell-1)}$41 epochs, batch size $h_i^{(\ell-1)}$42, learning rate $h_i^{(\ell-1)}$43, and weight decay $h_i^{(\ell-1)}$44. For LRGB, it is $h_i^{(\ell-1)}$45 layers, $h_i^{(\ell-1)}$46, $h_i^{(\ell-1)}$47, $h_i^{(\ell-1)}$48 epochs, batch size $h_i^{(\ell-1)}$49, learning rate $h_i^{(\ell-1)}$50, and weight decay $h_i^{(\ell-1)}$51. Positional encoding is RWSE(16)+LapPE(16) via SignNet, the optimizer is AdamW with cosine annealing, and the gate bias initialization is $h_i^{(\ell-1)}$52 (Guo et al., 19 Apr 2026).

The reported overhead is small but explicit. On OGB, for $h_i^{(\ell-1)}$53, $h_i^{(\ell-1)}$54, and $h_i^{(\ell-1)}$55, gating adds approximately $h_i^{(\ell-1)}$56 parameters, or about $h_i^{(\ell-1)}$57. On ZINC, for $h_i^{(\ell-1)}$58, $h_i^{(\ell-1)}$59, and $h_i^{(\ell-1)}$60, it adds approximately $h_i^{(\ell-1)}$61 parameters toward the $h_i^{(\ell-1)}$62 budget, or about $h_i^{(\ell-1)}$63. Runtime overhead is reported as $h_i^{(\ell-1)}$64 on an NVIDIA A100, with peak memory unchanged within rounding.

Two boundaries of interpretation are especially clear in the reported results. First, SigGate-GT is not presented as removing softmax attention; the attention matrix $h_i^{(\ell-1)}$65 remains softmax-based, and the intervention is an element-wise sigmoid gate on the head output. Second, the empirical case is strongest for improvement over GraphGPS, not for unconditional superiority over all alternatives. The method matches the prior best on ZINC, sets a new state of the art on ogbg-molhiv, and leads on Pep-struct, but GRIT remains ahead on Pep-func. Within those limits, the paper characterizes SigGate-GT as a lightweight enhancement to GraphGPS’s attention block that increases effective head rank, mitigates over-smoothing, raises attention entropy, and stabilizes training while incurring only about $h_i^{(\ell-1)}$66 parameter overhead and at most a few percent wall-clock cost.