SwiGLU-Activated Feed-Forward Networks

• The paper demonstrates that integrating Swish gating within the GLU framework enhances transformer convergence and reduces memory bandwidth requirements during inference.
• The methodology employs parallel linear projections and masked gating to achieve up to 19.7× speed-up and nearly 47% lower memory loads compared to standard FFNs.
• The approach paves the way for generalizations like PolyGLU, which offers dynamic, input-conditioned activation mixtures with minimal parameter overhead.

SwiGLU-Activated Feed-Forward Networks are a class of feed-forward neural architectures that integrate Swish gating within the Gated Linear Unit (GLU) framework, extensively deployed in state-of-the-art Transformer-based LLMs. These architectures provide a mechanism for conditional gating by combining parallel feature projections and a data-dependent gate, thereby enhancing expressivity and controllability in deep models. Significant research has focused on both the representational capacity and hardware performance of SwiGLU-activated networks, with recent advances targeting their memory and compute efficiency and the potential for further generalization.

1. Architectural Foundations

The canonical GLU architecture replaces the traditional multilayer perceptron (MLP) up-projection and nonlinearity with two parallel linear transformations—the gate and value streams—whose outputs are multiplied element-wise. For GLU, given input $x \in \mathbb{R}^h$ and projection matrices $W_g, W_v \in \mathbb{R}^{h \times d}$, the formulation is

$\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$

where $g(\cdot)$ is a nonlinearity, typically sigmoid, and $\odot$ denotes the Hadamard product.

SwiGLU replaces the gating activation $g$ with the Swish (SiLU) activation: $$\mathrm{Swish}(z) = z \cdot \sigma(z), \qquad \sigma(z) = \frac{1}{1+e^{-z}}$$ The output of a SwiGLU layer is thus

$$\mathrm{SwiGLU}(x) = \mathrm{Swish}(x W_g) \odot (x W_v)$$

followed by a linear output projection $W_o \in \mathbb{R}^{d \times h}$ (Tajima et al., 29 Jun 2025).

SwiGLU-activated blocks are now the standard FFN component in many leading LLMs due to empirically superior convergence and accuracy.

2. Memory and Throughput Bottlenecks

GLU and SwiGLU blocks require loading two separate $h \times d$ weight matrices for every token during inference, inducing a $W_g, W_v \in \mathbb{R}^{h \times d}$0 memory-read penalty compared to non-gated FFNs. Attempting to share weights for gate and value projections severely limits expressivity. This doubles global memory bandwidth requirements—an acute constraint for hardware-bound inference scenarios.

Masked Gated Linear Units (MGLUs), through the Mixture of Element-wise Gating (MoEG) formulation, address this by learning $W_g, W_v \in \mathbb{R}^{h \times d}$1 binary masks $W_g, W_v \in \mathbb{R}^{h \times d}$2 over a shared matrix $W_g, W_v \in \mathbb{R}^{h \times d}$3, partitioning each entry to function as a gate or value component per route: $W_g, W_v \in \mathbb{R}^{h \times d}$4 This masking mechanism preserves distinctive gating paths without duplicating all weights, reducing total memory transfer and facilitating optimized computation (Tajima et al., 29 Jun 2025).

3. Optimized Kernel Implementation

FlashMGLU is a highly efficient GPU kernel for MGLU/SwiMGLU evaluation. Key optimizations include:

• Packed Masks: All $W_g, W_v \in \mathbb{R}^{h \times d}$5 masks are packed into a byte per weight entry, enabling single memory transactions for co-located elements.
• Tiled Weight Layout: $W_g, W_v \in \mathbb{R}^{h \times d}$6 is partitioned into row×chunk tiles to maximize coalesced reads, minimizing latency.
• In-Register Accumulation: For each route, accumulation occurs entirely in registers, requiring just one atomic write per output row.

This kernel achieves up to 19.7× speed-up over the naïve PyTorch implementation (which performs $W_g, W_v \in \mathbb{R}^{h \times d}$7 serial masked matmuls), and delivers up to 47% lower memory bandwidth versus standard GLUs (using $W_g, W_v \in \mathbb{R}^{h \times d}$8 bits vs $W_g, W_v \in \mathbb{R}^{h \times d}$9 bits per token at FP16 precision) (Tajima et al., 29 Jun 2025).

4. Empirical Performance and Accuracy

Empirical results demonstrate that SwiMGLU maintains or surpasses the downstream accuracy of standard SwiGLU on zero- and two-shot tasks across LLaMA-style models, with significant reductions in parameter count and memory usage.

Model Size	SwiGLU Params	SwiMGLU Params	Zero-shot Avg (%)	Two-shot Avg (%)
Small (~141M)	141M	113M	46.20 (SwiGLU) / 46.48 (SwiMGLU)	45.52 / 46.40
Large (~1.08B)	1.08B	808M	56.00 / 56.85	57.36 / 57.87

Batch-1, FP16 inference with $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$0 on an RTX 5090 yields per-layer latency of 0.0265 ms for FlashMGLU ($\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$1) versus 0.521 ms for PyTorch GLU, corresponding to token throughput of 37,700 tok/s versus 1,920 tok/s.

Memory load per layer is reduced from 96 MB to 64 MB in 1B-parameter LLaMA blocks, with masks contributing only a few additional MB (Tajima et al., 29 Jun 2025).

5. Complexity Analysis

Let $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$2 (hidden size) and $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$3 (FFN intermediate size) denote standard transformer dimensions.

• Compute (FLOPs/token):
  • SwiGLU: $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$4 matmuls ($\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$5 multiplies), output matmul $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$6, totaling approximately $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$7.
  • SwiMGLU ($\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$8 masks): $\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)$9 per token. For $g(\cdot)$0, $g(\cdot)$1; for $g(\cdot)$2, $g(\cdot)$3.
• Memory-read bits (FP16):
  • SwiGLU: $g(\cdot)$4.
  • SwiMGLU: $g(\cdot)$5.

Maximum relative memory-load reduction for $g(\cdot)$6 is 47%. On memory-bound hardware, the increased arithmetic cost is offset by the reduction in data transfer (Tajima et al., 29 Jun 2025).

6. Generalizations: PolyGLU and State-Conditional Routing

PolyGLU extends the SwiGLU FFN paradigm by enabling each neuron to select among multiple activations (ReLU, $g(\cdot)$7, SiLU, GELU) via an input-conditioned, Gumbel-Softmax-routed mechanism. Each neuron maintains static logits and scales, combined with an MLP-derived gate conditioned on mean token embeddings. This approach allows nearly deterministic per-neuron activation specialization, with depth-dependent patterns: early transformer layers prefer probabilistic gates (GELU), while deep layers exhibit a strong $g(\cdot)$8 preference.

PolychromaticLM, built using PolyGLU, achieves between 62–89% of the performance of a SwiGLU baseline on six standard benchmarks despite 3,600-fold less pretraining data. The parameter overhead from routing is minimal (0.23%) and can be eliminated at inference by freezing activations (Medeiros, 7 Mar 2026).

A plausible implication is that state-conditional routing, as in PolyGLU, provides a flexible, compact extension to SwiGLU-activated FFNs, with capacity for biologically-inspired specialization.

7. Practical Impact and Emerging Directions

SwiGLU-activated feed-forward networks represent a critical architectural and systems-level advancement in modern LLMs, balancing expressivity, training stability, hardware throughput, and memory efficiency. MoEG-based SwiMGLU demonstrates a new design point: eliminating redundant memory reads by fusing gate/value projection over a single shared matrix, at minimal or zero cost to accuracy, with principled mask learning.

Further generalizations—such as PolyGLU's input-conditioned activation mixtures—suggest dynamic per-neuron specialization is feasible with marginal parameter cost and full compatibility with fine-tuning and downstream transfer. The analogy to neurotransmitter diversity highlights a shift away from fixed-function feed-forward blocks toward flexible, interpretable, and convergence-robust designs.

Continued research will likely explore scalable, inference-efficient mask and routing learning, fixed-activation distillation from dynamic routers, and more interpretable gating mechanisms, with implications for both architecture search and hardware-aware model training (Tajima et al., 29 Jun 2025, Medeiros, 7 Mar 2026).