---
title: SwiGLU-Activated Feed-Forward Networks
url: https://www.emergentmind.com/topics/swiglu-activated-feed-forward-networks
type: topic
---

# SwiGLU-Activated Feed-Forward Networks

SwiGLU-Activated Feed-Forward Networks are a class of feed-forward neural architectures that integrate Swish gating within the Gated Linear Unit (GLU) framework, extensively deployed in state-of-the-art Transformer-based Large Language Models (LLMs). These architectures provide a mechanism for conditional gating by combining parallel feature projections and a data-dependent gate, thereby enhancing expressivity and controllability in deep models. Significant research has focused on both the representational capacity and hardware performance of SwiGLU-activated networks, with recent advances targeting their memory and compute efficiency and the potential for further generalization.

## 1. Architectural Foundations

The canonical GLU architecture replaces the traditional multilayer perceptron (MLP) up-projection and nonlinearity with two parallel linear transformations—the gate and value streams—whose outputs are multiplied element-wise. For GLU, given input $x \in \mathbb{R}^h$ and projection matrices $W_g, W_v \in \mathbb{R}^{h \times d}$, the formulation is
\[
\mathrm{GLU}(x) = g(xW_g) \odot (xW_v)
\]
where $g(\cdot)$ is a nonlinearity, typically sigmoid, and $\odot$ denotes the Hadamard product.

SwiGLU replaces the gating activation $g$ with the Swish (SiLU) activation:
\[
\mathrm{Swish}(z) = z \cdot \sigma(z), \qquad \sigma(z) = \frac{1}{1+e^{-z}}
\]
The output of a SwiGLU layer is thus
\[
\mathrm{SwiGLU}(x) = \mathrm{Swish}(x W_g) \odot (x W_v)
\]
followed by a linear output projection $W_o \in \mathbb{R}^{d \times h}$ [2506.23225].

SwiGLU-activated blocks are now the standard FFN component in many leading LLMs due to empirically superior convergence and accuracy.

## 2. Memory and Throughput Bottlenecks

GLU and SwiGLU blocks require loading two separate $h \times d$ weight matrices for every token during inference, inducing a $2\times$ memory-read penalty compared to non-gated FFNs. Attempting to share weights for gate and value projections severely limits expressivity. This doubles global memory bandwidth requirements—an acute constraint for hardware-bound inference scenarios.

Masked Gated Linear Units (MGLUs), through the Mixture of Element-wise Gating (MoEG) formulation, address this by learning $n_m$ binary masks $M_i \in \{0,1\}^{h\times d}$ over a shared matrix $W$, partitioning each entry to function as a gate or value component per route:
\[
\mathrm{SwiMGLU}_{n_m}(x) = \sum_{i=1}^{n_m} \left[ \mathrm{Swish}\big(x(M_i \odot W)\big) \odot \big(x((1-M_i) \odot W)\big) \right]
\]
This masking mechanism preserves distinctive gating paths without duplicating all weights, reducing total memory transfer and facilitating optimized computation [2506.23225].

## 3. Optimized Kernel Implementation

FlashMGLU is a highly efficient GPU kernel for MGLU/SwiMGLU evaluation. Key optimizations include:

- **Packed Masks:** All $n_m$ masks are packed into a byte per weight entry, enabling single memory transactions for co-located elements.
- **Tiled Weight Layout:** $W$ is partitioned into row×chunk tiles to maximize coalesced reads, minimizing latency.
- **In-Register Accumulation:** For each route, accumulation occurs entirely in registers, requiring just one atomic write per output row.

This kernel achieves up to 19.7× speed-up over the naïve PyTorch implementation (which performs $n_m$ serial masked matmuls), and delivers up to 47% lower memory bandwidth versus standard GLUs (using $16\,hd + n_m\,hd$ bits vs $32\,hd$ bits per token at FP16 precision) [2506.23225].

## 4. Empirical Performance and Accuracy

Empirical results demonstrate that SwiMGLU maintains or surpasses the downstream accuracy of standard SwiGLU on zero- and two-shot tasks across LLaMA-style models, with significant reductions in parameter count and memory usage.

| Model Size    | SwiGLU Params | SwiMGLU Params | Zero-shot Avg (%) | Two-shot Avg (%) |
|---------------|--------------|---------------|-------------------|------------------|
| Small (~141M) | 141M         | 113M          | 46.20 (SwiGLU) / 46.48 (SwiMGLU) | 45.52 / 46.40  |
| Large (~1.08B)| 1.08B        | 808M          | 56.00 / 56.85     | 57.36 / 57.87    |

Batch-1, FP16 inference with $h=2048, d=8192$ on an RTX 5090 yields per-layer latency of 0.0265 ms for FlashMGLU ($n_m=8$) versus 0.521 ms for PyTorch GLU, corresponding to token throughput of 37,700 tok/s versus 1,920 tok/s.

Memory load per layer is reduced from 96 MB to 64 MB in 1B-parameter LLaMA blocks, with masks contributing only a few additional MB [2506.23225].

## 5. Complexity Analysis

Let $h$ (hidden size) and $d$ (FFN intermediate size) denote standard transformer dimensions.

- **Compute (FLOPs/token):**
  - SwiGLU: $2 \times$ matmuls ($2 \times 2hd = 4hd$ multiplies), output matmul $2dh = 2hd$, totaling approximately $6hd$.
  - SwiMGLU ($n_m$ masks): $2(1 + n_m)hd$ per token. For $n_m=1$, $4hd$; for $n_m=4$, $10hd$.
- **Memory-read bits (FP16):**
  - SwiGLU: $32hd$.
  - SwiMGLU: $16hd + n_m hd$.

Maximum relative memory-load reduction for $n_m = 1$ is 47%. On memory-bound hardware, the increased arithmetic cost is offset by the reduction in data transfer [2506.23225].

## 6. Generalizations: PolyGLU and State-Conditional Routing

PolyGLU extends the SwiGLU FFN paradigm by enabling each neuron to select among multiple activations (ReLU, $\tanh$, SiLU, GELU) via an input-conditioned, Gumbel-Softmax-routed mechanism. Each neuron maintains static logits and scales, combined with an MLP-derived gate conditioned on mean token embeddings. This approach allows nearly deterministic per-neuron activation specialization, with depth-dependent patterns: early transformer layers prefer probabilistic gates (GELU), while deep layers exhibit a strong $\tanh$ preference.

PolychromaticLM, built using PolyGLU, achieves between 62–89% of the performance of a SwiGLU baseline on six standard benchmarks despite 3,600-fold less pretraining data. The parameter overhead from routing is minimal (0.23%) and can be eliminated at inference by freezing activations [2603.13347].

A plausible implication is that state-conditional routing, as in PolyGLU, provides a flexible, compact extension to SwiGLU-activated FFNs, with capacity for biologically-inspired specialization.

## 7. Practical Impact and Emerging Directions

SwiGLU-activated feed-forward networks represent a critical architectural and systems-level advancement in modern LLMs, balancing expressivity, training stability, hardware throughput, and memory efficiency. MoEG-based SwiMGLU demonstrates a new design point: eliminating redundant memory reads by fusing gate/value projection over a single shared matrix, at minimal or zero cost to accuracy, with principled mask learning.

Further generalizations—such as PolyGLU's input-conditioned activation mixtures—suggest dynamic per-neuron specialization is feasible with marginal parameter cost and full compatibility with fine-tuning and downstream transfer. The analogy to neurotransmitter diversity highlights a shift away from fixed-function feed-forward blocks toward flexible, interpretable, and convergence-robust designs.

Continued research will likely explore scalable, inference-efficient mask and routing learning, fixed-activation distillation from dynamic routers, and more interpretable gating mechanisms, with implications for both architecture search and hardware-aware model training [2506.23225, 2603.13347].

Source: https://www.emergentmind.com/topics/swiglu-activated-feed-forward-networks