Modified SwiGLU: Efficiency & Stability

• Modified SwiGLU is a refinement of the original SwiGLU, introducing variants like SwiMGLU and Smooth-SwiGLU to enhance memory efficiency, computational throughput, and training stability.
• SwiMGLU consolidates gating and value projections via complementary binary masks, reducing memory traffic and cutting weight requirements by up to 30% while accelerating inference.
• Smooth-SwiGLU employs per-channel scaling to clamp activations in FP8 quantization, effectively preventing outlier amplification and ensuring stable training over extended token counts.

Modified SwiGLU encompasses a set of advancements to the original Swish-Gated Linear Unit (SwiGLU) formulation aimed at improving memory efficiency, computational throughput, and training stability in large-scale neural networks. Two major lines of modification—SwiMGLU and Smooth-SwiGLU—systematically address the observed hardware inefficiencies and quantization instability of standard SwiGLU, enabling its deployment in latency- and bandwidth-constrained scenarios as well as in full FP8-quantized training regimes (Tajima et al., 29 Jun 2025, Fishman et al., 2024).

1. Mathematical Formulations and Variants

Standard SwiGLU

Given an input $x\in\mathbb R^d$, and "up-projection" matrices $W_g,W_v\in\mathbb{R}^{d\times h}$, with Swish activation $\mathrm{Swish}(t)=t\sigma(t)$ where $\sigma(t)=1/(1+e^{-t})$, the SwiGLU is defined as: $$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$ The typical forward update with output projection $W_o\in\mathbb{R}^{h\times d}$ and residual connection is: $$y_{\mathrm{SwiGLU}} = x + \left( (x W_g) \odot \mathrm{Swish}(x W_v) \right) W_o$$

SwiMGLU (Masked Gated Linear Unit, Swish-activated)

MGLU replaces $W_g, W_v$ with a single full-rank $W\in\mathbb{R}^{d\times h}$ and $n_m$ elementwise binary masks $W_g,W_v\in\mathbb{R}^{d\times h}$0, each with complementary $W_g,W_v\in\mathbb{R}^{d\times h}$1. For each "route" $W_g,W_v\in\mathbb{R}^{d\times h}$2: $W_g,W_v\in\mathbb{R}^{d\times h}$3 The Mixture-of-Element-wise-Gating (MoEG) combines these as: $W_g,W_v\in\mathbb{R}^{d\times h}$4 with the final update: $W_g,W_v\in\mathbb{R}^{d\times h}$5

Smooth-SwiGLU

Under FP8 quantization, SwiGLU intermediate activations become unstable due to outlier amplification via weight alignment. Smooth-SwiGLU applies a per-channel scalar $W_g,W_v\in\mathbb{R}^{d\times h}$6 to clamp and later rescale the problematic branch, yielding for neuron/channel $W_g,W_v\in\mathbb{R}^{d\times h}$7: $W_g,W_v\in\mathbb{R}^{d\times h}$8 where $W_g,W_v\in\mathbb{R}^{d\times h}$9 is the FP8 quantization operator. The real-valued function remains unaltered, but all intermediates are guaranteed to reside within the representable FP8 dynamic range (Fishman et al., 2024).

2. Architectural and Implementation Differences

SwiGLU employs two independent $\mathrm{Swish}(t)=t\sigma(t)$0 projection matrices for the gate and value streams, doubling memory traffic compared to standard feed-forward layers. At inference, SwiGLU thus demands $\mathrm{Swish}(t)=t\sigma(t)$1 FP16 reads per token.

SwiMGLU consolidates gating and value projections by leveraging a single $\mathrm{Swish}(t)=t\sigma(t)$2 FP16 matrix plus $\mathrm{Swish}(t)=t\sigma(t)$3 binary masks, typically $\mathrm{Swish}(t)=t\sigma(t)$4. During training, masks are stored as real-valued logits, updated via backpropagation and binarized with a straight-through estimator. In inference, binary masks are fused with $\mathrm{Swish}(t)=t\sigma(t)$5 in a custom kernel (FlashMGLU) that coalesces memory reads and computations, minimizing data transfer and mat-vec operations (Tajima et al., 29 Jun 2025).

Smooth-SwiGLU focuses on numerical stability. It scales and then inversely rescales the pre-quantized activations, so it can be implemented with minimal computational cost: during inference, the scale factors $\mathrm{Swish}(t)=t\sigma(t)$6 can be absorbed into the first and third linear-layer weights, resulting in zero additional inference overhead (Fishman et al., 2024).

3. Performance Characteristics

Inference-time Speed and Memory Compression

Under FP16 on an NVIDIA RTX 5090 (h=8192, d=2048):

• Naïve PyTorch MGLU ($\mathrm{Swish}(t)=t\sigma(t)$7): $\mathrm{Swish}(t)=t\sigma(t)$8 ms per matvec,
• FlashMGLU ($\mathrm{Swish}(t)=t\sigma(t)$9): $\sigma(t)=1/(1+e^{-t})$0 ms, a $\sigma(t)=1/(1+e^{-t})$1 speed-up,
• PyTorch GLU: $\sigma(t)=1/(1+e^{-t})$2 ms; FlashMGLU is $\sigma(t)=1/(1+e^{-t})$3 faster than standard GLU.

SwiMGLU requires at most $\sigma(t)=1/(1+e^{-t})$4 bits, reducing memory transfer by up to $\sigma(t)=1/(1+e^{-t})$5 for $\sigma(t)=1/(1+e^{-t})$6 and cutting per-layer storage from $\sigma(t)=1/(1+e^{-t})$7 MB to $\sigma(t)=1/(1+e^{-t})$8 MB (plus $\sigma(t)=1/(1+e^{-t})$9 MB mask overhead) for a Llama-1B FFN layer ($$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$0) (Tajima et al., 29 Jun 2025).

Downstream Task Accuracy

Across six benchmarks (zero-shot and two-shot):

Model & Scale	Zero-shot (%)	Two-shot (%)	Weights
SwiGLU ($$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$1)	46.20	45.52	141M
SwiMGLU ($$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$2)	46.48	46.40	113M+mask
SwiGLU ($$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$3)	56.00	57.36	1.08B
SwiMGLU ($$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$4)	56.85	57.87	808M+mask

This demonstrates that SwiMGLU matches or slightly exceeds standard SwiGLU accuracy using $$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$5 fewer weights (Tajima et al., 29 Jun 2025).

Quantized Training Stability and Throughput

With full FP8 training:

• Standard FP8+SwiGLU diverges after $$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$6B tokens due to quadratic outlier amplification.
• FP8+Smooth-SwiGLU remains as stable as the BF16 baseline for at least $$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$7B tokens.
• Throughput: FP8+Smooth-SwiGLU achieves $$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$8 samples/sec (on 8 Gaudi2 cards, micro-batch 1), a $$h_{\mathrm{SwiGLU}} = (x W_g) \odot \mathrm{Swish}(x W_v)$$9 improvement over BF16 (Fishman et al., 2024).

4. Theoretical Analysis: Weight Alignment and Outlier Amplification

SwiGLU's two projection vectors $W_o\in\mathbb{R}^{h\times d}$0 and $W_o\in\mathbb{R}^{h\times d}$1 exhibit $W_o\in\mathbb{R}^{h\times d}$2-regularized alignment over prolonged training. At convergence, if the Swish activation saturates ($W_o\in\mathbb{R}^{h\times d}$3), KKT conditions enforce $W_o\in\mathbb{R}^{h\times d}$4. The network's output then approximates $W_o\in\mathbb{R}^{h\times d}$5, causing moderate increases in $W_o\in\mathbb{R}^{h\times d}$6 to be squared, amplifying outliers. In low-precision regimes, such as FP8, this produces spikes far outside the dynamic range, breaking delayed scaling assumptions and leading to loss divergence after extended training ($W_o\in\mathbb{R}^{h\times d}$7B tokens) (Fishman et al., 2024). Smooth-SwiGLU, by range-limiting the branch prior to quantization, effectively blocks this failure mode.

5. Deployment Considerations and Practical Takeaways

SwiMGLU is most advantageous in settings with memory or latency constraints. Its memory savings and inference acceleration (up to $W_o\in\mathbb{R}^{h\times d}$8 faster than naive MGLU, $W_o\in\mathbb{R}^{h\times d}$9 versus standard GLU) make it suitable for edge, HBM-limited servers, and mobile deployments, without sacrificing downstream accuracy or necessitating retraining (Tajima et al., 29 Jun 2025).

Smooth-SwiGLU is a direct drop-in replacement for SwiGLU in full FP8 pipelines, providing theoretical and empirical guarantees for training stability even across trillion-token, multi-hundred-billion-parameter scales. It achieves this without changing the functional map or introducing any measurable impact on final model quality (Fishman et al., 2024).

6. Implementation Notes and Hyperparameter Settings

For SwiMGLU:

• Training employs real-valued mask logits updated by backpropagation; masks are binarized using the straight-through estimator.
• FlashMGLU kernel coalesces memory reads and on-chip computation for maximal efficiency.

For Smooth-SwiGLU:

• FP8 formats "E4M3" and "E5M2" are used for activations/weights and gradients, respectively.
• Adam moments are quantized (first: E4M3, second: E5M2).
• Per-channel scales $$y_{\mathrm{SwiGLU}} = x + \left( (x W_g) \odot \mathrm{Swish}(x W_v) \right) W_o$$0 are computed via max-abs statistics over calibration minibatches; at inference, these are incorporated into the static weights (no runtime penalty).
• Other hyperparameters (learning rate, decay, layer norm, etc.) follow Llama 2 defaults (Fishman et al., 2024).

7. Significance and Outlook

Modified SwiGLU architectures—including SwiMGLU and Smooth-SwiGLU—resolve the two principal limitations of the standard SwiGLU: inefficient memory bandwidth and quantization-induced instability. SwiMGLU enables large-scale LLMs to meet hardware and deployment constraints, while Smooth-SwiGLU extends SwiGLU's applicability to ultra-low-precision domains such as FP8 training. These advances preserve or improve downstream performance at reduced computational and storage cost, providing robust building blocks for the next generation of efficient LLMs (Tajima et al., 29 Jun 2025, Fishman et al., 2024).