Spark FFN: Sparse Transformer Efficiency

• Spark FFN is a variant of feed-forward networks that employs top-k masking, statistical thresholding, and parameter-efficient gating to enforce activation sparsity in Transformers.
• The design systematically reduces per-token FLOPs and wall-time by leveraging the lazy neuron phenomenon without compromising model quality or standard training dynamics.
• It integrates a linear-time differentiable top-k approximation and predictor-value split to achieve hardware-friendly sparse computations in both training and inference.

Spark FFN is a feed-forward network (FFN) variant for Transformers that explicitly exploits activation sparsity by means of top-$k$ masking, statistical thresholding, and parameter-efficient gating. Developed in the context of the Spark Transformer architecture, Spark FFN systematically reduces computational cost in both training and inference, achieving significant FLOPs and wall-time reductions while preserving model quality and training dynamics. The core innovation lies in scalable, hardware-friendly sparsification of FFN activations and a low-cost, differentiable predictor for selecting active neurons, thereby reactivating “lazy neuron” sparsity in modern Transformer models (You et al., 7 Jun 2025).

1. Background: Standard Transformer FFN and the Lazy Neuron Phenomenon

In the canonical Transformer layer, the FFN processes each token embedding $x\in\mathbb{R}^{d_{\text{model}}}$ using a two-layer MLP:

$y = \left(x W_1\right)~\sigma~ W_2$

with $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$, $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$, and a nonlinearity $\sigma(\cdot)$ such as GELU or ReLU. This requires approximately $4 d_{\text{model}} d_{\text{ff}}$ FLOPs per token.

Li et al. (2022) identified the “lazy neuron” phenomenon: when $\sigma =$ ReLU, only a small subset of the $d_{\text{ff}}$ hidden units have nonzero activations per token. This intrinsic sparsity allows, in principle, for skipping $W_2$ multiplications on inactive neurons, reducing some computation but not the initial $x\in\mathbb{R}^{d_{\text{model}}}$0 product. The challenge is to efficiently and explicitly harness this per-token sparsity without degrading model quality or increasing parameter count (You et al., 7 Jun 2025).

2. Explicit Sparsification via Top-$x\in\mathbb{R}^{d_{\text{model}}}$1 Masking

Spark FFN enforces sparsity by selecting the top-$x\in\mathbb{R}^{d_{\text{model}}}$2 activations from the pre-nonlinearity score vector $x\in\mathbb{R}^{d_{\text{model}}}$3. The masking process is:

• Compute $x\in\mathbb{R}^{d_{\text{model}}}$4 where $x\in\mathbb{R}^{d_{\text{model}}}$5,
• Compute $x\in\mathbb{R}^{d_{\text{model}}}$6,
• Propagate via $x\in\mathbb{R}^{d_{\text{model}}}$7.

Here, $x\in\mathbb{R}^{d_{\text{model}}}$8 retains only the $x\in\mathbb{R}^{d_{\text{model}}}$9 largest elements (per token), annihilating the rest. If $y = \left(x W_1\right)~\sigma~ W_2$0, the final multiplication $y = \left(x W_1\right)~\sigma~ W_2$1 is reduced to $y = \left(x W_1\right)~\sigma~ W_2$2 FLOPs from $y = \left(x W_1\right)~\sigma~ W_2$3. However, $y = \left(x W_1\right)~\sigma~ W_2$4 selection by sorting is $y = \left(x W_1\right)~\sigma~ W_2$5 and is non-differentiable, necessitating an efficient relaxation (You et al., 7 Jun 2025).

3. Statistical Top-$y = \left(x W_1\right)~\sigma~ W_2$6: Linear-Time Differentiable Masking

To overcome the inefficiency of exact Top-$y = \left(x W_1\right)~\sigma~ W_2$7, Spark FFN introduces the “statistical Top-$y = \left(x W_1\right)~\sigma~ W_2$8” operator, which approximates Top-$y = \left(x W_1\right)~\sigma~ W_2$9 selection in linear time and is differentiable almost everywhere. For a vector $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$0 and target $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$1:

• Compute $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$2, where $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$3 is the standard Gaussian quantile function,
• Apply soft-thresholding: $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$4.

This procedure sets elements below the threshold to zero and for others subtracts $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$5. Since mean and standard deviation are $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$6 and soft-thresholding is elementwise, the total computation is $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$7. Empirically, the approach ensures $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$8 surviving entries under a Gaussian fit assumption, which is supported for FFN pre-activations in practice (You et al., 7 Jun 2025).

4. Predictor-Value Decomposition and Efficient Sparse Computation

Spark FFN partitions the input and first FFN layer for further efficiency. The weight matrix $$W_1\in\mathbb{R}^{d_{\text{model}}\times d_{\text{ff}}}$$9 and input $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$0 are split:

• Predictor block: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$1, operating on $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$2,
• Value block: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$3, operating on $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$4.

The mechanism is:

• Compute predictor scores: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$5,
• Build mask: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$6,
• Compute values: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$7,
• Select and activate: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$8,
• Output: $$W_2\in\mathbb{R}^{d_{\text{ff}}\times d_{\text{model}}}$$9.

Because $\sigma(\cdot)$0 is $\sigma(\cdot)$1-sparse, the expensive projections and matrix multiplications can be performed efficiently as sparse vector-matrix products. The predictor block’s cost is $\sigma(\cdot)$2; value block and output cost $\sigma(\cdot)$3 and $\sigma(\cdot)$4, respectively. Optimal empirical performance occurs at $\sigma(\cdot)$5 (You et al., 7 Jun 2025).

5. Measured Efficiency, Sparsity, and Model Quality

On a 2B-parameter Transformer pretrained according to the Gemma-2 recipe, Spark FFN achieves:

• Activation sparsity: $\sigma(\cdot)$6 (e.g., $\sigma(\cdot)$7 with $\sigma(\cdot)$8),
• End-to-end per-token FLOPs reduction: $\sigma(\cdot)$9 (72% in FFN, 75% in attention dot-products),
• Decoding speedup: up to $4 d_{\text{model}} d_{\text{ff}}$0 on 4-core CPU (prefill $4 d_{\text{model}} d_{\text{ff}}$1, decode $4 d_{\text{model}} d_{\text{ff}}$2), up to $4 d_{\text{model}} d_{\text{ff}}$3 on NVIDIA T4 GPU,
• No change to the optimizer, learning rate schedule, or pretraining curriculum.

Empirically, this procedure yields near-zero impact on pretraining loss or downstream quality, and enables hardware-efficient implementations using specialized sparse kernels (You et al., 7 Jun 2025).

6. Architectures, Hyper-Parameters, and Implementation

Key configurable aspects include:

• $4 d_{\text{model}} d_{\text{ff}}$4: Active neurons per token ($4 d_{\text{model}} d_{\text{ff}}$5 of $4 d_{\text{model}} d_{\text{ff}}$6 in Gemma-2 models),
• $4 d_{\text{model}} d_{\text{ff}}$7: Predictor width, optimal at $4 d_{\text{model}} d_{\text{ff}}$8 (multiple of $4 d_{\text{model}} d_{\text{ff}}$9 per Gemma-2 constraint),
• Thresholding: Statistical Top-$\sigma =$0 is parameter-free, almost everywhere differentiable, and $\sigma =$1,
• Training: No modifications to standard Transformer training pipeline.

In attention, a similar predictor-value split is employed, with $\sigma =$2 halved ($\sigma =$3), and per-token attention restricted to the top $\sigma =$4 keys.

7. Pseudocode and Integration in Transformer Layers

The Spark FFN forward pass for a single token $\sigma =$5 involves:

$d_{\text{ff}}$0

Gradients flow through the statistical Top-$\sigma =$6 everywhere except at zero crossings. Inference reuses the same forward pass, with per-token computation dropping from $\sigma =$7 FLOPs to $\sigma =$8. Practical implementations employ specialized SIMD/tiling/CUDA kernels for memory and compute efficiency (You et al., 7 Jun 2025).

---

Spark FFN reactivates latent activation sparsity in Transformer FFNs by combining explicit top-$\sigma =$9 masking, scalable thresholding, and efficient parameter reuse, resulting in substantial computational savings and wall-time improvements without compromising model quality or standard training dynamics.