---
title: Efficient Feed-Forward Networks
url: https://www.emergentmind.com/topics/efficient-feed-forward-network-effn
type: topic
---

# Efficient Feed-Forward Networks

An Efficient Feed-Forward Network (EFFN) is a family of architectural and algorithmic innovations that dramatically improve the computational, memory, and deployment efficiency of feed-forward neural networks, with a particular emphasis on modern deep architectures such as Transformers, Mixture-of-Experts, and dense multi-layer perceptrons. EFFN designs exploit architectural redundancy, conditional computation, explicit algebraic optimization, and adaptive parameter compression to either substitute or enhance conventional dense feed-forward modules. Empirical evidence demonstrates that EFFNs can achieve substantial speedups, aggressive model compression, and even accuracy improvements relative to standard baselines, across domains including language modeling, vision, and general supervised learning.

## 1. Motivations and Core Principles

Feed-forward networks constitute the bulk of learnable parameters and floating-point operations in deep neural architectures; for instance, Transformer-style FFNs account for ∼2/3 of parameters and latency in state-of-the-art language models [2401.04044]. Historically, these modules exhibit high degrees of redundancy—either across depth (multiple FFNs per layer with shared or similar functions), within width (many neurons remain idle or underutilized per input), or in parameterization (over-parameterized networks yielding compressible solutions). EFFN frameworks seek to systematically eliminate this redundancy through one or more of:

- Parameter sharing across layers and/or tokens (shared wide FFNs)
- Conditional activation and routing (tree-based FFF/EFFN, expert selection)
- Heavy-hitter neuron identification and selective resource allocation
- Model pruning with transfer learning for width/depth reduction
- Closed-form or explicit algebraic solutions for weights in supervised training
- Mixture-of-Experts analogues with load-balancing and master expert enhancements

These approaches are motivated by the need to minimize both the wall-clock and memory requirements for deployment on commodity or low-resource hardware, without sacrificing task accuracy or model robustness.

## 2. Parameter Sharing, Wide Sharing, and Redundancy Removal

The standard approach in Transformers is a stack of $L$ independently parameterized 2-layer FFNs, each mapping $d_{\text{model}} \to d_{\text{ff}} \to d_{\text{model}}$. Empirically, a large portion of these FFNs learn overlapping “key-value” memories, leading to redundancy [2309.01826]. EFFN strategies address this by:

- Removing decoder-side FFNs entirely, replacing them with identity mappings or bypasses.
- Tying all encoder-side FFNs to a single shared module: for $N_{\text{enc}}$ encoder layers, a single FFN is used, re-applied identically at each depth.

To compensate, the shared FFN’s hidden size $D_{\text{shared}}$ is increased: $D_{\text{shared}} = f$ (standard), $D_{\text{shared}} \approx N_{\text{enc}} f$ (recovers baseline accuracy with fewer parameters), $D_{\text{shared}} = L f$ (equal to total original FFN capacity). This technique yields significant parameter savings ($>$40%), with speedups of 20–24% in decoding, and—when sufficiently widened—can even surpass the baseline BLEU in machine translation (+0.9 BLEU for WMT22 En→De at identical parameter count) [2309.01826].

These results show that the bulk of Transformer FFN parameters are redundant across depth, and that wide, shared EFFNs are both more efficient and potentially more accurate.

## 3. Conditional, Split, and Fast Feed-Forward Computation

Several EFFN implementations use conditional computation—only evaluating a small subnetwork per input, based on a learned routing structure. The FFF (Fast Feed-Forward Network) [2308.14711], and its enhanced variant eFFF [2405.16836], use a binary tree of $D$ differentiable sigmoid gates atop $2^D$ small “leaf” FF blocks (experts). During training, the entire tree is softly traversed for differentiable mixture output:
\[
y_{\text{train}}(x) = \sum_{i=1}^{2^D} c_i(x) \, \ell_i(x)
\]
where $c_i(x)$ is the product of binary gate probabilities along the path, and $\ell_i(x)$ is the small expert output. At inference, each gate is hard-thresholded, and only one expert is activated:
\[
y_{\text{inference}}(x) = \ell_{i^*(x)}(x)
\]

In terms of computational complexity, a standard layer of width $w$ costs $O(w)$ per input, while FFF/EFFN variants cost only $O(\log w + \ell)$, with $\ell \ll w$ the leaf width. Empirically, this achieves up to 220× speedup over dense FFNs and preserves $>$94% of original accuracy in ViT-like architectures [2308.14711]. Enhanced FFF introduces two further elements:

- Load balancing penalty, pushing routing probabilities to use all leaves equally (reduces run-to-run performance variance from ±29% to ±1–3% on MNIST).
- Addition of a small Master Leaf “global expert” whose output is always included in the prediction:
\[
y_{\text{eFFF\_inference}}(x) = k\,\ell_{i^*(x)}(x) + (1-k)\,\mathrm{ML}(x)
\]
This improves accuracy by up to 3% on test sets and substantially reduces output variance compared to vanilla FFF [2405.16836].

## 4. Explicit Algebraic Solutions and Layerwise Optimization

A further avenue for EFFN efficiency is the derivation of explicit, closed-form solutions for FFN weights, avoiding gradient-based iterative training. The SAFFU approach [2311.07510] derives a “column-translation” solution for single-layer softmax-FFNs, based on co-occurrence statistics:
\[
W_{d,i} = \log C_{d,i} - \frac{K-1}{K} \log\left(\sum_{d'} C_{d', i}\right)
\]
where $C_{d,i} = \sum_m x_{m,d} y_{m,i}$ is the co-occurrence of input component $d$ and output category $i$, and $K = \lVert x_m \rVert_1$.

For multi-layer networks, weights are sequentially solved given explicit targets at each layer, possibly in concert with neural attention (e.g., Self-Attentive Feed-Forward Units). This approach enables rapid prototype training, allowing ablation of hundreds of architectural variants on modest datasets in orders-of-magnitude less time than backprop alone. Empirically, explicit initialization followed by backprop-based fine-tuning yields lower perplexity (test PPL ≈ 23.8 vs. 64) and enables parameter-efficient, well-generalized models for low-resource or embedded deployment [2311.07510].

## 5. Compression, Heavy-Hitter Subnetworks, and Pruning

Highly efficient EFFN variants split the FFN into two subnetworks based on neuron “activity” or importance. The FFSplit strategy [2401.04044] identifies a heavy-hitter set $H$ of neurons (most of the $\ell_2$ norm of activations for any input), executing the following process:

- For each FFN neuron, compute average per-neuron output norm over a calibration set.
- Select top-$k$ neurons (heavy hitters) as $H$, the rest as $S$.
- Partition $W_1, W_2$ into $(W_{1,H}, W_{1,S}; W_{2,H}, W_{2,S})$, define $\mathrm{FFN}_H$ and $\mathrm{FFN}_S$.
- Assign high capacity to $\mathrm{FFN}_H$ (full precision), apply strong compression (low-rank SVD, quantization) to $\mathrm{FFN}_S$.
- Fine-tune to recover accuracy.

In BERT-Base and BERT-Large models, FFSplit achieves a 43% parameter reduction with accuracy drops of ≤1% and $1.25$–$1.56\times$ speedup on language model inference [2401.04044]. When integrated with extreme quantization (e.g., 3-bit weight quant), FFSplit recovers much of the perplexity loss, showing robust generalization under aggressive compression.

Another orthogonal approach prunes entire neurons based on low output variance, transferring pruned neuron means into biases and retraining. This achieves 70–99% parameter reduction with accuracy preservation or improvement across datasets, outperforming classical layer-wise pruning pipelines [2312.10560].

## 6. Applications, Key Results, and Limitations

EFFNs have been demonstrated in multiple domains:
- Efficient approximations of dynamic recurrent nets for attractor computations, enabling single-forward-pass inference in $O(n^2)$ time, maintaining noise rejection and WTA behaviors [1704.06645].
- Transformer language models with wide-shared or split FFNs, supporting translation, BERT, and OPT-class models [2309.01826, 2401.04044].
- Vision Transformers where fast (conditional) FFNs replace dense layers, attaining large speedups with minimal accuracy loss [2308.14711].

Empirical performance across techniques is summarized below:

| Paper           | Main Approach           | Compression/Speedup                | Accuracy/Task Impact           |
|-----------------|------------------------|------------------------------------|-------------------------------|
| 2309.01826      | Shared+Wide FFN        | 20–40% param. reduction, +24% spd  | +0.9 BLEU on WMT22, baseline+ |
| 2401.04044      | Split heavy-hitter FFN | 43.1% param, 1.25–1.56× speedup    | ≤1% drop, 5× quantization PPL |
| 2308.14711      | Tree-based (FFF)       | up to 220× FFN, 6× MoE             | 94.2% ViT accuracy retained   |
| 2311.07510      | Explicit algebraic init| 100× training speed, 3–10× less data| Test PPL ≈24 vs. 64           |
| 2312.10560      | Pruning + retrain      | 70–99% param, 2–40× smaller        | ACC↑ or preserved             |
| 2405.16836      | Load-bal, master leaf  | Accuracy +3%, var.↓~10× (MNIST/FMNIST)| Sublinear inference cost      |

Limitations and caveats include:
- Extreme compression can degrade performance in out-of-distribution settings.
- Conditional computation architectures may introduce overhead if not efficiently implemented in target hardware.
- For autoregressive or decoder-only models, wide sharing or highly conditional FFNs create latency bottlenecks.
- Explicit algebraic solutions assume restricted activation and loss forms, possibly limited for large vocabulary or regression tasks.
- Most EFFNs focus solely on FFN modules, leaving attention and embedding submodules as potential further bottlenecks.

## 7. Outlook and Integration into Modern Deep Learning

EFFN methodologies provide a comprehensive toolkit for scaling, compressing, and accelerating deep neural networks without compromising learning capacity. Key practical guidelines include:

- For Transformer encoder-decoders: share FFN layers, widen as needed, and drop decoder-side FFNs for speed.
- For resource-constrained deployment: partition FFNs into heavy- and sparse-components, compress accordingly, fine-tune for recovery.
- For hardware-critical inference: tree-based EFFNs achieve maximal reduction in runtime at minimal expressivity loss.
- For training efficiency and architecture search: closed-form layerwise EFFNs provide rapid ablation and warm-start points for fine-tuning.
- Add load balancing and global expert terms to stabilize conditional EFFN training.

EFFNs thus serve not as a single architecture, but as a paradigm for advancing deep learning efficiency, pushing the limits of compression, speed, and task generalization—both in high-resource and embedded domains [1704.06645, 2309.01826, 2311.07510, 2308.14711, 2401.04044, 2312.10560, 2405.16836].

Source: https://www.emergentmind.com/topics/efficient-feed-forward-network-effn