AttentionMLP: MLP-Integrated Attention Models

• AttentionMLP is a neural architecture class that replaces or fuses traditional attention with feed-forward MLPs using techniques like spatial gating and dynamic mixing.
• It encompasses variants such as gMLP, DynaMixer, MDMLP, and SID-MLP, each employing specific strategies to mimic or augment cross-token interactions.
• Empirical benchmarks show these models achieve competitive performance in image classification, language modeling, and recommendation tasks with enhanced computational efficiency.

AttentionMLP refers to the class of neural architectures in which multi-layer perceptrons (MLPs) either replace or structurally fuse components traditionally implemented by attention mechanisms, particularly in Transformer models. The term encompasses architectures such as gMLP, DynaMixer, MDMLP, and related variants, which exploit spatial, contextual, or dynamic gating schemes within feed-forward block designs to approximate or augment attention-like communication among tokens in sequence or vision tasks, as well as analyses of functional interactions between attention and MLP submodules within standard Transformer pipelines.

1. Architectural Foundations and Key Variants

The original motivation for AttentionMLP models emerges from the recognition that classical attention—specifically multi-head self-attention—can be substantially mimicked, replaced, or complemented by carefully designed MLPs coupled with channel mixing and gating mechanisms. gMLP introduces a paradigm in which each residual block is almost entirely feed-forward and attention-free, relying on a Spatial Gating Unit (SGU) to instantiate cross-token communication using a static learned mixing matrix. The block applies channel expansion, activation (GeLU), spatial gating, channel projection, and a residual update. Formally, for input $X \in \mathbb{R}^{n \times d}$, the SGU operates on an expanded activation $Z \in \mathbb{R}^{n \times e}$ by splitting $Z = [Z_1 ~\|~ Z_2]$, applying normalization and a static $n \times n$ weight $W$ to $Z_2$, and finally performing $s(Z) = Z_1 \odot (W \,\mathrm{LayerNorm}(Z_2) + b)$, where $\odot$ denotes elementwise multiplication. No position encodings are required, as the static gating can infer position dependencies during learning (Liu et al., 2021).

DynaMixer generalizes the static mixing of gMLP to a dynamic, content-adaptive regime. By projecting input tokens into a reduced key space, segmenting channels, and generating a mixing matrix based on the current content, DynaMixer constructs a dynamic $N \times N$ mixing matrix $P$ for $Z \in \mathbb{R}^{n \times e}$0 input tokens, applied as $Z \in \mathbb{R}^{n \times e}$1. The complexity is mitigated with projection and segmentation, yielding a content-adaptive mixing operation that outperforms prior static MLP architectures (Wang et al., 2022).

MDMLP demonstrates a decomposition of spatial and channel mixing into a sequence of MLP transformations along axes of height, width, channel, and token, using overlapping patch embeddings and residual MDLayer blocks. MDMLP targets data-constrained settings, addressing the well-known limitation that MLPs underperform ConvNets and ViTs on small datasets unless architecturally adapted (Lv et al., 2022).

In certain recommendation and decoding pipelines, attention-heavy Transformer decoders are replaced via knowledge distillation by position-specific MLP heads, reducing the redundancy of repeated attention computations and enabling highly efficient, context-aware token generation (e.g., SID-MLP) (Guo et al., 12 May 2026).

2. Theoretical and Mechanistic Analysis

At a theoretical level, AttentionMLP models demonstrate that cross-token communication can be instantiated without dynamic, content-based softmax-weighted pairwise interactions. In gMLP, the spatial gating unit learns local or shift-invariant filters in vision and converges to Toeplitz or convolution-like kernels in NLP, evidencing that static mixing matrices can emulate convolutional or fixed pattern mixing seen in attention heads. The multiplicative gating effect enables higher-order token interactions, as the elementwise product in the SGU ($Z \in \mathbb{R}^{n \times e}$2) integrates quadratic terms $Z \in \mathbb{R}^{n \times e}$3. The role of attention's softmax mixing in grounding content-based alignment is subsumed in these static or learned linear operators, especially when channel expansion precedes gating (Liu et al., 2021).

Empirically, both gMLP and Transformer models demonstrate power-law scaling in pretraining perplexity versus parameter count, indicating that depth and width contribute similarly to capacity and generalization, even in the absence of explicit self-attention. The necessity of dynamic content-based attention arises primarily in tasks with strong long-range or cross-span dependencies, as observed in challenging NLP benchmarks (e.g., SQuAD, MNLI), where pure gMLP underperforms but the addition of even a small attentional submodule (as in aMLP) recovers or outperforms classic Transformer baselines (Liu et al., 2021).

Analyses of standard Transformers further show that the interplay between attention and MLP components is highly structured: upstream attention heads often act as context detectors, activating downstream "next-token" MLP neurons specialized for particular vocabulary items. Direct attribution and ablation experiments evidence that certain attention heads serve as gates for context-appropriate token prediction via their effect on the pre-activations in downstream feed-forward (MLP) modules, reinforcing the view that attention-MLP synergy is central to next-token inference and circuit-level functionality (Neo et al., 2024).

3. Comparative Empirical Performance

The empirical benchmarks highlight the practical competitiveness of AttentionMLP-derived models. In image classification on ImageNet-1K, gMLP achieves 72.3–81.6% top-1 accuracy (Tiny–Base), matching DeiT models of the same scale within 0.2–0.3% despite the absence of dynamic attention (Liu et al., 2021). DynaMixer further surpasses prior pure MLPs, attaining 84.3% top-1 with 97M parameters (ImageNet-1K) (Wang et al., 2022). On CIFAR-10, MDMLP achieves 90.90% top-1 with only 0.3M parameters, outperforming MLP-Mixer (85.45% with 17M params) (Lv et al., 2022).

In language modeling, gMLP matches BERT$Z \in \mathbb{R}^{n \times e}$4 on MLM pretraining perplexity (4.28 vs 4.17) and exceeds on certain finetuning tasks. For recommendation decoders, SID-MLP recovers teacher-level NDCG@10 while yielding an $Z \in \mathbb{R}^{n \times e}$5 end-to-end speedup and over 95% reduction in memory footprint compared to full Transformer decoders (Guo et al., 12 May 2026).

Ablations reveal that:

• Static mixing underperforms dynamic mixing (DynaMixer vs static synthesizer);
• Tiny self-attention modules inserted into gMLP (aMLP) bridge or surpass the performance gap relative to BERT, especially on cross-sentence tasks (Liu et al., 2021);
• DynaMixer's accuracy improves with increased number of segments and key dimension up to a modest value, and dynamic mixing is essential for maximal performance (Wang et al., 2022).

4. Complexity, Scaling, and Efficiency Considerations

AttentionMLP models exhibit favorable computational profiles relative to classical attention. Static gating (gMLP) introduces negligible parameter and compute overhead compared to attention, since $Z \in \mathbb{R}^{n \times e}$6 static matrices replace content-dependent $Z \in \mathbb{R}^{n \times e}$7 computations. DynaMixer, while using a dynamic mixing matrix, employs aggressive channel dimensionality reduction ($Z \in \mathbb{R}^{n \times e}$8) and multi-segment fusion to reduce both parameter count and FLOPs (Wang et al., 2022).

In generative recommendation, the transition from autoregressive, attention-based decoding to positionwise MLP heads (SID-MLP) eliminates $Z \in \mathbb{R}^{n \times e}$9 sequential self-attention and per-token cross-attention, yielding large throughput and memory gains. Encoder-side replacement of attention by mean-pooled, role-specific MLP transformations introduces further efficiency, with only a minor impact on ranking metrics (Guo et al., 12 May 2026).

5. Interpretability and Mechanistic Insights

Mechanistic interpretability studies show that in LLMs, attention-MLP interactions underpin context-dependent token prediction. Specific attention heads can be causally attributed to the activation of downstream "next-token" neurons. These heads specialize in non-trivial, context-sensitive look-ups (e.g., prescriptive linguistic patterns, comparative constructs, identifier schemes), feeding output directly into MLP pre-activation channels aligned with vocabulary embeddings. Causal ablation of active heads results in pronounced drops in the probability of the associated token output, substantiating a gating paradigm rather than simple copy-attention (Neo et al., 2024).

The implication is that transformer architectures might be further optimized by explicitly encoding or modularizing this head-to-neuron gating structure, potentially via hybrid AttentionMLP blocks or sparse, interpretable gating layers.

6. Limitations and Application Domains

Despite their advantages, pure AttentionMLP models demonstrate limitations. For vision, while gMLP and DynaMixer closely approach Transformer performance, they remain inferior to top ConvNet and hybrid models on datasets with strong spatial invariance. On small datasets, MLP-based models without convolutional or attention priors typically underperform unless carefully regularized or rearchitected (e.g., MDMLP with multi-dimensional axis mixing) (Lv et al., 2022).

In NLP, fully static or content-agnostic mixing struggles with tasks requiring precise, content-based alignment or long-range dependencies. However, these deficits can be largely alleviated by depth/width scaling or minimal attention augmentation (e.g., a single head per block) (Liu et al., 2021).

7. Broader Implications and Future Directions

AttentionMLP research challenges the prevailing view that multi-head self-attention is categorically indispensable for high-fidelity sequence and vision modeling. The ability of static or simple dynamic gating schemes to capture much of the token interaction power of attention has practical consequences for model efficiency, scaling, and mechanistic transparency. Hybrid designs and distillation strategies (SID-MLP, aMLP) open pathways to highly efficient inference, especially for structured prediction and recommendation tasks (Guo et al., 12 May 2026).

Interpretability findings suggest architectural innovation may benefit from modeling explicit attention-to-MLP routing, modular context gating, and sparse input dependencies. Future directions include designing more explainable, efficient, and modular AttentionMLP blocks, especially for constrained or on-device scenarios, as well as deeper mapping of the internal circuit-level mechanisms that emerge from such interactional architectures (Neo et al., 2024).

---

Key references:

• "Pay Attention to MLPs" (Liu et al., 2021)
• "DynaMixer: A Vision MLP Architecture with Dynamic Mixing" (Wang et al., 2022)
• "MLPs are Efficient Distilled Generative Recommenders" (Guo et al., 12 May 2026)
• "Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions" (Neo et al., 2024)
• "MDMLP: Image Classification from Scratch on Small Datasets with MLP" (Lv et al., 2022)