MLP Representation with Attention Heads

• The paper proves that every neuron in a single-layer MLP can be exactly implemented by a masked attention head with tailored activation functions.
• It demonstrates that transformer blocks can collapse MLP and attention modules into a unified structure, enhancing model interpretability and design.
• Dynamic models like HyperMLP leverage context-conditioned attention as MLPs to improve expressive capacity and efficiency across modalities.

A multi-layer perceptron (MLP) representation with attention heads refers to the observation, construction, and transformation of feed-forward neural modules (traditionally implemented as MLPs) using the primitives, operators, and sometimes the computational graph of multi-head self-attention. This paradigm—spanning proof-of-principle theorems, mechanistic studies, and architectural innovations—establishes a tight and sometimes interchangeable correspondence between attention mechanisms and MLP function classes, frequently in the context of transformer and related sequence models.

1. Theoretical Foundations: Implementing MLPs with Attention Heads

Huben and Morris provide a formal proof that every neuron in a single-layer MLP can be exactly implemented by a masked attention head with internal dimension 1, with the activation function drawn from a restricted but expressive class, including SiLU and arbitrarily close approximations to ReLU and GeLU. Specifically, a depth-one MLP $f(X) = \alpha(XV_1)V_2$ (for $X\in M_{N,D}$ and $V_1, V_2$ of appropriate shape, and $\alpha$ pointwise, e.g., $\alpha(x)=a_1\mathrm{SiLU}(a_2 x)$) can be simulated by constructing $\ell$ masked attention heads, where each head computes exactly the output of a single MLP neuron, up to an appended bias token. This transformation establishes a bijection (modulo parameter count and width expansion) between the functional capacity of an MLP block and a set of tailored attention heads (Huben et al., 2023).

Moreover, these results generalize: attention heads, when equipped with proper masking and constructed QK/OV parameters, can separately realize the two standard components of an MLP—arbitrary linear transformations and pointwise nonlinear activations—supported by theorems on masking and entrywise activations. Arbitrarily complex masking patterns can also be encoded into QK matrices via pseudo-masking constructions to an arbitrarily fine accuracy, ensuring that the attention-based implementation can precisely replicate the computational structure of standard MLPs.

2. Architectural Reductions: Collapsing Transformer Sublayers

The practical implication of this equivalence is that the canonical transformer pattern—alternating multi-head attention and feed-forward MLPs—admits radical simplification. One can convert a two-sublayer transformer block into a strictly attention-only block by replacing each MLP neuron by a masked attention head. The trade-off is parameter and compute redundancy: a typical “MLP of width $\ell$” becomes $\ell$ separate heads of internal dimension 1, effectively expanding the attention head count by a factor corresponding to the hidden size. Despite this expansion, the entire feed-forward operation is subsumed by the attention operator alone, opening the door to uniform analysis, interpretability, and model design built exclusively on attention-based primitives (Huben et al., 2023).

3. Dynamic and Contextual MLPs via Attention: HyperMLP and the Memory View

Recent work reinterprets self-attention through the lens of dynamic, context-instantiated MLPs. In this view, the attention head at position $t$ can be written as a two-layer MLP where the effective weights $W^{(1)},\, W^{(2)}$ are functions of the entire prefix $X\in M_{N,D}$0—that is, $X\in M_{N,D}$1, $X\in M_{N,D}$2—with each contextual “slot” corresponding to a memory vector from the sequence (Lu et al., 13 Feb 2026).

The HyperMLP and HyperGLU architectures generalize this by introducing learned, dynamic mixing in both feature and sequence space, leveraging a “lag” layout for autoregressive semantics. Theoretically, these heads realize strictly richer function classes (continuous piecewise-smooth splines instead of polyhedral partitions), and empirically, HyperMLP/HyperGLU consistently outperform strong softmax-attention baselines under matched parameter budgets (Lu et al., 13 Feb 2026).

4. Mechanistic Links and Interpretability: Attention-MLP Interaction Circuits

Empirical studies of transformer representations elucidate the mechanism by which attention heads and MLPs interact for predictive modeling. In GPT-2, highly specialized “next-token neurons” in the MLP are modulated by the output of particular upstream attention heads. These heads recognize context patterns (for example, phrase structure, idioms) and, via the residual stream, selectively activate the projection weights corresponding to target tokens through alignment in activation space. Attribution scores quantify the strength and specificity of these head-to-neuron interactions. Intervention (ablation) experiments confirm that heads identified as context-specific causally drive next-token neurons, validating a modular “attention → MLP” motif for token prediction (Neo et al., 2024).

Additionally, behaviorally-salient features (e.g., sycophancy) are often most linearly separable—and most effectively steerable—within a sparse subspace of attention heads, even if the MLP and residual stream distributedly encode them. This supports the view of attention heads as sparse, high-leverage functional “gateways” in contrast to the more diffuse, distributed representations of MLPs (Genadi et al., 23 Jan 2026).

5. MLP-Style Attention for Efficiency and Alternative Modalities

Replacing or approximating attention with MLP-based or “MLP-style attention” mechanisms is an emerging theme for efficiency and modality adaptation. Approaches such as RAM (Residual-Attention Approximation with MLP) replace the Q/K/V projections, attention score computation, and score-weighted value aggregation with stacked 2-layer MLPs plus normalization and residuals. This dramatically reduces FLOPs and model parameters (up to 62.6% FLOPs reduction in spatio-temporal modeling) while maintaining high predictive accuracy (<3% performance drop in benchmarks), especially for long sequences or resource-constrained settings (Guo et al., 2024).

In vision models, multidimensional MLP mechanisms can be designed as “attention tools.” The MDMLP Attention Tool (MDAttnTool) demonstrates that sequential per-dimension MLPs—operating across width and height—combined with spatial pooling can recover context-dependent, position-aware “attention” maps, despite lacking any explicit dot-product or pairwise dependency computation. This design achieves competitive accuracy with strong interpretability (saliency of object structure) and low parameter count (Lv et al., 2022).

6. Extensions: Mixture-of-Experts and State-Space Models Emulating Multi-Head Attention

Recent architectures combine mixture-of-experts (MoE) mechanisms with state-space models to emulate multi-head attention, integrating MLP-based channel-mixing with recurrent or linear time-mixing. In MossNet, both channel projections (queries, keys, values) and temporal kernels are implemented via sparsely-gated MoE blocks. Each expert or “head” processes distinct feature channels or time scales, and the MoE mechanism ensures high expressiveness at fraction-of-linear compute per token. MossNet recapitulates the mathematical structure of multi-head linear attention while enjoying SSM efficiency and cache cost, yielding models that scale favorably with sequence length and outperform both standard transformers and SSM hybrids on large-scale language modeling tasks (Tuli et al., 30 Oct 2025).

7. Conclusion and Implications

The synthesis and interchangeability of MLPs and attention heads now constitute a well-established facet of theoretical, mechanistic, and applied transformer research. The classical boundary between “attention” and “MLP” modules is not fundamental: every key transformation—linear projection, activation, masking, memory access—can be instantiated within the attention framework, subject to resource tradeoffs. Conversely, many computational benefits and architectural simplifications accrue from realizing attention through the lens of dynamic, context-conditioned MLPs. This frame provides both a toolkit for model compression, hardware optimization, and interpretability, and a theoretical lens through which to analyze depth, modularity, and specialization in large sequence models (Huben et al., 2023, Lu et al., 13 Feb 2026, Neo et al., 2024, Genadi et al., 23 Jan 2026, Guo et al., 2024, Lv et al., 2022, Tuli et al., 30 Oct 2025).