HyperGLU: Dynamic MLP in Sequence Modeling

• HyperGLU is a neural sequence modeling block that employs a residual two-layer MLP with weights dynamically generated from full prefix context.
• It enhances autoregressive models by decoupling gating and scaling, enabling effective temporal and feature mixing using context-aware GLU mechanisms.
• Empirical and theoretical studies demonstrate that HyperGLU outperforms traditional attention methods by providing richer routing and stability under matched parameter budgets.

HyperGLU is a neural sequence modeling block that instantiates a residual two-layer multi-layer perceptron (MLP), with all weights dynamically generated from the full prefix context. Introduced within the HyperMLP framework, HyperGLU is specifically designed to enhance temporal and feature mixing for autoregressive models by implementing both context-wide mixing and decoupled gating via gated linear units (GLU). Unlike classical attention mechanisms that treat memory access as probabilistic query-key lookup, HyperGLU adopts a formulation in which the autoregressive context directly parameterizes the routing and readout transformations, demonstrating both theoretical and empirical advantages under matched parameter budgets (Lu et al., 13 Feb 2026).

1. Mathematical Formulation and Architectural Overview

Each HyperGLU head operates as a residual two-layer MLP, with weights parameterized by the lag-ordered prefix context. For a current token $x_t \in \mathbb{R}^d$ and its lag-ordered prefix $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$, the two principal parameterizations per head are:

• First-layer routing (input-to-memory):

$$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$

Here, $L^{(1)}(x_t)$ and $R^{(1)}(x_t)$ are context-dependent, with a diagonal-plus-low-rank factorization involving low-rank matrices, diagonal matrices, and sample-specific sigmoidal nonlinearities.

• Second-layer readout (memory-to-output):

$$W^{(2)}_{\mathrm{MLP}}(X_{t:1}) = R^{(2)\,T}(x_t)\; X_{t:1}\; L^{(2)\,T}(x_t)$$

Again, $L^{(2)}(x_t)$ and $R^{(2)}(x_t)$ employ analogous diagonal-plus-low-rank factorizations.

After applying $W^{(1)}_{\mathrm{MLP}}(X_{t:1})$, the $t$-dimensional preactivation vector is split into two halves: $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$0 and $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$1. The slot-wise activation is computed as:

$$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$2

where $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$3 for a small $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$4, ensuring invariance up to rescaling.

The final output is:

$$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$5

This structure ensures that feature selection ("gating") and activation magnitude ("scaling") remain decoupled, increasing stability and expressivity.

2. HyperGLU as Attention-as-Dynamic-MLP

Traditional autoregressive attention can be recast as a special case of a dynamic two-layer MLP, where effective weight matrices are contextually instantiated by the full prefix $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$6. That is:

$$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$7

In this reformulation:

• The context $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$8 forms an ever-growing memory pool.
• Nonlinearities such as ReLU or GLU route activations into a select subset of $$X_{t:1} = [x_t; x_{t-1}; \ldots; x_1] \in \mathbb{R}^{t \times d}$$9 slots.
• Multi-head variants correspond to parallel instantiations of this dynamic MLP.

HyperGLU departs from classical attention by introducing:

• Feature-space decoupling: The activation function splits input into gating and scaling branches, using $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$0 (for selection) and $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$1 (for slotwise scaling), yielding richer and more stable selection than classic ReLU or softmax.
• Context-wide sequence mixing: Both the first-layer ($$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$2) and second-layer ($$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$3) transformations perform learned, length-agnostic temporal mixing, enabling each memory slot to encode information aggregated across the entire context, rather than only token-wise attention.

3. Lag-Ordered Layout and Consistency Properties

HyperGLU employs a lag-ordered, reverse-offset sequence layout, $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$4, for all temporal mixing. This design guarantees autoregressive consistency in several respects:

• Extension Consistency: Sequence mixing adapters $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$5 are parameterized so they can be zero-padded/sliced—ensuring that outputs are compatible under natural prefix extension of the sequence.
• Truncation Invariance: With lag-order layout and padding-invariant L2 normalization, the output at $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$6, $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$7, depends only on the most recent $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$8 tokens—the causal window is always local to the newest context.
• Theoretical Guarantee: Given the above, both extension and truncation operations preserve the semantics necessary for autoregressive modeling, with outputs properly reflecting only the current prefix and not farther-past tokens.

4. Expressivity and Theoretical Analysis

Multiple theoretical results demonstrate that HyperGLU encompasses and extends the classical softmax/ReLU attention family:

• Routing Geometry: Standard ReLU attention induces a polyhedral partition in $$W^{(1)}_{\mathrm{MLP}}(X_{t:1}) = L^{(1)}(x_t)\; X_{t:1}\; R^{(1)}(x_t)$$9, while HyperGLU’s dynamic sequence mixing warps these into curved partitions, increasing function class complexity.
• Context-wide Slot Construction: The learned $L^{(1)}(x_t)$0 enables each slot to aggregate context-wide linear combinations, rather than being constrained to pointwise access, greatly increasing representational richness (Proposition H.3).
• Budget-Asymmetry: Compression of the value/output side (VO) of the MLP directly restricts update subspace, while compression of the query/key (QK) side limits only conditioning, making VO compression more damaging to expressivity (Theorem 2.5).
• GLU Decoupling: The feature-space separation between gating (slot selection) and scaling (slot magnitude) increases both expressivity and robustness—minor weight changes less frequently alter routing behaviors (Proposition 2.4, Lemma D.11).
• Strict Containment: When parameter budgets are matched (by allocating parameters only to $L^{(1)}(x_t)$1 and $L^{(1)}(x_t)$2), HyperGLU strictly contains the classical attention class and is not approximable by finite hyperplane arrangements (Proposition I.1, I.2).

5. Empirical Results and Ablation Studies

HyperGLU demonstrates consistent empirical gains across various controlled and large-scale sequence modeling benchmarks:

Study	Key Findings	Source
MAD & NanoGPT	Sequence mixing with lag layout yields MAD avg ≈ 81.0 vs. ≈ 66.8 and NanoGPT loss 2.9956 vs. 3.0828. Lag-order is necessary; one-sided mixing or omitting lag is suboptimal.	Table 1 (Lu et al., 13 Feb 2026)
OpenLLM Leaderboard	At 1.3B/100B tokens, best ranking (1.88 v 4.22), leads in 7/15 tasks (MMLU-P 0.120 v 0.114, BBH 0.321 v 0.296).	Table 2 (Lu et al., 13 Feb 2026)
OpenWebText2 & FineWeb-Edu	Accelerated convergence (loss <3.1 in 26 vs. 76 steps); final loss reduced by ~2–4% under strict budget matching.	(Lu et al., 13 Feb 2026)
Efficiency (Triton, GPT-small)	Per-head, extra work $L^{(1)}(x_t)$3 and state $L^{(1)}(x_t)$4. Latency 39ms, throughput 207K tokens/s—comparable to other expressive variants.	Table 5 (Lu et al., 13 Feb 2026)

Performance degrades markedly if sequence mixing omits lag order or is only one-sided. Empirical ablations confirm that VO compression harms performance more than QK compression, in agreement with theoretical expectations. The best GLU variant achieves state-of-the-art average score under matched budgets.

6. Relationship to Classical Attention and Broader Significance

HyperGLU emerges as a unifying block that subsumes and generalizes classical attention under the proposed "attention-as-dynamic-two-layer-MLP" perspective. By learning both feature-space gating (via GLU) and general sequence-space mixing, it forms a strictly richer and practically efficient superset of softmax/ReLU attention. These advantages are substantiated both by theoretical analysis (including context-wide slot construction, routing geometry, and budget asymmetry) and by consistent empirical improvements across benchmarked LLMs and sequence tasks (Lu et al., 13 Feb 2026).

A plausible implication is the broader applicability of the HyperGLU formulation to future work on dynamic context mixing, adaptive memory systems, and efficient latent routing architectures within sequence modeling frameworks.