Exchange-Gate and Mixer Layers

• Exchange-Gate and Mixer-Based Layers are neural sequence modeling blocks that alternate between quadratic (softmax) attention and linear recurrent mixers to balance rich contextualization and computational efficiency.
• They employ extensive parameter sharing and integrated gating mechanisms to ensure smooth, dynamic switching between modes and robust performance across language and retrieval tasks.
• The design allows dynamic per-token or per-chunk mode selection, achieving over 90% shared parameters and demonstrating significant improvements in perplexity and accuracy.

Exchange-gate and mixer-based layers comprise a class of flexible neural sequence modeling blocks that can alternate between different token mixing paradigms within the same architectural framework. These approaches, exemplified by the Oryx Multi-Mixer model, are motivated by the complementary strengths of quadratic (softmax) attention and linear recurrent mixers in handling long-context retrieval and efficient sequence processing. By sharing over 90% of parameters across two distinct mixer types—attention and state-space–style linear recurrence—such layers enable dynamic switching either per chunk or per token, maximizing expressive power while maintaining computational efficiency (Li et al., 27 May 2026).

1. Token Mixers: Quadratic Attention and Linear Recurrence

Oryx implements two main mixer types:

• Quadratic (Softmax) Attention: At each position $t$, queries are computed as $q_t = x_t W^{q_{attn}}$. Keys and values are shared across all mixers via $k_t = x_t W^K$ and $v_t = x_t W^V$. Standard causal self-attention is applied:

$$\text{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q K^T}{\sqrt{d_k}} + \text{mask}\right) V$$

where $\text{mask}_{ij} = -\infty$ for $j > i$.

• Linear Recurrence (Mamba-2, Gated DeltaNet): Uses the same $k_t$ and $v_t$ projections, but a different query $q_t = x_t W^{q_{lin}}$. The state $q_t = x_t W^{q_{attn}}$0 is updated recurrently:
  • Mamba-2: $q_t = x_t W^{q_{attn}}$1, $q_t = x_t W^{q_{attn}}$2, with $q_t = x_t W^{q_{attn}}$3 a learned decay.
  • Gated DeltaNet: $q_t = x_t W^{q_{attn}}$4, $q_t = x_t W^{q_{attn}}$5, with $q_t = x_t W^{q_{attn}}$6 as data-dependent scalars.

These dual mixers facilitate context-dependent token mixing, where quadratic attention provides rich contextualization and linear recurrence yields scalable efficiency.

2. Sequence-Axis Hybridization

Oryx introduces hybridization on the sequence axis rather than the layer axis. Instead of stacking different mixer layers, the model dynamically selects the active mixer (attention or linear) per sequence chunk or token, denoted $q_t = x_t W^{q_{attn}}$7. During training, fixed-length chunks (e.g., 128 tokens) are randomly assigned to attention or linear modes with a ratio of $q_t = x_t W^{q_{attn}}$8 for attention and $q_t = x_t W^{q_{attn}}$9 for linear, finding this balance optimal for performance. At inference, any arbitrary pattern of mixer activation can be applied, including switching mid-stream, due to the shared key/value and state projections that ensure representational compatibility (Li et al., 27 May 2026).

Sequence-axis hybridization allows flexible trade-offs between context modeling capacity and efficiency, with the possibility to allocate computationally expensive attention only to critical segments of a long sequence.

3. Parameter Tying and Architectural Design

A central mechanism is the extensive parameter sharing across mixers:

• Shared Projections: Key ($k_t = x_t W^K$0), value ($k_t = x_t W^K$1), output projection ($k_t = x_t W^K$2), short 1D convolution, and gating projection ($k_t = x_t W^K$3) are common to all mixers.
• Separate Projections: Only the query projections for attention ($k_t = x_t W^K$4) and linear recurrence ($k_t = x_t W^K$5), along with linear mixer “support” parameters (e.g., $k_t = x_t W^K$6), are distinct.
• Parameter Distribution: A 1.4B-parameter Oryx model has approximately 90% weight sharing, with only ~10% specific to mixer types. This is achieved by rebalancing MLP widths and allocating extra parameters for linear mixer support and gating (Li et al., 27 May 2026).

This extensive tying ensures both efficiency and maximal representational sharing, enabling seamless mode switches within a unified model state.

4. Gating Mechanisms in Mixer-Based Layers

Oryx integrates gating at two principal locations:

• Output Gate ("GatedRMSNorm"): After mixer computation, a gate is applied: compute $k_t = x_t W^K$7, then $k_t = x_t W^K$8. This mechanism consistently improves perplexity in both mixer modes by a few points.
• Mixer-Specific Gating: In Gated DeltaNet, the update scalars $k_t = x_t W^K$9 are computed as $v_t = x_t W^V$0. These internal gates control contributions to the state update; however, the inner gating is distinct from Oryx’s output gate and is not tied between the mixers (Li et al., 27 May 2026).

Gating enhances both information flow and selective activation of pathways through the model, refining the expressive capacity of both attention and recurrent blocks.

5. Mixed-Mode Training Strategy

Training proceeds by dividing the input sequence into fixed-size chunks, with each assigned at random to attention or linear mode according to the empirically optimal 1:3 ratio. All network blocks share the same scheduling, and standard cross-entropy loss is computed over all tokens. No explicit regularization or “mode-balancing” terms are used, aside from the chunk assignment (Li et al., 27 May 2026).

This mixed-mode regime forces the shared key-value-state spaces to remain compatible across modes, supporting smooth and dynamic inference-time mode switches. The compatibility is validated by the empirical finding that, even when mixers switch mid-stream, performance remains robust.

6. Empirical Results and Performance Analysis

Key results from Oryx at the 1.4B parameter scale, trained on a fixed 100B-token budget from FineWeb-Edu, include:

• Language Modeling: Attention-only Oryx achieves test perplexity $v_t = x_t W^V$1 versus $v_t = x_t W^V$2 for a pure Transformer (an absolute improvement of $v_t = x_t W^V$3 pp). Linear-only Oryx (Mamba-2 or GDN) outperforms pure linear baselines by $v_t = x_t W^V$4 pp. Across LAMBADA, HellaSwag, PIQA, ARC, WinoGrande, OBQA, average accuracy increases by $v_t = x_t W^V$5 points.
• Retrieval (<10% attention): With only the first $v_t = x_t W^V$6 of chunks using attention, Oryx-TM matches Transformer-level retrieval on SWDE, SQuAD, FDA, TriviaQA, NQ, DROP. On synthetic Needle-in-a-Haystack tests, Oryx with 10% attention outperforms linear baselines by 38 points (e.g., from $v_t = x_t W^V$7 to $v_t = x_t W^V$8 correct) (Li et al., 27 May 2026).
• Parameter Counts: With $v_t = x_t W^V$9B of the $$\text{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q K^T}{\sqrt{d_k}} + \text{mask}\right) V$$0B total parameters shared, and mixer-specific parameters and gate projection accounting for the remainder, Oryx maintains exact parameter budget matching to single-mixer baselines by adjusting MLP widths.

The results demonstrate the viability of sequence-axis hybridization and extensive parameter sharing for enhancing both language modeling quality and long-context retrieval under constrained compute.

7. Context and Implications

The development of exchange-gate and mixer-based layers, typified by the Oryx Multi-Mixer block, suggests that hybrid models can efficiently leverage the strengths of both softmax attention and linear recurrent mechanisms within a single state and projection space. This approach enables new directions for sequence modeling architecture design, emphasizing flexible, context-adaptive, and efficient computation without increasing overall parameter or token budgets.

A plausible implication is that sequence-axis mixer switching could become a general strategy for balancing representational richness and scalability in future large models. The empirical evidence that $$\text{Attention}(Q, K, V) = \mathrm{softmax}\left(\frac{Q K^T}{\sqrt{d_k}} + \text{mask}\right) V$$1 parameter sharing delivers robust performance across both modes further motivates the investigation of unified architectures for mixture-of-token-mixers designs (Li et al., 27 May 2026).