Quasiseparable Matrix Mixers

• Quasiseparable matrix mixers are structured linear operators that generalize low-rank, semiseparable, and banded matrices, enabling efficient bidirectional data mixing.
• They leverage two-sided factorizations and divide-and-conquer algorithms to achieve subquadratic computational complexity in sequence models.
• The Hydra architecture exemplifies their practical use, outperforming traditional self-attention in language and vision tasks with linear time and space costs.

A quasiseparable matrix mixer is a structured linear operator acting on sequences, formalized as a generalization of classical low-rank, semiseparable, and banded matrix structures. In the context of deep sequence models, especially those covered by the Matrix Mixer framework, these mixers enable mathematically principled, data-dependent, and highly efficient bidirectional sequence mixing with rigorously established computational and expressive properties. The Hydra architecture exemplifies the state-of-the-art implementation of quasiseparable matrix mixers, achieving bidirectional subquadratic sequence modeling that supersedes traditional self-attention mechanisms in both language and vision benchmarks (Hwang et al., 2024).

1. Structural Definition and Formal Properties

A matrix $M \in \mathbb{F}^{n \times n}$ is said to be $(r_L, r_U)$-quasiseparable if, for all $1 \leq k \leq n{-}1$, the maximal strictly lower and upper off-diagonal submatrices have rank at most $r_L$ and $r_U$, respectively: $\rank\bigl(M_{k+1:n,\,1:k}\bigr) \leq r_L, \qquad \rank\bigl(M_{1:k,\,k+1:n}\bigr) \leq r_U.$ This pair is called the lower and upper quasiseparable orders. The structure directly generalizes banded, Toeplitz, Hankel, and semiseparable matrices. The notion connects to the rank-profile matrix (RPM) invariant, enabling partitioning and fast algebraic computations (Pernet, 2016, Sa et al., 2016).

2. Quasiseparable Matrix Mixers in Sequence Models

Sequence mixers in the Matrix Mixer framework act on an input $X \in \mathbb{R}^{L \times C}$ as (possibly data-dependent) linear maps: $Y = M X',$ where $X' = f_X(X)$ is a sub-quadratic preprocessing and $M \in \mathbb{R}^{L \times L}$ is the mixer matrix. The introduction of sequence alignment (SAM property) constrains $(r_L, r_U)$0 so that both data-dependence (adaptivity to actual token features) and length-extendability (ability to generalize to longer sequences) are achieved.

Semiseparable mixers, as in state space models (SSMs) like Mamba, utilize strictly lower-triangular matrices where each off-diagonal block has low rank. Bidirectional modeling—beyond pure causality—necessitates the more general quasiseparable matrices, wherein both the strictly lower and strictly upper parts have bounded rank. This yields a mixer capable of fusing context both forward and backward in linear time (Hwang et al., 2024).

3. Factorizations and Algorithmic Implementations

Any $(r_L, r_U)$1-quasiseparable $(r_L, r_U)$2 admits a two-sided factorization: $(r_L, r_U)$3 with free diagonal entries $(r_L, r_U)$4. This formula generalizes semiseparable (one-sided) and low-rank structures, capturing them as special cases. The forward and backward branches may utilize parameters from distinct state space models or other SSM parameterizations.

Application to a vector can be efficiently reduced to combinations of semiseparable matrix-vector products: $(r_L, r_U)$5 where $(r_L, r_U)$6 denotes a semiseparable apply ($(r_L, r_U)$7 when $(r_L, r_U)$8, typically $(r_L, r_U)$9 for typical head widths), and $1 \leq k \leq n{-}1$0 is diagonal (Hwang et al., 2024).

Structured representations, notably the binary-tree of PLUQ decompositions and the Bruhat generator, allow storage and computation in $1 \leq k \leq n{-}1$1 or $1 \leq k \leq n{-}1$2, greatly reducing the overhead compared to the classical $1 \leq k \leq n{-}1$3 approaches (Pernet, 2016).

4. Computational Complexity and Mixing Algorithms

Quasiseparable matrix multiplication and vector application achieve sub-quadratic computational complexity:

• Matrix-vector multiplication: Once in Bruhat or binary-tree form, $1 \leq k \leq n{-}1$4 requires $1 \leq k \leq n{-}1$5 field operations.
• Matrix-matrix multiplication (mixing): Both binary-tree and compact Bruhat generators allow $1 \leq k \leq n{-}1$6 cost ($1 \leq k \leq n{-}1$7), a substantial improvement over $1 \leq k \leq n{-}1$8 for classical generators (Pernet, 2016).

Divide-and-conquer algorithms, often implemented as pivot-local PLUQ decompositions on a binary partition tree, ensure efficient propagation of low-rank factors. For structured models, these efficiencies are central for scaling sequence mixing to very high dimensions as in large language and vision models (Sa et al., 2016).

5. Expressivity, Data-Dependence, and Empirical Performance

Quasiseparable mixers are both data-dependent and length-extendable through the SAM property—each subset of matrix parameters is aligned with a sequence position, ensuring both adaptivity to local context and scalability to arbitrary sequence lengths. Empirical investigations demonstrate that the SAM property is highly correlated with transformer-like expressivity in structured matrix families (Hwang et al., 2024).

The Hydra module, operationalizing a bidirectional quasiseparable mixer, outperforms standard self-attention mechanisms. On language tasks (GLUE at 84.3% vs. BERT 83.5% at similar scale) and vision tasks (ImageNet-1K, Top-1 of 81.0% vs. ViT-B 78.8%) Hydra achieves state-of-the-art performance with linear-time and linear-space computational and parametric cost—facilitated by the quasiseparable structure.

6. Practical Implementations and Framework Integration

Hydra-style quasiseparable mixers directly replace transformer attention blocks via:

• Projecting input to SSM parameters for both forward and backward branches
• Computing forward and backward scan operations (via semiseparable applies on original and reversed input)
• Combining shifted results and a diagonal component for the final output
• Leveraging shared projections and parameter reuse to maintain minimal cost overhead

The resulting module is compatible with any existing SSM engine, eliminating the need for attention kernels, yet providing broad bidirectional context and subquadratic computation (Hwang et al., 2024).

Structured representation techniques such as compact Bruhat or binary-tree PLUQ forms are also applicable in broader numerical, algebraic, and scientific computing contexts. Applications include fast inversion of banded/“almost” banded matrices, dense block algorithms in HSS or $1 \leq k \leq n{-}1$9 solvers, and algebraic algorithms where quasiseparable structure recurs (Pernet, 2016).

7. Connections to Nearby Structured Matrix Classes

Quasiseparable matrices strictly generalize semiseparable, low-rank, banded, and many classical structured matrices (Toeplitz, Hankel, Vandermonde, Cauchy) through their low recurrence width and displacement rank properties. In the displacement framework: $r_L$0 low displacement rank with quasiseparable $r_L$1 yields new algorithmic classes with unified fast matrix-vector operations. This includes, among others, transforms used in orthogonal polynomial models, block companion, and multivariate evaluation matrices. The same divide-and-conquer algorithmic machinery thus uniformly applies across much of structured linear algebra and fast signal processing (Sa et al., 2016).

---

References:

• “Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers” (Hwang et al., 2024)
• “Computing with quasiseparable matrices” (Pernet, 2016)
• “A Two Pronged Progress in Structured Dense Matrix Multiplication” (Sa et al., 2016)