Linear Complexity Sequence Models

• Linear Complexity Sequence Models are mathematical frameworks and neural architectures that model sequences via linear recurrence relations and minimal polynomials.
• They leverage algebraic and combinatorial tools, such as LFSRs, Hankel determinants, and deficiency measures, to analyze sequence predictability and cryptographic resilience.
• Modern implementations extend these models into deep learning with innovations like linear attention and state-space models, achieving efficient scaling and robust performance.

Linear complexity sequence models encompass a broad class of mathematical frameworks and machine learning architectures characterizing or leveraging sequence predictability subject to linear constraints and recurrences. At their theoretical core, these models analyze or synthesize sequences whose future evolution is constrained by linear recurrence relations of finite (and ideally minimal) order, typically assessed via algebraic or combinatorial tools such as linear feedback shift-registers (LFSRs), minimal polynomials, Hankel determinants, and extensions to multidimensional and nonlinear structures. In modern applications, linear complexity sequence models include not only classical algebraic and combinatorial constructions (arising in cryptography, combinatorics, and theory of computation), but also extensive innovations in efficient neural architectures for massive-scale sequence modeling.

1. Algebraic Foundations: Linear Recurrences, Minimal Polynomials, and the Number Wall

The classical measure of sequence unpredictability is the minimal order $r$ for which a given sequence $(S_n)$ over a ring or field admits a linear recurrence,

$$\sum_{i=0}^r J_i S_{n+i} = 0,\quad \text{for all } n,$$

with $J_0\neq0$. This defines the minimal polynomial of the sequence and yields the so-called linear complexity profile (LCP), assigning to each prefix the smallest such $r$. The LCP forms the basis for LFSR-based cryptography and stream cipher analysis, but provides only a one-dimensional projection of local recurrence structure.

The number wall paradigm, introduced as a geometric alternative, synthesizes these LFSR relations across all intervals by forming a two-dimensional array of Hankel determinants: $$S_{m,n} = \det\begin{pmatrix} S_n & S_{n+1} & \cdots & S_{n+m} \ S_{n-1} & S_n & \cdots & S_{n+m-1} \ \vdots & \vdots & \ddots & \vdots \ S_{n-m} & S_{n-m+1} & \cdots & S_n \end{pmatrix}.$$ A zero at $(m,n)$ signals the existence of a nontrivial LFSR spanning $S_{n-m},\dots,S_{n+m}$; larger “windows” of zeros signify lower-order recurrence relations over larger spans. The Sylvester–Jacobi identity,

$$S_{m,n}^2 = S_{m+1,n}S_{m-1,n} + S_{m,n+1}S_{m,n-1},$$

enables efficient computation in nonvanishing wall regions and connects to numerical linear algebra.

This geometric “number wall” approach captures not just the local minimal order at each prefix (as in the LCP), but the global recurrence landscape—allowing detection of subtle, non-local recurrence structures and aiding the analysis of sequence randomization, combinatorial construction, and cryptographic strength (0906.3286).

2. Combinatorial Extremes: Deficiency, The Pagoda Sequence, and Aperiodic Tiling

Examining the number wall for combinatorially defined sequences immediately reveals extremal linear complexity behavior. A striking example is the ternary “Pagoda sequence”, a D0L (deterministic zero-context Lindenmayer system) extension of the Thue–Morse sequence, constructed by a morphic sequence augmented by a final mapping, for instance,

$P_n = R_{n+1} - R_{n-1} \pmod{3},$

where $R_n$ is derived from the binary development of $n$ (the rook sequence). This sequence’s ternary number wall exhibits “deficiency 2 modulo 3”: no interval of $2m+2$ consecutive symbols ever admits a recurrence of order $m$, so the largest block of zeros in the wall is $1\times1$. More generally, the deficiency measures the largest block of zeros (recurrence span) in the number wall, with deficiency 2 being maximal for $\mathbb{Z}_3$ sequences.

The proof leverages a deep link with aperiodic tilings: encoding number-wall entries as tiles, D0LEC morphisms generate plane tilings whose only zeros are isolated. The divisibility constraint

$v_2(m+2) > v_2(n),$

with $v_2(\cdot)$ denoting 2-adic valuation, ensures no extended zero-windows can occur—the spatial structure of the tiling rigidly constrains linear recurrences in the original sequence, providing tight cryptographic and combinatorial guarantees (0906.3286).

These links illustrate the hierarchy: $$\text{polynomial sequences} \to \text{LFSR sequences} \to \text{D0LEC sequences}$$ where each class rigidly extends the previous, and the “deficiency” property encodes their resistance to short LFSR approximations.

3. Algorithmic Tools: Minimal Polynomial Algorithms and Bézout Identities

Computationally, the linear complexity of finite sequences is determined via minimal polynomial algorithms (including efficient Berlekamp–Massey and Games–Chan variants for special periodicities). These algorithms recursively synthesize an LFSR, updating its minimal polynomial when discrepancies between predicted and actual sequence values arise. Algorithmic improvements yield time complexity $O(N)$ in special cases, and factorization-based frameworks further generalize such approaches (Chee et al., 2019).

For in-depth algebraic understanding, Bézout identities for minimal polynomials enable tight characterizations of linear complexity jump profiles and equivalence classes among sequences. For instance, a sequence has a perfect linear complexity profile (PLCP) if its complexity jumps by 1 at every odd index (i.e., $L_j-L_{j-1} = 1$ for odd $j$, $0$ otherwise) (Norton, 2011). This behavior is connected to the vanishing of even-indexed components in a stability transform, and is foundational in LFSR synthesis for optimal keystream sequences.

4. Structural Extensions: Expansion Complexity, Multidimensionality, and k-Error Robustness

Linear complexity profiles alone have limitations: for example, certain highly predictable $q$-automatic sequences (e.g., Thue–Morse, Rudin–Shapiro), though having linear complexity order $N$, are trivially generated by finite automata—highlighting the need for stronger measures (e.g., expansion complexity, correlation bounds) (Mérai et al., 2017, Mérai et al., 2016). Expansion complexity, introduced by Diem, analyzes the minimal total degree of polynomial relations satisfied by the sequence’s generating function, and is more sensitive than linear complexity to structure in short subsequences and aperiodic cases.

The extension to multidimensional sequences generalizes linear complexity to ideal theory in $\mathbb{F}_q[X_1,\dots,X_n]$. Here, the linear complexity is the dimension of the quotient ring modulo the sequence’s annihilator ideal, with probabilistic bounds showing high complexity is generic among periodic multidimensional sequences (Gómez-Pérez et al., 2018).

In cryptographic and coding applications, the notion of $k$-error linear complexity (the minimum complexity reachable after up to $k$ errors/alterations) is critical. Cube theory—decomposing a binary periodic sequence into combinatorial cubes—enables the explicit construction of sequences with maximum $k$-error linear complexity, with tight bounds $L_k(s) = 2^n-(2^l-1)$ for $2^{l-1}\leq k<2^l$ (Zhou, 2011).

5. Application-Driven Linear Complexity Models: Deep Learning, Attention, and Hardware-Aware Scaling

Modern linear complexity sequence models extend beyond algebraic sequences to encompass efficient sequence modeling in deep learning. These include linear attention mechanisms (Linformer, RetNet, GLA), state space models (Mamba2), and highly parallelizable bidirectional/multisource recurrent architectures (BLUR) (Wang et al., 2020, Liao et al., 28 May 2024, Liu et al., 11 Apr 2025). All share the haLLMark property:

• The per-step computation and memory for processing a sequence scale as $O(n)$ or even $O(1)$ at inference, where $n$ is the sequence length.

Unified frameworks (e.g., LCSM) resolve these models as instances of a generic linear update,

$$m_t = g_p(o_t, m_{t-1}) + e_t i_t^\top, \quad y_t = m_t^\top s_t,$$

with an EOS (Expand–Oscillation–Shrink) structure. The Expand step projects inputs to high-dimensional memory; Oscillation applies recursive, usually element-wise or matrix, transformations (mimicking LFSR dynamics with implicit or explicit “forget gates”/diagonal matrices); Shrink projects the memory to output space. Performance on dense prediction and retrieval tasks reveals that data-driven parameterizations of these steps yield best-in-class results for generative tasks, while hand-crafted schemes can improve retrieval (Qin et al., 27 May 2024).

Scaling to ultra-long contexts (up to millions of tokens) on distributed hardware utilizes novel sequence parallelism techniques—ZeCO’s All-Scan collective communication removes inter-device bottlenecks, transmitting only the minimal operator state required, and achieving near-linear scalability in practice (Chou et al., 1 Jul 2025).

The table summarizes structural paradigms and efficiency guarantees:

Model Family	Recurrence Structure/Update	Complexity
Number wall/LFSR	Hankel determinant, LFSR relation	$O(N^2)$/$O(N)$
Minimal polynomial alg.	Recursive update via discrepancies	$O(N)$
Linformer/Linear Attn.	Projected key/value, low-rank attn.	$O(n)$
LCSM/BLUR/MoM	EOS, bidirectional LRU, mixture-of-memory	$O(n)$
ZeCO SP (parallel)	All-Scan pipelined operator update	$O(n/P)$

6. Cryptographic and Algorithmic Implications

The paper and implementation of linear complexity sequence models have immediate applications:

• Cryptographic keystream generation, with guarantees of high minimal LFSR order, maximized $k$-error complexity, and resistance to shortcut attacks.
• Pseudorandom generator design, e.g., via elliptic or hyperelliptic curve mappings, achieving provably high linear complexity under algebraic group structure (Anupindi et al., 2021, Anupindi, 2022, Mérai et al., 2015).
• Optimized machine learning architectures (State Space Models, Linear/Hybrid MoE, Retentive Networks) supporting efficient scaling to long-range sequence dependence with minimal memory (Sun et al., 7 Mar 2025, Du et al., 19 Feb 2025).
• Efficient distributed large-scale training, guaranteed by optimal SP primitives like ZeCO, for next-generation foundation models handling unprecedented context lengths (Chou et al., 1 Jul 2025).

7. Interdisciplinary Synthesis: From Formal Structures to Machine Learning Systems

The proliferation of linear complexity sequence models marks a convergence of algebraic, combinatorial, analytic, and deep learning methodologies. Number wall structures model detailed LFSR behavior; cube decompositions enable error-resilient design; generalized state-space and attention models implement these concepts at extreme scale with real-world applications in language, vision, and forecasting. This synthesis demystifies both the mathematical underpinnings (recurrence, deficiency, tiling) and the system-level optimizations (hardware-aware, parallelized, mixture-of-experts) essential in modern AI deployments.

The current research trajectory increasingly emphasizes:

• Unified algebraic–neural representation schemes
• Provably robust memory architectures (e.g., MoM, Linear-MoE)
• Hardware–communication co-design for exascale sequential inference

Thus, linear complexity sequence models, in both their classical and modern incarnations, constitute the mathematical and computational backbone for the analysis, synthesis, and deployment of efficient, robust, and scalable sequence-processing systems in both theory and practice.