LMUs: 10x Data Efficiency at Small Scale
This lightning talk examines a recurrent architecture that challenges Transformer dominance in the small-to-medium model regime. By combining Legendre Memory Units—a fixed dynamical system that compresses recent history into orthogonal basis functions—with attention over compressed state rather than raw tokens, the authors achieve Transformer-comparable language modeling loss using one-tenth the training data. We explore the architectural mechanisms, scaling evidence, context-length trade-offs, and the critical open question: does this advantage persist beyond one million parameters?Script
At one million parameters, a Legendre Memory Unit model matches a Transformer's language modeling performance using just one-tenth the training tokens. That's 13 billion tokens doing the work of 130 billion, a tenfold data-efficiency gap that challenges our assumptions about what architectural priors buy you at small scale.
The Legendre Memory Unit encodes sliding-window history as coefficients in a fixed polynomial basis, much like a Fourier transform decomposes a signal into frequencies. The state is a matrix with embedding dimension down one axis and memory order along the other, preserving high-fidelity temporal structure without learning recurrent weights.
Instead of attending over token positions, the architecture applies query, key, and value projections to the compressed memory state itself. Attention happens within the small memory-basis dimension, not across sequence length, so there's no quadratic attention matrix. The authors call this implicit self-attention, though it sacrifices direct token-level access for linear complexity.
The scaling curves are smooth power laws, but the advantage isn't primarily in the exponent. The LMU curve sits below the Transformer and LSTM references across the tested range, up to one million non-embedding parameters. The intercept is better, not necessarily the asymptotic slope, and the paper stops exactly where the interesting question begins: does this hold at ten million parameters, or a billion?
The pure LMU improves rapidly over short contexts but plateaus around 100 tokens, revealing that compressed state alone cannot exploit arbitrarily long dependencies. Adding conventional global attention restores the smooth context-length improvement, but it also restores quadratic complexity. The two mechanisms are complementary, not redundant, and combining them sacrifices the very scaling advantage that motivated the architecture.
The evidence is compelling within a narrow window: small to medium models with strong temporal priors need far fewer tokens to reach a given loss. But the tested regime spans just three orders of magnitude in parameter count, stopping exactly where Transformers begin their steep ascent. Whether Legendre memory continues to outscale attention at a billion parameters remains an open question, and until it's answered, this is a small-scale efficiency win, not a universal architecture shift. Visit EmergentMind.com to explore the full paper and generate your own videos on the latest research.
