Researchers are working to increase sequence length in machine learning foundation models to enable learning from longer contexts and multiple media sources.
New models like S4, H3, and Hyena have been developed to address the quadratic scaling of attention layers in Transformers and show promising results in matching Transformers on perplexity and downstream tasks.
Sequence length: The length of input data in a machine learning model, which researchers are working to increase for improved learning from longer contexts and multiple media sources.
Foundation models: Machine learning models that researchers are aiming to enable learning from longer contexts, multiple media sources, complex demonstrations, and more.
Transformers: A type of machine learning model whose attention layers scale quadratically in sequence length, leading to increased computational costs as sequence length grows.
S4: A new sequence model based on structured state space models (SSMs) that scale with O(NlogN) in sequence length, introduced to better model long-range dependencies.
Hyena: The latest architecture in the line of work, replacing attention layers in H3 to create a nearly linear-time convolutional model that can match Transformers on perplexity and downstream tasks.