Long Range Arena: Benchmark for Sequence Models

Updated 4 March 2026

Long Range Arena (LRA) is a benchmark designed for evaluating neural sequence models on tasks with long-range dependencies, spanning modalities such as language, vision, and reasoning.
It systematically tests models’ algorithmic efficiency and expressive power using classification tasks with sequences up to 16,384 tokens, setting rigorous standards for performance.
LRA has driven architectural innovations—from efficient attention mechanisms to structured state-space models—pushing forward long-context processing in modern AI systems.

The Long Range Arena (LRA) is a widely adopted benchmark designed to evaluate and compare the ability of neural sequence models—especially transformers and efficiency-oriented variants—to capture long-range dependencies across a diverse suite of tasks and input modalities. It has been central to the development of efficient architectures for modeling sequences spanning thousands to tens of thousands of steps, providing both a unified task platform and a de facto leaderboard for long-context model comparisons. Over time, LRA has catalyzed both architectural progress and nuanced methodological debates about the nature of "long-range" computation itself, its actual demands in practical data, and the roles of architectural inductive biases, data augmentation, and positional encoding in mediating performance.

1. Formal Definition and Motivation

The Long Range Arena benchmark, proposed by Tay et al. (Tay et al., 2020), targets the limitations of standard Transformers—specifically, the $O(n^2)$ compute and memory scaling of self-attention with respect to input sequence length $n$ . LRA was constructed to systematically evaluate the ability of sequence models to process inputs of extreme length (up to 16,000 tokens) under moderate resource constraints and in the absence of large-scale pretraining.

LRA suite composition:

Six encoder-only tasks spanning synthetic language (ListOps), language (byte-level classification, document retrieval), vision (flattened CIFAR-10), and relational reasoning (Pathfinder, Pathfinder-X).
Sequence lengths range from 1,024 (images, Pathfinder), through 2,000–4,096 (ListOps, Text, Retrieval), to 16,384 (Pathfinder-X).
All tasks are cast as classification, measuring the model’s top-1 accuracy on test data.

The benchmark aims to probe both the expressive power of sequence models (can a model, in principle, solve long-range relational or compositional tasks?) and their algorithmic efficiency (can this be accomplished with sub-quadratic scaling, permitting practical training and inference on long inputs?) (Tay et al., 2020).

2. Benchmark Tasks and Evaluation Protocol

LRA defines a fixed set of tasks that collectively span a variety of dependency structures and input types (Tay et al., 2020, Zhang et al., 2022, Miralles-González et al., 24 Jan 2025):

Task	Sequence Length	Input Modality	Core Challenge	Output Type
ListOps	2,000–2,048	Synthetic tokens	Hierarchical parsing	10-class
Text	2,000–4,096	Byte-level text	Document sentiment	Binary
Retrieval	2,000–8,192	Byte-level doc pair	Information retrieval	Binary
Image	1,024	Flattened pixels	Visual pattern (CIFAR)	10-class
Pathfinder	1,024	Flattened grid	Path connectivity	Binary
Pathfinder-X	16,384	Flattened grid	Long path reasoning	Binary

All tasks utilize accuracy as the principal metric; no log-likelihoods or sequence generation tasks are included (Tay et al., 2020, Zhang et al., 2022). Vanilla Transformers with standard attention mechanisms and positional embeddings serve as the canonical (but intentionally non-optimized) baseline.

3. Architectural Developments Driven by LRA

LRA has facilitated a large-scale, controlled comparison of efficient transformer variants, structured state-space models, convolutional surrogates, and non-attention architectures, with profound impact on model design and analytical understanding.

Key architectural categories evaluated and advanced on LRA:

Efficient self-attention mechanisms: Sparse patterns (Longformer, BigBird), low-rank approximations (Linformer), kernel-based (Performer), random-feature, and sketching-based models (Skeinformer, S³Attention) (Tay et al., 2020, Chen et al., 2021, Wang et al., 2024).
Structured State-Space Models (SSMs): Diagonal or low-rank canonical SSMs (S4, S4D, S5, S6/Mamba, B₂S₆, HOPE) leveraging fast convolution/FFT techniques and HiPPO-inspired memory kernels to achieve asymptotic $O(n)$ cost (Gu et al., 2021, Yu et al., 2024, Yu et al., 13 May 2025).
Hybrid and neuro-inspired systems: BLRP's bidirectional local-global parsing (Leotescu et al., 2024), astrocyte-inspired RMAAT architectures (Mia et al., 1 Jan 2026), and expressive artificial neuron models (ELM) for direct long-memory (Spieler et al., 2023).
Recursive and compositional RNN structures: Two-level nested recursion (RIR) models balancing tree-structured composition and efficiency (Chowdhury et al., 2023).

State-space models—particularly S4 and B₂S₆—have set SOTA results, demonstrating robust scaling to 16K tokens and effective long-convolutional memory, with B₂S₆ introducing principled amendments to Mamba-style SSMs via block-wise gating and channel bias to restore universality and improve gradient dynamics (Gu et al., 2021, Yu et al., 13 May 2025). Advanced experimental protocols have emerged: ablation studies (block, bias, complex parameters), sensitivity sweeps, and per-task hyperparameter optimization (Wang et al., 2024, Yu et al., 13 May 2025).

4. Core Results, Performance Trends, and Analysis

Canonical results firmly establish several trends:

Vanilla Transformers (with sinusoidal or learned positional embeddings and naive training) yield 54–55% mean accuracy across LRA, with catastrophic failures on extreme-length tasks (e.g., $\sim$ 50% on Pathfinder-X, effectively random) (Tay et al., 2020, Zhang et al., 2022, Miralles-González et al., 24 Jan 2025).
Efficient attention variants (Performer, Linformer, Longformer) and sparse convolutional surrogates reach 50–62% (Tay et al., 2020, Yuan et al., 2023, Chen et al., 2021).
Structured state-space models (S4, S4D, S5, B₂S₆, HOPE) achieve 86–88% mean, with near-perfect retention on long-sequence tasks (e.g., 96.35% on Pathfinder-X for S4). B₂S₆ attains SOTA on five out of six tasks without loss on language modeling performance (Gu et al., 2021, Yu et al., 2024, Yu et al., 13 May 2025).
Further optimization of Transformers (rotary encoding, multitask denoising, data augmentation) closes most of the gap, attaining ≈85–86% with appropriate inductive bias and regularization (Miralles-González et al., 24 Jan 2025).
Hybrid architectures (Diffuser, S³Attention, BLRP) approach or exceed efficient-attention SOTA with linear-in- $n$ memory, via spectral diffusion, Fourier smoothing, or latent cross-attention (Feng et al., 2022, Wang et al., 2024, Leotescu et al., 2024).

Summary table (representative LRA test accuracies, bold=best):

Model	ListOps	Text	Retrieval	Image	Pathfinder	Path-X	Avg.
Transformer	36.4	64.3	57.5	42.4	71.4	55.0	54.4
S4	59.6	86.8	90.9	88.7	94.2	96.4	86.1
S5	62.2	89.3	91.4	88.0	95.3	98.6	87.5
B₂S₆	63.9	88.3	91.4	88.8	95.9	97.9	87.7
HOPE-SSM	62.6	89.8	91.8	88.7	95.7	98.5	87.9

This table is derived from the results presented in (Yu et al., 13 May 2025, Gu et al., 2021, Yu et al., 2024), and reflects the strongest averages among non-pretrained models.

5. Insights on Inductive Bias, Locality, and True Long-Range Modeling

Research leveraging LRA has illuminated the critical role of model inductive bias and provided a nuanced interpretation of what constitutes "long-range" (Zimerman et al., 2023, Miralles-González et al., 24 Jan 2025):

Inductive bias toward local, smooth, and time-decaying patterns (either via convolutional kernels, exponential-moving-average, or rotary-position encoding) is both necessary and sufficient to attain high performance on most LRA tasks (Zimerman et al., 2023).
State-space model parameterizations (HiPPO, Hankel, low-rank) serve to regularize and efficiently allocate memory resources, but fully learnable unconstrained global convolutions, when combined with strong regularization and data augmentation, match S4/MEGA performance, indicating that the SSM structure per se is not a strict requirement (Miralles-González et al., 24 Jan 2025).
Empirical ablations show that short-range dependencies frequently dominate, and simple 1D convolutional models with bounded receptive fields (even kernels of 5–61) achieve near state-of-the-art, challenging the interpretation of LRA as a "pure" long-range memory suite (Miralles-González et al., 24 Jan 2025).
On compositional or deeply-structured tasks (e.g., ListOps), advanced sequence models excel by restoring effective width or balancing block selectivity and channel bias (B₂S₆), in contrast to architectures that entangle all channels through input-dependent gating.

6. Limitations, Critiques, and Benchmark Evolution

Recent literature has scrutinized the LRA paradigm and its ability to meaningfully measure true long-range dependency modeling (Miralles-González et al., 24 Jan 2025, Zhang et al., 2022):

Task design: All LRA tasks use non-causal self-attention and do not test cross-attention or causal/auto-regressive settings, limiting generality to broader applications (Zhang et al., 2022).
Locality dominance: Observed high performance with small-receptive-field convolutions and properly regularized transformers suggests that most tasks are solvable with predominantly local information.
Positional encoding: Absolute and sinusoidal embeddings are substantially outperformed by rotary or relative positional methods in the long-context regime, underscoring the importance of encoding temporal locality and decay (Miralles-González et al., 24 Jan 2025).
Data efficiency and pretraining: Augmentations, multitask auxiliary objectives (MLM), and denoising pretraining alter the apparent capacity and close the SSM/Transformer gap, raising concerns over uncontrolled confounds in comparative studies.
Calls for next-generation benchmarks: Recommendations include explicit parametrization of synthetic dependency distance/separation, graph-based or symbolic tasks where dependency span can be systematically varied and measured, and comprehensive evaluation of all attention patterns—non-causal, causal, cross-modal—under resource constraints [(Miralles-González et al., 24 Jan 2025), 2210