CLRS Algorithmic Reasoning Benchmark
- CLRS Algorithmic Reasoning Benchmark is a unified suite that standardizes algorithm execution via graph-based tasks and intermediate state traces.
- It consolidates diverse evaluation protocols across sorting, dynamic programming, and graph algorithms into a common framework for rigorous OOD generalization analysis.
- Empirical results highlight strong in-distribution performance with marked drops on out-of-distribution tasks, driving advances in neural algorithmic reasoning models.
The CLRS Algorithmic Reasoning Benchmark is a unified benchmark for neural algorithmic reasoning that evaluates whether neural models can learn and execute classical algorithms, rather than merely fit static question-answer datasets. Introduced as CLRS-30, it consolidates a previously fragmented literature into a common graph-based format with standardized tasks, intermediate execution traces, and out-of-distribution evaluation on larger problem sizes. Its core emphasis is algorithm execution, algorithmic generalization, and comparability across architectures and training regimes (Veličković et al., 2022).
1. Origins and rationale
CLRS was proposed in response to a fragmented research landscape in which different papers used bespoke tasks, incompatible input encodings, different intermediate supervision targets, and inconsistent train/test protocols. The benchmark was designed to provide a single benchmark suite spanning classical algorithms from Introduction to Algorithms and to support in-distribution evaluation, out-of-distribution extrapolation to larger problem sizes, analysis of inductive bias, transfer across related tasks, and future extension with new algorithms (Veličković et al., 2022).
The benchmark’s central premise is that algorithmic reasoning is especially suitable for rigorous evaluation because the input generation process, target function, and intermediate computational steps are all known. CLRS therefore formalizes algorithm learning as prediction over inputs, hints (intermediate states), and outputs, enabling evaluation of whether a model reproduces structured computation rather than only final answers (Veličković et al., 2022).
A broader implication, explicitly developed in later work, is that CLRS became the canonical testbed for studying whether neural models can generalize to larger graphs / arrays / problem sizes than those seen during training, rather than only interpolate within a fixed dataset (Rodionov et al., 2023).
2. Graph-based specification and execution traces
A defining design choice of CLRS is its graph-oriented format. Even non-graph algorithms are mapped to graphs: in sorting, each array element becomes a node; in string matching, each character becomes a node; and in graph problems, nodes are the original graph vertices. Features are organized along three axes—stage (input, output, or hints), location (node, edge, or graph-level), and type—and each feature is represented as a probe
The five feature types are scalar, categorical, mask, mask_one, and pointer (Veličković et al., 2022).
Each node also receives a position scalar input probe in , which uniquely indexes nodes and serves as a tie-breaker when the algorithm has arbitrary choices. This positional design later became a point of methodological scrutiny, because subsequent work argued that the deterministic scalar index
induces an input representation shift when graph size changes (Mahdavi et al., 2022).
CLRS includes hints as intermediate algorithm states or trajectories. These are intended as compressed representations of the algorithm’s internal state, and the benchmark aims for a hint trajectory where step is predictable from step via roughly one message-passing step. The benchmark paper emphasizes that hints matter for studying compositionality, long-range rollouts, out-of-distribution generalization, and reuse of subroutines (Veličković et al., 2022).
The worked example in the benchmark paper is insertion sort. Rather than requiring direct numerical prediction of literal swaps, the benchmark encodes the evolving list structure using predecessor pointers, with hints such as iter, slot, and pred. This illustrates the benchmark’s preference for structural representations aligned with algorithmic state transitions (Veličković et al., 2022).
3. Task coverage, generation, and evaluation protocol
CLRS-30 contains 30 classical algorithms spanning sorting, searching, divide and conquer, greedy methods, dynamic programming, graph algorithms, string algorithms, and geometry (Veličković et al., 2022).
| Family | Example algorithms | Scope |
|---|---|---|
| Sorting | insertion sort, bubble sort, heapsort, quicksort | comparison-based sequence algorithms |
| Searching and selection | minimum, binary search, quickselect | ordered and unordered lookup tasks |
| Dynamic programming and divide and conquer | maximum subarray, matrix chain multiplication, longest common subsequence, optimal binary search tree | structured optimization |
| Graph, string, and geometry | BFS, DFS, Dijkstra, Bellman-Ford, Prim, Kruskal, topological sort, SCC, naïve string matching, KMP, segment intersection, Graham scan, Jarvis’ march | relational and combinatorial procedures |
The first version of CLRS explicitly excludes NP-hard / approximation problems, tasks with numerical outputs as primary outputs, standalone data structures, and algorithms requiring dynamically allocated auxiliary memory not naturally attached to the input objects (Veličković et al., 2022).
Data are generated synthetically by running textbook algorithms on randomly sampled inputs satisfying each algorithm’s preconditions. Graph tasks are typically generated from random graphs, and scalar inputs are sampled from . The benchmark paper reports that CLRS-30 is about 4.5 GB uncompressed and takes about 1 hour to generate (Veličković et al., 2022).
The canonical split is explicitly out-of-distribution in scale:
- Training: inputs with 16 nodes, 1,000 trajectories
- Validation: also 16 nodes, 32 trajectories
- Test: inputs with 64 nodes, 32 trajectories
For graph-level outputs, more trajectories are generated to balance the number of targets (Veličković et al., 2022).
Models are trained with teacher forcing, Adam, batch size 32, learning rate $0.001$, for 10,000 steps with early stopping on validation performance. The main metric is micro-F1 on predicted probes and outputs. The benchmark also defines a win/tie/loss comparison based on per-algorithm means and standard deviations (Veličković et al., 2022).
4. Baselines and the core empirical challenge
The original benchmark evaluates several baseline architectures in an encode-process-decode framework: Deep Sets, GAT / GATv2, MPNN, PGN, and Memory Networks. In the out-of-distribution 64-node setting, the reported average scores are approximately 42.72% for Deep Sets, 43.17% for GAT, 38.88% for Memnet, 44.99% for MPNN, and 50.84% for PGN. The corresponding win/tie/loss counts are $0/3/27$, $1/5/24$, $4/2/24$, 0, and 1, respectively (Veličković et al., 2022).
These results established a recurring pattern in the CLRS literature: strong in-distribution fitting is substantially easier than out-of-distribution algorithmic generalization. The benchmark paper reports that MPNNs perform very strongly on the 16-node validation setting, often exceeding 90% F1, but that performance drops substantially on the 64-node test setting. PGN becomes the best overall model on average, yet many tasks remain far from solved (Veličković et al., 2022).
The benchmark also surfaced specific open difficulties. The original paper identifies long-range rollouts / DFS-like tasks, recursive reasoning such as quicksort and quickselect, and string matching, especially KMP, as persistent challenges (Veličković et al., 2022). Later work retained CLRS-30 as the standard OOD testbed and substantially raised reported scores. For example, Triplet-GMPNN reached 74.14% average OOD micro-F1 (Ibarz et al., 2022), G-ForgetNet reported 82.89% average OOD micro-F1 (Bohde et al., 2024), and FloydNet reported 90.13% on the CLRS-30 test set, with FloydNet2 at 96.64% (Yu et al., 27 Jan 2026). These later numbers indicate that the benchmark continued to discriminate among increasingly strong algorithmic reasoners.
5. Critiques, methodological disputes, and benchmark-driven revisions
CLRS became influential not only because it standardized evaluation, but also because it exposed benchmark-design issues that later work analyzed explicitly. One line of critique argued that the deterministic node position feature creates an input distribution shift when graph size changes, that the graph generator is often non-representative, and that in-distribution validation can be uninformative for OOD model selection (Mahdavi et al., 2022). That work proposed increasing the training set from 1,000 to 100,000 samples, replacing deterministic positions with Random Scalar Index, and introducing controlled variants such as L-CLRS-Len, L-CLRS-Deg, and L-CLRS-Len-Deg on 3-regular graphs (Mahdavi et al., 2022).
A second dispute concerns the role of hints. In the original benchmark, hints are a central mechanism for dense supervision (Veličković et al., 2022). Subsequent papers, however, reached different conclusions. One study reported that hints usually hurt OOD performance and therefore disabled them in its main experiments (Mahdavi et al., 2022). Another argued that strong reasoners can be trained without intermediate supervision, using only input-output pairs plus architectural changes and a self-supervised invariance objective, and reported 98.74% F1 on sorting (Rodionov et al., 2023). FloydNet similarly reported that hints are often not helpful and can hurt performance because memory overhead forces shallower models; its hint-vs-no-hint table includes Floyd-Warshall: 64.20 with hints vs 96.20 without hints and Binary Search: 24.70 with hints vs 94.47 without hints (Yu et al., 27 Jan 2026).
A third line of work used CLRS to study architectural bias directly. Examples include Hint-ReLIC, which reported up to 4 improvement on OOD data by using hints as a self-supervised causal regularizer rather than ordinary auxiliary supervision (Bevilacqua et al., 2023); ForgetNet and G-ForgetNet, which argued that historical latent reuse conflicts with the Markov property of algorithmic reasoning tasks (Bohde et al., 2024); and RNAR, which challenged permutation-invariant aggregation and reported a new state of the art on Quickselect with 87.08% mean micro-F1 (Xu et al., 2024). This suggests that CLRS has functioned not merely as a leaderboard, but as a controlled environment for testing competing inductive biases.
6. Extensions, descendants, and broader influence
Several later benchmarks and model families are direct descendants or reinterpretations of CLRS. The most direct textual extension is CLRS-Text, which serializes CLRS traces into plain text for LLMs while retaining the same 30 default algorithms, procedural generation, and controlled interpolation/extrapolation design (Markeeva et al., 2024). CLRS-Text evaluates LLMs as generalist executors with a standard next-token prediction objective and exact string match evaluation. Its main conclusion is that LLMs generalize much worse on CLRS-Text than graph models did on CLRS: in CLRS-Text, LLMs “barely extrapolate at all” (Markeeva et al., 2024).
A second major descendant is SALSA-CLRS, which reworks CLRS around sparse execution and scalability. It keeps the CLRS style of inputs, hints, and outputs, but replaces the dense fully connected execution model with sparse graphs, adapts four original CLRS tasks—BFS, DFS, Dijkstra, and maximum spanning tree—and adds Distributed Maximal Independent Set and Distributed Eccentricity. Its test sizes are 5, and it evaluates on Erdős–Rényi, Watts–Strogatz, and Delaunay graphs (Minder et al., 2023). This extension is explicitly motivated by the claim that the dense execution model of CLRS becomes a memory and runtime bottleneck (Minder et al., 2023).
CLRS also became a foundation for research on generalist and multitask reasoning. A generalist neural algorithmic learner showed that a single shared processor can be trained across all 30 tasks and improved average single-task performance to 74.14% OOD micro-F1 (Ibarz et al., 2022). Later work on Open-Book Neural Algorithmic Reasoning reported an average of 82.91% by allowing the model to attend over training instances during inference (Li et al., 2024). Still later, AutoBRANE studied structured parameter sharing for multitask CLRS and reported that, on CLRS, it outperforms the strongest single multitask GNN by 3.7%, improves over the best baseline by 1.2%, reduces runtime by 48%, and reduces memory usage by 26% (Li et al., 30 Nov 2025).
Taken together, these extensions indicate that CLRS has evolved from a benchmark into a family of evaluation paradigms: dense graph execution, sparse scalable execution, textual execution traces, multitask transfer, retrieval-augmented reasoning, and equilibrium or reinforcement-learning reinterpretations. A plausible implication is that CLRS’s enduring importance lies less in any single protocol detail than in its role as a common formal substrate for studying algorithm execution, intermediate state prediction, and out-of-distribution generalization across model classes (Veličković et al., 2022).