---
title: CLRS Algorithmic Reasoning Benchmark
url: https://www.emergentmind.com/topics/clrs-algorithmic-reasoning-benchmark
type: topic
---

# CLRS Algorithmic Reasoning Benchmark

The **CLRS Algorithmic Reasoning Benchmark** is a unified benchmark for **neural algorithmic reasoning** that evaluates whether neural models can **learn and execute classical algorithms**, rather than merely fit static question-answer datasets. Introduced as **CLRS-30**, it consolidates a previously fragmented literature into a common graph-based format with standardized tasks, intermediate execution traces, and out-of-distribution evaluation on larger problem sizes. Its core emphasis is algorithm execution, algorithmic generalization, and comparability across architectures and training regimes [2205.15659].

## 1. Origins and rationale

CLRS was proposed in response to a fragmented research landscape in which different papers used bespoke tasks, incompatible input encodings, different intermediate supervision targets, and inconsistent train/test protocols. The benchmark was designed to provide a **single benchmark suite** spanning classical algorithms from *Introduction to Algorithms* and to support in-distribution evaluation, out-of-distribution extrapolation to larger problem sizes, analysis of inductive bias, transfer across related tasks, and future extension with new algorithms [2205.15659].

The benchmark’s central premise is that algorithmic reasoning is especially suitable for rigorous evaluation because the input generation process, target function, and intermediate computational steps are all known. CLRS therefore formalizes algorithm learning as prediction over **inputs**, **hints** (intermediate states), and **outputs**, enabling evaluation of whether a model reproduces structured computation rather than only final answers [2205.15659].

A broader implication, explicitly developed in later work, is that CLRS became the canonical testbed for studying whether neural models can generalize to **larger graphs / arrays / problem sizes** than those seen during training, rather than only interpolate within a fixed dataset [2306.13411].

## 2. Graph-based specification and execution traces

A defining design choice of CLRS is its **graph-oriented format**. Even non-graph algorithms are mapped to graphs: in sorting, each array element becomes a node; in string matching, each character becomes a node; and in graph problems, nodes are the original graph vertices. Features are organized along three axes—**stage** (input, output, or hints), **location** (node, edge, or graph-level), and **type**—and each feature is represented as a probe
\[
(\text{stage}, \text{loc}, \text{type}, \text{values}).
\]
The five feature types are `scalar`, `categorical`, `mask`, `mask_one`, and `pointer` [2205.15659].

Each node also receives a **position scalar input probe** in \([0,1]\), which uniquely indexes nodes and serves as a tie-breaker when the algorithm has arbitrary choices. This positional design later became a point of methodological scrutiny, because subsequent work argued that the deterministic scalar index
\[
\text{pos}_{v_i} = \frac{i-1}{n}
\]
induces an input representation shift when graph size changes [2211.00692].

CLRS includes **hints** as intermediate algorithm states or trajectories. These are intended as compressed representations of the algorithm’s internal state, and the benchmark aims for a hint trajectory where step \(t+1\) is predictable from step \(t\) via roughly one message-passing step. The benchmark paper emphasizes that hints matter for studying compositionality, long-range rollouts, out-of-distribution generalization, and reuse of subroutines [2205.15659].

The worked example in the benchmark paper is **insertion sort**. Rather than requiring direct numerical prediction of literal swaps, the benchmark encodes the evolving list structure using **predecessor pointers**, with hints such as `iter`, `slot`, and `pred`. This illustrates the benchmark’s preference for structural representations aligned with algorithmic state transitions [2205.15659].

## 3. Task coverage, generation, and evaluation protocol

CLRS-30 contains **30 classical algorithms** spanning sorting, searching, divide and conquer, greedy methods, dynamic programming, graph algorithms, string algorithms, and geometry [2205.15659].

| Family | Example algorithms | Scope |
|---|---|---|
| Sorting | insertion sort, bubble sort, heapsort, quicksort | comparison-based sequence algorithms |
| Searching and selection | minimum, binary search, quickselect | ordered and unordered lookup tasks |
| Dynamic programming and divide and conquer | maximum subarray, matrix chain multiplication, longest common subsequence, optimal binary search tree | structured optimization |
| Graph, string, and geometry | BFS, DFS, Dijkstra, Bellman-Ford, Prim, Kruskal, topological sort, SCC, naïve string matching, KMP, segment intersection, Graham scan, Jarvis’ march | relational and combinatorial procedures |

The first version of CLRS explicitly excludes **NP-hard / approximation problems**, tasks with **numerical outputs** as primary outputs, standalone **data structures**, and algorithms requiring **dynamically allocated auxiliary memory** not naturally attached to the input objects [2205.15659].

Data are generated **synthetically** by running textbook algorithms on randomly sampled inputs satisfying each algorithm’s preconditions. Graph tasks are typically generated from random graphs, and scalar inputs are sampled from \(U(0,1)\). The benchmark paper reports that CLRS-30 is about **4.5 GB** uncompressed and takes about **1 hour** to generate [2205.15659].

The canonical split is explicitly out-of-distribution in scale:

- **Training**: inputs with **16 nodes**, **1,000 trajectories**
- **Validation**: also **16 nodes**, **32 trajectories**
- **Test**: inputs with **64 nodes**, **32 trajectories**

For graph-level outputs, more trajectories are generated to balance the number of targets [2205.15659].

Models are trained with **teacher forcing**, **Adam**, batch size **32**, learning rate \(0.001\), for **10,000 steps** with early stopping on validation performance. The main metric is **micro-F1** on predicted probes and outputs. The benchmark also defines a **win/tie/loss** comparison based on per-algorithm means and standard deviations [2205.15659].

## 4. Baselines and the core empirical challenge

The original benchmark evaluates several baseline architectures in an **encode-process-decode** framework: **Deep Sets**, **GAT / GATv2**, **MPNN**, **PGN**, and **Memory Networks**. In the out-of-distribution 64-node setting, the reported average scores are approximately **42.72%** for Deep Sets, **43.17%** for GAT, **38.88%** for Memnet, **44.99%** for MPNN, and **50.84%** for PGN. The corresponding win/tie/loss counts are \(0/3/27\), \(1/5/24\), \(4/2/24\), \(8/3/19\), and \(8/6/16\), respectively [2205.15659].

These results established a recurring pattern in the CLRS literature: strong in-distribution fitting is substantially easier than out-of-distribution algorithmic generalization. The benchmark paper reports that MPNNs perform very strongly on the 16-node validation setting, often exceeding 90% F1, but that performance drops substantially on the 64-node test setting. PGN becomes the best overall model on average, yet many tasks remain far from solved [2205.15659].

The benchmark also surfaced specific open difficulties. The original paper identifies **long-range rollouts / DFS-like tasks**, **recursive reasoning** such as **quicksort** and **quickselect**, and **string matching**, especially **KMP**, as persistent challenges [2205.15659]. Later work retained CLRS-30 as the standard OOD testbed and substantially raised reported scores. For example, **Triplet-GMPNN** reached **74.14%** average OOD micro-F1 [2209.11142], **G-ForgetNet** reported **82.89%** average OOD micro-F1 [2403.04929], and **FloydNet** reported **90.13%** on the CLRS-30 test set, with **FloydNet\(_{full}\)** at **96.64%** [2601.19094]. These later numbers indicate that the benchmark continued to discriminate among increasingly strong algorithmic reasoners.

## 5. Critiques, methodological disputes, and benchmark-driven revisions

CLRS became influential not only because it standardized evaluation, but also because it exposed benchmark-design issues that later work analyzed explicitly. One line of critique argued that the deterministic node position feature creates an **input distribution shift** when graph size changes, that the graph generator is often **non-representative**, and that in-distribution validation can be **uninformative** for OOD model selection [2211.00692]. That work proposed increasing the training set from **1,000 to 100,000** samples, replacing deterministic positions with **Random Scalar Index**, and introducing controlled variants such as **L-CLRS-Len**, **L-CLRS-Deg**, and **L-CLRS-Len-Deg** on \(K\)-regular graphs [2211.00692].

A second dispute concerns the role of **hints**. In the original benchmark, hints are a central mechanism for dense supervision [2205.15659]. Subsequent papers, however, reached different conclusions. One study reported that hints **usually hurt OOD performance** and therefore disabled them in its main experiments [2211.00692]. Another argued that strong reasoners can be trained **without intermediate supervision**, using only input-output pairs plus architectural changes and a self-supervised invariance objective, and reported **98.74% F1** on sorting [2306.13411]. FloydNet similarly reported that hints are **often not helpful** and can hurt performance because memory overhead forces shallower models; its hint-vs-no-hint table includes **Floyd-Warshall: 64.20 with hints vs 96.20 without hints** and **Binary Search: 24.70 with hints vs 94.47 without hints** [2601.19094].

A third line of work used CLRS to study architectural bias directly. Examples include **Hint-ReLIC**, which reported **up to \(3\times\)** improvement on OOD data by using hints as a self-supervised causal regularizer rather than ordinary auxiliary supervision [2302.10258]; **ForgetNet** and **G-ForgetNet**, which argued that historical latent reuse conflicts with the **Markov property** of algorithmic reasoning tasks [2403.04929]; and **RNAR**, which challenged permutation-invariant aggregation and reported a new state of the art on **Quickselect** with **87.08% mean micro-F1** [2409.07154]. This suggests that CLRS has functioned not merely as a leaderboard, but as a controlled environment for testing competing inductive biases.

## 6. Extensions, descendants, and broader influence

Several later benchmarks and model families are direct descendants or reinterpretations of CLRS. The most direct textual extension is **CLRS-Text**, which serializes CLRS traces into plain text for language models while retaining the same **30 default algorithms**, procedural generation, and controlled interpolation/extrapolation design [2406.04229]. CLRS-Text evaluates language models as **generalist executors** with a standard **next-token prediction** objective and **exact string match** evaluation. Its main conclusion is that language models generalize much worse on CLRS-Text than graph models did on CLRS: in CLRS-Text, language models **“barely extrapolate at all”** [2406.04229].

A second major descendant is **SALSA-CLRS**, which reworks CLRS around sparse execution and scalability. It keeps the CLRS style of inputs, hints, and outputs, but replaces the dense fully connected execution model with sparse graphs, adapts four original CLRS tasks—**BFS**, **DFS**, **Dijkstra**, and **maximum spanning tree**—and adds **Distributed Maximal Independent Set** and **Distributed Eccentricity**. Its test sizes are \(n \in \{16, 80, 160, 800, 1600\}\), and it evaluates on **Erdős–Rényi**, **Watts–Strogatz**, and **Delaunay** graphs [2309.12253]. This extension is explicitly motivated by the claim that the dense execution model of CLRS becomes a memory and runtime bottleneck [2309.12253].

CLRS also became a foundation for research on **generalist** and **multitask** reasoning. A generalist neural algorithmic learner showed that a single shared processor can be trained across all 30 tasks and improved average single-task performance to **74.14%** OOD micro-F1 [2209.11142]. Later work on **Open-Book Neural Algorithmic Reasoning** reported an average of **82.91%** by allowing the model to attend over training instances during inference [2501.00072]. Still later, **AutoBRANE** studied structured parameter sharing for multitask CLRS and reported that, on CLRS, it outperforms the strongest single multitask GNN by **3.7%**, improves over the best baseline by **1.2%**, reduces runtime by **48%**, and reduces memory usage by **26%** [2512.01113].

Taken together, these extensions indicate that CLRS has evolved from a benchmark into a family of evaluation paradigms: dense graph execution, sparse scalable execution, textual execution traces, multitask transfer, retrieval-augmented reasoning, and equilibrium or reinforcement-learning reinterpretations. A plausible implication is that CLRS’s enduring importance lies less in any single protocol detail than in its role as a common formal substrate for studying **algorithm execution**, **intermediate state prediction**, and **out-of-distribution generalization** across model classes [2205.15659].

Source: https://www.emergentmind.com/topics/clrs-algorithmic-reasoning-benchmark