CLRS-Text Algorithmic Reasoning Benchmark

• CLRS-Text is a standardized, procedurally-generated text benchmark that evaluates algorithmic reasoning in language models using execution traces from 30 core algorithms.
• The benchmark converts graph-based algorithm traces into a text format, enabling detailed analysis of chain-of-thought, intermediate state accuracy, and out-of-distribution generalization.
• It offers granular evaluation metrics and cross-architecture comparisons, driving research into hybrid LM designs and memory-augmented reasoning approaches.

The CLRS-Text Algorithmic Reasoning Language Benchmark is a standardized, procedurally-generated suite for evaluating algorithmic reasoning in LMs, derived from the graph-based CLRS benchmark and targeting a language-based interface. By translating execution traces of thirty fundamental algorithms into a text serialization, CLRS-Text enables systematic investigation of the algorithmic capabilities, generalization errors, and failure modes of both pretrained and fine-tuned LMs on tasks spanning classical algorithms, with a primary focus on out-of-distribution (OOD) generalization and procedural fidelity (Markeeva et al., 2024).

1. Historical Background and Motivation

The original CLRS benchmark, introduced by Veličković et al., unified the evaluation of neural algorithmic reasoners, especially Graph Neural Networks (GNNs), by generating graph-structured execution traces for 30 textbook algorithms and enabling OOD testing across arbitrary problem sizes (Veličković et al., 2022). Despite strong adoption in the GNN/NAR community, prior benchmarks for algorithmic reasoning in LMs were typically synthetic, narrow in task variety, and static, impeding progress through fragmented and overfitted evaluations (Markeeva et al., 2024). CLRS-Text directly addresses these deficits by leveraging: (1) CLRS’s unified generator and task taxonomy; (2) conversion of traces to compact, language-model-friendly text suitable for next-token prediction; and (3) a procedural pipeline allowing the addition of new tasks or variations with minimal friction (Markeeva et al., 2024, McLeish et al., 2024).

The driving motivation is the robust and diagnostic testing of LLMs' capacity to perform stepwise, compositional algorithmic reasoning—beyond merely pattern matching or final-answer retrieval—under challenging OOD regimes.

2. Benchmark Structure and Task Coverage

CLRS-Text consists of thirty algorithmic tasks drawn from Cormen et al.’s Introduction to Algorithms, organized into:

• Sorting & Selection: Bubble Sort, Insertion Sort, Heapsort, Quicksort, Quickselect
• Searching: Binary Search
• Graph Algorithms: BFS, DFS, Dijkstra, Bellman–Ford, DAG-SP, Floyd–Warshall, Articulation Points, Bridges, SCC, Kruskal’s MST, Prim’s MST, Topological Sort
• Dynamic Programming and Divide-and-Conquer: Matrix Chain Multiplication, Maximum Subarray, Optimal Binary Search Tree, Longest Common Subsequence
• Greedy Algorithms: Activity Selector, Task Scheduling
• String Algorithms: Naive String Matcher, Knuth–Morris–Pratt (KMP)
• Geometry: Graham Scan, Jarvis’s March, Segment Intersection

Each task is procedurally generated: inputs are sampled from distributions appropriate to the algorithm’s semantics (e.g., Erdős–Rényi graphs for graph tasks, random integer sequences for sorting, etc.) (Markeeva et al., 2024, Veličković et al., 2022). For every instance, the full algorithm is executed to completion, and the value trajectory for a specified key internal variable (e.g., the array in sorting, predecessor array in shortest paths, DP table in dynamic programming) is exported as the “trace” for textual prediction. The text format is regularized across tasks, featuring a header, input serialization, initial-trace state, tokenized trace, and final answer (Markeeva et al., 2024).

3. Dataset Composition and Instance Specification

Inputs and outputs are standardized as follows:

• Input Specification: Each prompt includes the algorithm name and a sequence of key-value pairs representing inputs (array values, adjacency matrices, pattern strings, etc.), together with an initial value for the tracked variable (e.g., initial array, all-zero predecessor list).
• Trace and Target: The textual “trace” prints the variable’s full value at each atomic update, forming a sequence for chain-of-thought evaluation. Only one variable per task is traced in the CLRS-Text format.
• Final Answer: The trace culminates in a final output (e.g., sorted array, minimum cost matrix, list of matching indices).
• Tokenization: No special tokens are required; prompts are compatible with byte-pair encoding and other standard tokenizers, and are confined to fit the LM context window (controlled by maximum problem size, e.g., n ≤ 64 for Gemma) (Markeeva et al., 2024).

Sample sizes for training, validation, and test splits are programmable. Tasks are configured for both in-distribution (training size) and OOD regimes (longer sequences or larger graphs) (Markeeva et al., 2024, McLeish et al., 2024).

4. Evaluation Protocols and Metrics

CLRS-Text supports a suite of granular evaluation metrics:

• Final-Answer Accuracy: For each prompt, the extracted model output is compared to the ground-truth final variable. This forms the principal accuracy metric.
• Trace-Level Accuracy: Models may be evaluated on the exact match of the full predicted trace to ground truth, thus probing chain-of-thought fidelity.
• Stepwise Accuracy: The proportion of correctly predicted variable values at each intermediate step (where applicable).
• F₁ Score: For token-level outputs (e.g., tables, sequences), micro-F₁ is computed by aggregating true/false positives/negatives across tokens (Veličković et al., 2022, McLeish et al., 2024).
• Additional Failure Categorization: Error types such as formatting mistakes, API/runtime errors, or explicit LLM uncertainty (e.g., asking clarifying questions) are also tracked (McLeish et al., 2024).

Testing protocols include zero-shot (fresh instances, no in-context training), two-shot (with two examples in the prompt), and multi-round resampling to ensure robust estimation free of static dataset artifacts. OOD generalization is specifically measured as the accuracy drop or F₁ gap on instance sizes above the training maximum (Markeeva et al., 2024, Bevilacqua et al., 2023).

5. Comparative Model Performance and Analysis

CLRS-Text serves as a common ground for cross-architecture benchmarking:

• GNN/NAR Baselines: Standard models include MPNN, PGN, DeepSets, GAT, Memory Networks, and advanced approaches such as Triplet-GMPNN. On in-distribution tasks, micro-F₁ routinely exceeds 90% for most tasks, but performance typically collapses on OOD sizes, especially for recursion-intensive or string-based algorithms (some as low as ≈2–3% for string matching) (Veličković et al., 2022).
• Augmented Neural Reasoners: Causal regularization and open-book memory integration can multiply OOD F₁ scores on selected tasks (e.g., up to 3× for sorting), indicating the value of leveraging instance invariances and auxiliary data (Bevilacqua et al., 2023, Li et al., 2024).
• Transformer LMs: Baseline LMs (e.g., Gemma 2B) exhibit strong in-distribution accuracy but rapid OOD degradation (often <20% at double the training size), even with advanced positional encoding. The requirement to emit every chain-of-thought token compounds error propagation and context limitations (Markeeva et al., 2024).
• Hybrid LM–NAR Architectures: Models such as TransNAR integrate NAR cross-attention into Transformer pipelines, yielding substantial OOD gains (e.g., size-14 extrapolation CLRS score rise from ≈0.22 to ≈0.58) and enhanced output shape reliability, although certain index-finding tasks remain bottlenecks (Bounsi et al., 2024, Markeeva et al., 2024).
• LLM with Code Interpreter: ChatGPT (GPT-4) evaluated on CLRS-Text demonstrates competitive or superior performance to specialist GNNs on two-thirds of the tasks (test-split F₁ ≈ 88%), especially for sorting, searching, and graph problems. Limitations are observed in large-scale dynamic programming, primarily due to the complexity of intermediate table management and potential reliance on retrieval instead of genuine OOD generalization (McLeish et al., 2024).

6. Technical Challenges and Research Directions

Persistent technical obstacles identified via CLRS-Text include:

• Autoregressive Bottlenecks: LMs trained for next-token prediction struggle to consistently update long, high-dimensional state vectors, hindering compositional reasoning and propagation of pointer-dependent values (Markeeva et al., 2024).
• Pointer and Index Management: Maintaining index coherence over long traces, especially in pointer-heavy algorithms (e.g., Bellman–Ford, DP tables), frequently degrades with increasing instance size.
• OOD Generalization: Combinatorial and recursive task extrapolation remains a key unsolved problem, with existing inductive biases insufficient for robust transfer (Veličković et al., 2022, Bevilacqua et al., 2023).
• Architectural and Methodological Innovations: Promising extensions involve non-autoregressive or blockwise decoding for entire state vectors, hybrid GNN-LM systems, and explicit memory or tool-use augmentation (e.g., code interpreter, open-book memory, cross-task attention) (Markeeva et al., 2024, Bounsi et al., 2024, Li et al., 2024).
• Evaluation Metrics Expansion: Richer partial-credit and step-level scoring, trace-level diagnostics, and visualization of intra/task cross-attentions are under active development.

Planned directions include expanding the task suite, integrating more diverse algorithms, devising compositional or meta-learning schemes, and using the benchmark for algorithm discovery as well as emulation (Markeeva et al., 2024, McLeish et al., 2024).

7. Significance and Broader Implications

CLRS-Text provides a translational bridge between the algorithmic reasoning paradigms of graph-based NARs and natural-language LMs. Its significance is multi-fold:

• Unified and Extensible Testing Ground: By standardizing task generation and format while remaining extensible, CLRS-Text sets a baseline for multi-architecture, multi-modal evaluation.
• Reproducibility and Procedural Generation: Tasks are generated on-demand, circumventing static training/test artifacts and facilitating robust OOD diagnostics.
• Accelerator for LM Research: It highlights the unique challenges of algorithmic reasoning in LMs—distinct from conventional language tasks—while promoting direct comparison with specialist architectures (Markeeva et al., 2024).
• Tool for Algorithmic Neuroscience: Cross-task attention matrices have revealed previously uncharted curriculum and dependency graphs across algorithmic families, offering insight into the neural learning of algorithmic abstractions (Li et al., 2024).

A plausible implication is that advances on CLRS-Text, especially in hybrid or memory-augmented LMs, may directly accelerate progress in interpretable, generalizable machine reasoning architectures. Ongoing limitations in dynamic-programming and recursive tasks underscore the open frontier in neural algorithmic reasoning and the continued value of benchmark-driven inquiry.