---
title: 'NPHardEval: Benchmark for LLM Algorithmic Reasoning'
url: https://www.emergentmind.com/topics/nphardeval
type: topic
---

# NPHardEval: Benchmark for LLM Algorithmic Reasoning

Searching arXiv for NPHardEval and closely related benchmark papers.
NPHardEval is a dynamic benchmark for evaluating the reasoning abilities of large language models through the lens of computational complexity classes. Introduced as a benchmark of 900 algorithmic questions extending up to the NP-Hard complexity class, it organizes reasoning assessment around classic problems in \(P\), NP-complete, and NP-hard, with monthly datapoint refreshes intended to reduce overfitting to a static public test set. Its design emphasizes algorithmically checkable answers, automated evaluation, and difficulty scaling, and it was presented as the first LLM reasoning benchmark structured around established computational complexity classes [2312.14890].

## 1. Conceptual basis and motivation

NPHardEval was proposed in response to two limitations attributed to existing reasoning benchmarks for large language models. First, existing benchmarks were described as inadequate for offering a rigorous evaluation of the full extent of reasoning abilities that LLMs are capable of achieving. Second, publicly accessible static benchmarks were described as vulnerable to overfitting, because models can potentially tailor their responses to benchmark metrics and thereby inflate reported performance. NPHardEval addresses these concerns by grounding evaluation in computational complexity classes and by adopting a dynamic update mechanism that refreshes datapoints monthly [2312.14890].

The benchmark is explicitly centered on algorithmic reasoning rather than broad, loosely specified problem solving. Its problem set excludes pure numerical computation in order to focus on logical reasoning. This framing places NPHardEval closer to theoretical computer science than to exam-style or commonsense-oriented benchmark design. A plausible implication is that its scores are best interpreted as measurements of structured algorithmic reasoning under controlled verification conditions, not as a general proxy for all forms of language-model intelligence.

The benchmark’s complexity-theoretic framing is also meant to make inter-model comparisons more principled. By comparing model behavior across problems drawn from \(P\), NP-complete, and NP-hard, NPHardEval attempts to provide what the paper characterizes as an objective and rigorous perspective on current LLM reasoning.

## 2. Complexity classes and task composition

NPHardEval divides its problem suite into three classes. The paper presents \(P\) as the class consisting of tasks that can be solved by a deterministic Turing machine in polynomial time; NP-complete as problems that are in NP and as hard as any problem in NP; and NP-hard as problems that are at least as hard as the hardest problems in NP and may not have solutions verifiable in polynomial time. The benchmark selects non-overlapping tasks for each class in order to assess class boundaries [2312.14890].

The task inventory is organized around nine classic problems, three per class.

| Complexity class | Problems | Role in benchmark |
|---|---|---|
| \(P\) | Sorted Array Search (SAS), Edit Distance Problem (EDP), Shortest Path Problem (SPP) | Tractable algorithmic reasoning |
| NP-complete | Traveling Salesman Problem Decision Version (TSP-D), Graph Coloring Problem (GCP-D), Knapsack Problem (KSP) | Polynomial-time verifiable hard decision problems |
| NP-hard | Traveling Salesman Optimization (TSP), Graph Coloring Optimization (GCP), Meeting Scheduling Problem (MSP) | Optimization problems beyond NP-complete decision form |

Across these nine problems, NPHardEval contains 900 algorithmic questions. Each problem is synthesized with 10 escalating levels of difficulty. Difficulty scaling is implemented by increasing structural parameters of the generated instances: graph data increase nodes, edges, and weights, while linear data increase array length and value range. All data are saved for consistency and reproducibility, and prompts require step-by-step reasoning with answers in a specified format for automatic parsing and validation [2312.14890].

The benchmark also includes formal problem statements for representative tasks. For example, the Edit Distance Problem is defined as follows: given two strings \(A\) and \(B\), find a minimum number \(d\) such that \(A\) can be transformed into \(B\) via \(d\) insertions, deletions, and substitutions, with time and space complexity \(\mathcal{O}(ab)\) where \(a,b\) are string lengths. For the Traveling Salesman Problem, the decision version is written as
$$
\sum_{i=1}^{n-1} d(P(i),P(i+1)) + d(P(n),P(1)) \leq D?
$$
and the optimization version as
$$
\min_{P} d(P(n),P(1)) + \sum_{i=1}^{n-1} d(P(i),P(i+1)).
$$

## 3. Data generation and evaluation methodology

A defining feature of NPHardEval is that its questions are automatically generated and automatically checked. The benchmark uses algorithmically checkable answers and requires no human annotation. This choice makes the benchmark scalable and directly compatible with repeated regeneration of the test set [2312.14890].

Its main quantitative metrics are Weighted Accuracy and Failure Rate. Weighted Accuracy is intended to assign higher weights to correct answers at higher difficulty levels, thereby quantifying what the paper terms true reasoning depth:
$$
WA = \frac{\sum_{i=1}^{10} (w_i \times A_i)}{\sum_{i=1}^{10} w_i}
$$
where \(w_i\) is the difficulty weight and \(A_i\) is the accuracy at level \(i\).

Failure Rate measures the proportion of outputs that do not conform to the required format, including unparsable or failed submissions:
$$
FR = \frac{\sum_{i=1}^{10} F_i}{100}
$$
where \(F_i\) is the number of failed attempts at difficulty level \(i\).

These metrics are integral to the benchmark’s attempt to distinguish reasoning failures from output-format failures. The answer-format requirement is not incidental: solutions must be produced in a parsable form so that correctness can be verified automatically. This design links benchmark validity to machine-verifiable ground truth rather than to human judgement. A plausible implication is that NPHardEval privileges problems for which both instance generation and answer checking can be formalized precisely.

## 4. Empirical results on language-model reasoning

The empirical study reported in NPHardEval evaluates both closed-source and open-source LLMs. The paper reports a common trend across models: weighted accuracy decreases and failure rates increase as complexity rises from \(P\) to NP-complete to NP-hard. At the NP-hard level, weighted accuracy drops close to zero. Closed-source models such as GPT-4 Turbo and Claude 2 outperform open-source models such as Mistral-7b and Yi-34b, while Yi-34b, Qwen-14b, and Mistral-7b are described as notable within the open-source category [2312.14890].

The paper also reports statistical significance tests. Wilcoxon tests show significant differences with \(p<0.05\), especially between \(P\)/NP-Complete and NP-Hard. Performance is not uniform even within a single complexity class: models often perform better on Sorted Array Search than on Edit Distance or Shortest Path within \(P\), and some models exhibit strengths on specific tasks or classes.

These findings support the paper’s central claim that current LLM reasoning degrades sharply on harder algorithmic tasks. At the same time, the within-class variability indicates that complexity class alone does not determine difficulty at the level of individual benchmark items. This suggests that NPHardEval functions both as a complexity-based benchmark and as a task-specific probe of algorithmic competence.

## 5. Dynamic updating, overfitting resistance, and in-context learning

NPHardEval’s dynamic update mechanism is central to its benchmark philosophy. The entire test set is regenerated monthly, with variable instantiations, graph structures, and parameter sizes refreshed on each update. The stated purpose is to prevent memorization and pretraining on a static public test set and to support an ongoing leaderboard for future models [2312.14890].

The paper supplements this design claim with finetuning experiments. Open-source models including Phi-2, Mistral-7b, and Qwen-14b were finetuned on multiple prior versions of the benchmark. Reported gains appeared only on very easy \(P\) problems. For NP-complete and NP-hard problems, the paper reports that no hacking was observed: the finetuned models did not generalize to unseen hard tasks, and performance even degraded on harder variants.

NPHardEval also studies few-shot behavior. Closed-source models are reported to show genuine learning, in the sense that performance remains stable regardless of whether in-context examples are easier or harder than the target question. Open-source models, by contrast, are reported often to generalize from harder-to-easier problems but not the reverse, which the paper characterizes as “mimicking” rather than algorithmic learning.

These results are significant because they tie benchmark dynamics to a concrete empirical question: whether performance reflects abstraction or benchmark-specific adaptation. The reported outcome is that dynamic regeneration appears to limit overfitting on hard tasks, while few-shot transfer behavior exposes substantial differences between model families.

## 6. Extensions, related benchmarks, and place in the literature

NPHardEval has already served as the basis for a multimodal extension. NPHardEval4V directly extends NPHardEval by converting the textual description of its questions into image representations for evaluating Multimodal Large Language Models. It retains the same complexity-class organization and monthly update schedule, but introduces Recognition Accuracy, Instruction-following Effective Rate, and Aggregated Accuracy in order to disentangle image recognition, instruction following, and reasoning [2403.01777].

Subsequent work situates NPHardEval within a broader trend toward complexity-based, dynamically generated, automatically verifiable reasoning benchmarks. NPPC presents an ever-scaling benchmark over 25 NP-complete problems and defines “ever-scalingness” through the properties uncrushable, unhackable, auto-verifiable, and general [2504.11239]. NP-ENGINE, which focuses on training and evaluating LLMs on NP-hard optimization problems, explicitly contrasts itself with previous NP-hard evaluation setups such as NPHardEval and NPPC by claiming scalable generation, automatic verification, heuristic ground truths, and trainability through a generator-verifier-heuristic pipeline [2510.16476].

Within this literature, NPHardEval occupies a specific position. It is not an optimization-training framework, and it is not a multimodal benchmark in its original form. Its distinctive contribution is a dynamic, text-based, complexity-class benchmark for algorithmic reasoning with automatic evaluation and monthly refreshes. This suggests that NPHardEval helped establish a methodological template in which procedurally generated instances, verifier-backed scoring, and explicit computational complexity serve as the backbone of reasoning evaluation.

Source: https://www.emergentmind.com/topics/nphardeval