BeyondBench: Dynamic Reasoning Benchmark

• BeyondBench is a dynamic evaluation framework that generates fresh, mathematically verified algorithmic problems to overcome static benchmark contamination.
• It uses parameterized instance generation, formal checking, and token-aware scaling to reliably assess reasoning across Easy, Medium, and Hard tasks.
• Empirical results show significant performance drops on NP-complete tasks, highlighting the importance of tool augmentation in enhancing model reasoning.

BeyondBench is a contamination-resistant, dynamically generated benchmark for evaluating the reasoning of LLMs through algorithmic problem generation rather than fixed test sets (Srivastava et al., 29 Sep 2025). It is designed to measure a model’s true algorithmic reasoning power rather than its memorization capacity by creating mathematically grounded problems on the fly, verifying solutions deterministically, and scaling instance difficulty to fit model-specific context constraints (Srivastava et al., 29 Sep 2025). In the reported study, the framework covers 44 algorithmic tasks with 117 total variations, organized into Easy, Medium, and Hard suites, and is used to evaluate 101 LLMs spanning open-source and closed-source families (Srivastava et al., 29 Sep 2025).

1. Definition and motivating problem

BeyondBench is introduced in response to a central evaluation problem for contemporary LLMs: static benchmarks available on the internet risk contamination by training data, making it unclear whether strong scores reflect reasoning or answer recall (Srivastava et al., 29 Sep 2025). Its defining move is to replace fixed benchmark items with parameterized generators that instantiate fresh problems at evaluation time. This keeps each test uncontaminated in the intended sense and shifts the target of measurement from benchmark exposure to algorithmic competence (Srivastava et al., 29 Sep 2025).

The framework is organized around three stated pillars: on-the-fly instance generation from enormous parameter spaces, formal verification of unique (or exhaustively enumerated) solutions, and token-budget-aware difficulty scaling (Srivastava et al., 29 Sep 2025). These design choices jointly address three failure modes of static evaluation: item memorization, unverifiable open-ended outputs, and accidental mismatch between task instance size and model context budget.

A potential misconception is that “benchmark-free evaluation” implies the absence of a benchmark structure. BeyondBench instead retains a formal evaluation framework, task taxonomy, and explicit metrics; what it avoids is reliance on a static, internet-exposed collection of benchmark items (Srivastava et al., 29 Sep 2025).

2. Problem generation, verification, and token-aware scaling

For each task $\tau$, BeyondBench defines a generator

$$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$

mapping a finite “size-and-rule” parameter $\theta \in \Theta_\tau$ and random seed $r \in \mathcal{R}$ to a problem instance $p \in \mathcal{P}_\tau$ (Srivastava et al., 29 Sep 2025). The construction is explicitly parameterized so that the instance space is extremely large. The paper gives the example $\Theta_\tau \approx [-10^6,10^6]^n$ for list tasks and states that one can ensure

$|\Theta_\tau \times \mathcal{R}| > 10^{15},$

yielding

$$P(\mathrm{collide}) \le (1-10^{-15})^N \approx \exp(-N \cdot 10^{-15}) < 10^{-3}$$

for a static training corpus $\mathcal{C}$ with $|\mathcal{C}| < 10^{12}$ (Srivastava et al., 29 Sep 2025).

Each generated instance is paired with a solver or CSP/SAT-based checker $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$0 such that either

$$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$1

or all valid solutions $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$2 are exhaustively enumerated (Srivastava et al., 29 Sep 2025). This is the framework’s formal verification layer. The paper states that verification costs remain polynomial, for example $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$3 or better, even when the solution space is exponential (Srivastava et al., 29 Sep 2025). In operational terms, this allows BeyondBench to score model outputs against mathematically defined ground truth rather than heuristic answer matching.

Difficulty is additionally constrained by context length. For a model $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$4 with context budget $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$5, BeyondBench estimates required tokens by

$$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$6

and enforces $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$7 by iteratively reducing size parameters, for example $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$8 (Srivastava et al., 29 Sep 2025). After inference, actual token counts are measured with model-specific tokenizers, specifically tiktoken for GPT models and Hugging Face tokenizers for open-source models, and responses exceeding $$G_\tau : \Theta_\tau \times \mathcal{R} \to \mathcal{P}_\tau$$9 are flagged (Srivastava et al., 29 Sep 2025). This makes token budget a first-class constraint in task generation rather than an after-the-fact artifact of prompting.

3. Task taxonomy and difficulty structure

BeyondBench organizes 44 base tasks into three suites with 117 total variations (Srivastava et al., 29 Sep 2025). The taxonomy is intended to separate relatively routine polynomial-time problems from sequence-based exponential reasoning tasks and from NP-complete or CSP-heavy problems.

Suite	Scope	Representative tasks
Easy Suite	29 tasks; polynomial-time problems $\theta \in \Theta_\tau$0	arithmetic, sorting, max/min, parity, perfect squares, palindromic numbers, mean, median, mode
Medium Suite	5 tasks, 49 variations; exponential $\theta \in \Theta_\tau$1–$\theta \in \Theta_\tau$2 sequence continuation	linear recurrences, geometric/exponential sequences, prime/number-theoretic patterns, composite patterns, algebraic/transcendental nests
Hard Suite	10 tasks, 68 variations; NP-complete and beyond	Tower of Hanoi, N-Queens, Graph Coloring, Boolean SAT, Sudoku, Cryptarithmetic, Matrix Chain Multiplication, Modular Systems, Constraint Optimization, Logic Grid Puzzles

The Easy Suite contains 29 tasks for basic arithmetic and statistics, including sum $\theta \in \Theta_\tau$3, product $\theta \in \Theta_\tau$4, difference, division with tolerance $\theta \in \Theta_\tau$5, sorting, max/min, parity, perfect squares, palindromic numbers, mean, median, and mode (Srivastava et al., 29 Sep 2025). These are described as polynomial-time problems with $\theta \in \Theta_\tau$6 in $\theta \in \Theta_\tau$7 (Srivastava et al., 29 Sep 2025).

The Medium Suite comprises 5 tasks with 49 variations and focuses on sequence continuation whose generation or inference can involve exponential complexity from $\theta \in \Theta_\tau$8 to $\theta \in \Theta_\tau$9 (Srivastava et al., 29 Sep 2025). The listed categories include Fibonacci and general linear recurrences of the form

$r \in \mathcal{R}$0

geometric and exponential sequences

$r \in \mathcal{R}$1

prime and number-theoretic patterns such as sieve, $r \in \mathcal{R}$2, $r \in \mathcal{R}$3, and Mersenne tests, as well as complex composite patterns and algebraic or transcendental nests including $r \in \mathcal{R}$4 and continued fractions (Srivastava et al., 29 Sep 2025).

The Hard Suite comprises 10 tasks with 68 variations and targets NP-complete and related forms of combinatorial reasoning (Srivastava et al., 29 Sep 2025). Its tasks include Tower of Hanoi with $r \in \mathcal{R}$5 moves and PSPACE-complete variants; N-Queens with conflict constraints

$r \in \mathcal{R}$6

Graph Coloring with chromatic number $r \in \mathcal{R}$7 under the $r \in \mathcal{R}$8 bound and Brooks’ theorem; Boolean SAT with formula $r \in \mathcal{R}$9, search space $p \in \mathcal{P}_\tau$0, and phase transition at $p \in \mathcal{P}_\tau$1; Sudoku in 4×4, 6×6, 9×9, irregular, and diagonal forms; Cryptarithmetic with letter-to-digit bijections and carry propagation; Matrix Chain Multiplication with

$p \in \mathcal{P}_\tau$2

in $p \in \mathcal{P}_\tau$3; Modular Systems; Constraint Optimization; and Logic Grid Puzzles as Latin-square-style CSPs with directed and conditional clues (Srivastava et al., 29 Sep 2025).

4. Evaluation protocol, model coverage, and metrics

The reported evaluation spans 101 models, including 85 open-source and 16 closed-source systems, with model sizes from 0.5B to 141B parameters and multiple quantization schemes (Srivastava et al., 29 Sep 2025). For each of the 117 variations, 1,000 distinct instances with seed $p \in \mathcal{P}_\tau$4 were generated for each open-source model, while proprietary models were evaluated on 100 instances per variation to reduce API cost (Srivastava et al., 29 Sep 2025).

The open-source families listed in the paper include Qwen3 (0.6B–32B, MoE, thinking/math variants), Qwen2.5 (0.5B–72B, math-fine-tuned), Llama-3.x (1B–70B), Phi3/Phi4 (3.8B–14B), Gemma (1–27B), Mistral (7B–176B MoE), and Smollm2/3 (Srivastava et al., 29 Sep 2025). The closed-source families include OpenAI’s GPT-4/4o and GPT-5 in full, mini, and nano variants, as well as Gemini-2.0 and Gemini-2.5 in pro, flash, and lite variants (Srivastava et al., 29 Sep 2025). The paper further reports that FP8 and GPTQ 8-bit/4-bit variants exhibited less than 3% average accuracy drop (Srivastava et al., 29 Sep 2025).

BeyondBench defines three metrics for each suite and task. Accuracy is the percentage of instances for which

$p \in \mathcal{P}_\tau$5

Instruction Following is the percentage of outputs matching the prescribed \boxed{…} format. Token Usage is mean output length (Srivastava et al., 29 Sep 2025). These metrics combine semantic correctness, formatting compliance, and generation length.

Tool usage is treated as an explicit evaluation variable. Proprietary models were evaluated both with their full toolchain and in a language-only setting (Srivastava et al., 29 Sep 2025). This distinction becomes central to the interpretation of hard-suite performance because the paper reports substantial differences between tool-augmented and language-only inference.

5. Empirical results and observed failure modes

The principal empirical pattern reported by BeyondBench is a sharp degradation in performance as task complexity increases from polynomial to exponential (Srivastava et al., 29 Sep 2025). The paper states that even 120B-parameter open-source models achieve greater than 90% on the Easy Suite but fall to 20–40% on the Hard Suite (Srivastava et al., 29 Sep 2025). This is presented as evidence of consistent reasoning deficiencies across model families rather than isolated weaknesses of a single architecture.

The paper gives several concrete examples. Qwen-OSS-120B achieves Easy 93.5%, Medium 75.1%, Hard 59.6%; Llama-3.3-70B achieves Easy 74.6%, Medium 46.7%, Hard 26.9%; and Gemini-2.5-pro achieves Easy 89.2%, Medium 77.2%, Hard 56.4% (Srivastava et al., 29 Sep 2025). In the abstract, the corresponding hard-suite averages are given as 56.38% for Gemini-2.5-pro, 26.91% for Llama-3.3-70B, and 33.60% for Qwen2.5-72B (Srivastava et al., 29 Sep 2025). These figures show that models that are strong on arithmetic and structured sequence tasks do not maintain comparable performance on NP-complete and CSP-style tasks.

The paper also identifies a steep hard-suite cliff on specific problems, especially Sudoku and N-Queens (Srivastava et al., 29 Sep 2025). It states that models often exceed 80% on 4×4 Sudoku but fall below 20% on 9×9 (Srivastava et al., 29 Sep 2025). This suggests that BeyondBench is not only measuring whether a model can solve a problem family at all, but whether it preserves competence under scaling within that family.

Scaling with parameter count exhibits diminishing returns. The paper gives the example that increasing Qwen3 from 0.6B to 1.7B adds approximately 17 percentage points on Hard tasks, whereas increasing from 14B to 32B adds less than 2 percentage points (Srivastava et al., 29 Sep 2025). A plausible implication is that BeyondBench exposes a saturation regime in which larger parameter count alone does not yield proportional gains on combinatorial reasoning.

6. Tool use, reproducibility, and relation to benchmark science

Tool augmentation materially changes outcomes on hard tasks. For the GPT-5 family, the paper reports that use of tools improves hard-suite results by 6–12 points and gives a concrete comparison: GPT-5 with tools scores 97.3% / 81.6% / 71.8% and 83.6% overall, whereas GPT-5 with no tools scores 88.4% / 61.6% / 50.3% and 66.8% overall (Srivastava et al., 29 Sep 2025). The hard-suite drop is stated as $p \in \mathcal{P}_\tau$6 for GPT-5, 15.9% for GPT-5-mini, and 47.6% for GPT-5-nano; the abstract gives the corresponding declines as 16.81%, 28.05%, and 47.59% on the hard suite for GPT-5, GPT-5-mini, and GPT-5-nano (Srivastava et al., 29 Sep 2025). The exact percentages differ slightly between the abstract and detailed summary, but both passages agree that language-only performance drops drastically without tool usage.

All problems, code, and up-to-date leaderboards are publicly available at https://ctrl-gaurav.github.io/BeyondBench/, and the framework provides an API through which new models can generate fresh instances and compare against the 117-variation, 44-task suite without risk of contamination (Srivastava et al., 29 Sep 2025). In this sense, reproducibility is tied not to a frozen item set but to a reproducible generation-and-verification process.

BeyondBench can also be situated in a broader line of work on benchmark methodology. “A BenchCouncil View on Benchmarking Emerging and Future Computing” formalizes a benchmark as

$p \in \mathcal{P}_\tau$7

where $p \in \mathcal{P}_\tau$8 is a problem definition, $p \in \mathcal{P}_\tau$9 a solution instantiation, $\Theta_\tau \approx [-10^6,10^6]^n$0 a measurement function, and $\Theta_\tau \approx [-10^6,10^6]^n$1 and $\Theta_\tau \approx [-10^6,10^6]^n$2 the measurement standard and tool (Zhan, 2022). That paper argues that benchmark properties are extrinsic rather than inherent and emphasizes the entanglement of definition, instantiation, and measurement, together with risks of technology inertia and instantiation bias (Zhan, 2022). BeyondBench’s emphasis on parameterized generation, formal checking, traceable measurement, and publicly available tooling is consistent with that broader benchmark-science perspective. This suggests that BeyondBench is not only a reasoning evaluation suite but also an example of a more general shift away from static benchmark artifacts toward dynamically instantiated, formally verifiable evaluation regimes (Zhan, 2022).