ASP-Bench: NL to ASP Benchmark
- ASP-Bench is a benchmark suite that converts natural language descriptions into executable ASP models using semantic validators.
- It comprises 128 instances from 64 base problems with both easy and hard variants to thoroughly assess modeling challenges.
- The benchmark employs validator-based semantic verification and expressive ASP constructs to capture diverse reasoning aspects.
Asper-Bench commonly refers to ASP-Bench, a benchmark suite for translating natural language problem descriptions into Answer Set Programs (ASPs) and solving them with an ASP solver (Szeider, 1 Feb 2026). The benchmark comprises 128 benchmark instances built from 64 base problems, each with an easy and a hard variant, and evaluates whether a system can move from informal specification to a correct executable logical model rather than merely reproduce a reference encoding (Szeider, 1 Feb 2026). A source of terminological ambiguity is the earlier ASPER work, where no benchmark is explicitly named “Asper-Bench,” although the paper’s Sudoku-based evaluation setup can function as a reusable low-data neural-symbolic benchmark defined by the ASPER/ASPEN framework (Machot, 2023). In contemporary usage, however, the explicit benchmark designation is ASP-Bench.
1. Name, scope, and problem setting
ASP-Bench is designed to evaluate systems that automatically construct ASP encodings from natural language and produce correct solutions, not just syntactically similar programs (Szeider, 1 Feb 2026). Each instance consists of a natural language problem description, a precise JSON output format specification, and a reference validator implemented as a Python script that checks whether a candidate solution satisfies the intended constraints and, when relevant, is optimal (Szeider, 1 Feb 2026). The benchmark therefore targets the neurosymbolic problem of converting an informal textual specification into a formal declarative program that an automated solver can execute.
The benchmark addresses several limitations identified in earlier NL→ASP evaluation settings. Evaluation based on exact string matching against a single reference ASP file rejects alternative but semantically equivalent encodings and cannot handle problems with many valid solutions when only one reference solution is known (Szeider, 1 Feb 2026). Problem sets were also described as opportunistic rather than systematically constructed to probe ASP’s expressive features, and many neurosymbolic systems assumed the ASP program was already given rather than assessing the modeling step itself (Szeider, 1 Feb 2026). ASP-Bench responds by using semantic verification through validators, systematically covering core ASP language constructs and reasoning styles, and providing a platform for studying modeling hardness (Szeider, 1 Feb 2026).
A recurring misconception is to treat ASP-Bench as a dataset of reference programs. Its unit of evaluation is instead the validator-accepted solution produced by an NL→ASP→solver pipeline. This means that semantically correct alternative encodings are admissible, provided they generate a solution that passes the validator (Szeider, 1 Feb 2026).
2. Benchmark composition and representational design
ASP-Bench defines 64 distinct problem templates with two variants each: an easy variant and a hard variant, yielding 128 instances in total (Szeider, 1 Feb 2026). Easy variants involve smaller input size and/or simpler constraints, while hard variants introduce larger instance sizes, additional constraint types, multi-objective optimization, or more intricate reasoning patterns (Szeider, 1 Feb 2026). Hard problems are verified during design to remain solvable within 20 seconds of solver time, and are often constructed with planted solutions, ensuring solvability and reasonable runtime (Szeider, 1 Feb 2026).
The problem set spans logic puzzles, graph problems, scheduling, optimization, spatial puzzles, and several domain-specific settings. Examples listed in the benchmark description include “Who Is Killer,” “Knights and Knaves,” “Zebra Puzzle,” “Historical Counterfactual,” as well as weighted graph coloring, Hamiltonian path, vertex cover, maximum independent set, Steiner tree, graph partition, network flow, job shop scheduling, nurse rostering, course timetabling, exam scheduling, meeting scheduling, car sequencing, Nonogram solver, queens domination, crossword, ricochet robots, dungeon generation, DNA sequence assembly, protein structure simplification, quantum circuits, Metroidvania level generation, ecosystem balance, and strategic voting (Szeider, 1 Feb 2026).
ASP is the target formalism because it is a declarative logic programming paradigm based on the stable model semantics, with rules and constraints whose models are computed as answer sets (Szeider, 1 Feb 2026). The benchmark description emphasizes expressive constructs that map naturally to many reasoning patterns: choice rules, aggregates, optimization statements, and integrity constraints, together with recursion, temporal patterns, and spatial/grid modeling (Szeider, 1 Feb 2026). Programs are assumed to be executable by clingo, and the benchmark is built around clingo-compatible syntax and semantics (Szeider, 1 Feb 2026).
The basic ASP rule schema given in the benchmark is:
This formal framing is not itself the benchmark objective; rather, it defines the representational substrate that systems must synthesize from natural language (Szeider, 1 Feb 2026).
3. Coverage of reasoning aspects and semantic validation
A central design feature of ASP-Bench is its systematic coverage of ASP language features and reasoning modes. The benchmark classifies each hard variant along seven reasoning aspects, which are described as largely independent (Szeider, 1 Feb 2026).
| Aspect | Description | Typical manifestation |
|---|---|---|
| OPT | Optimization | #minimize / #maximize or equivalent optimization constructs |
| TEMP | Temporal or sequential reasoning | time steps, ordering of events, or action sequences |
| DEFAULT | Default logic / soft constraints / preferences | weak constraints, optimization with penalty weights, defaults and exceptions |
| RESOURCE | Resource allocation with capacity limits | capacity constraints using aggregates such as #sum |
| RECURSIVE | Recursion, cycles, fixed-point reasoning | reachability, transitive closure, cycle detection |
| SPATIAL | Grid-based or neighborhood logic | predicates like cell(X,Y) and neighbor relations |
| QUANT | Quantitative complexity | seven or more distinct constraints |
These aspects are used to analyze what makes natural-language-to-ASP modeling difficult, but the paper explicitly reports that hardness does not strongly correlate with the count of aspects in a problem (Szeider, 1 Feb 2026). Some single-aspect problems, such as Nonogram Solver (SPATIAL only), are among the hardest (Szeider, 1 Feb 2026). This suggests that benchmark difficulty is driven less by the nominal number of categories and more by the structure of the constraints and the subtlety of the required encoding.
Every instance includes a reference validator. The validator reads a candidate JSON solution from standard input, checks structural correctness, verifies logical constraints, and, for optimization problems, checks that the solution attains the known optimum (Szeider, 1 Feb 2026). This architecture is central to the benchmark’s semantic orientation: correctness is established at the level of satisfaction and optimality rather than textual similarity to a reference ASP program (Szeider, 1 Feb 2026).
This validator-based design also addresses a common evaluation controversy in program synthesis benchmarks. Exact-match scoring implicitly assumes a unique canonical encoding, whereas ASP-Bench accepts any encoding that induces a valid solution. A plausible implication is that the benchmark is better aligned with the declarative nature of ASP, where distinct encodings may be semantically equivalent even when syntactically dissimilar.
4. Agentic methodology and execution model
The principal evaluation in ASP-Bench uses an ASP-Agent built by adapting a general-purpose coding agent to the ASP domain (Szeider, 1 Feb 2026). The agent follows the ReAct (Reason and Act) framework, operates through a persistent IPython kernel, and primarily uses two tools: python_exec, which executes arbitrary Python code in the kernel, and save_code, used once at the end to store the final solution file or files (Szeider, 1 Feb 2026).
ASP-specific support is provided through a project prompt, clingo.md, which specifies the mission of writing Python+ASP code that solves the natural-language problem and outputs JSON as required (Szeider, 1 Feb 2026). The prompt includes ASP syntax and semantics, modeling patterns, anti-patterns, and a required five-step workflow: Plan; Analyze & model; Implement; Solve & extract; Format & verify (Szeider, 1 Feb 2026). The benchmark description further notes that the agent has no direct access to the validator and must instead rely on solver feedback and its own reasoning (Szeider, 1 Feb 2026).
The operational loop is iterative. The agent reads the problem description and JSON schema, reasons about entities and constraints, writes Python+ASP code, executes it, inspects syntax errors, safety errors, unsatisfiable outcomes, or partial outputs, and refines the model until it obtains a solution it deems correct (Szeider, 1 Feb 2026). The paper emphasizes that one-shot generation of ASP often fails because of variable safety requirements, aggregate syntax, and complex constraint interactions; feedback-driven iterative refinement is therefore a core methodological assumption (Szeider, 1 Feb 2026).
A detailed case study in the benchmark description concerns Tower of Hanoi, where an “expert” run required 6 python_exec calls, while a “debugging” run required 20 calls and involved solving smaller instances first before rescaling to the full task (Szeider, 1 Feb 2026). This is presented as evidence that the agent can adopt sophisticated problem-solving strategies, including decomposition, targeted testing, and iterative constraint repair.
5. Empirical performance and hardness analysis
The main ASP-Bench experiment uses Claude Sonnet 4.5 with temperature 0, max tokens 16384, and timeouts of 600 s for easy problems and 1200 s for hard problems; each clingo call is limited to 20 s (Szeider, 1 Feb 2026). Under this configuration, the benchmark reports full saturation: 64 easy problems × 2 runs each → 128/128 successes, and 64 hard problems × 3 runs each → 192/192 successes (Szeider, 1 Feb 2026).
For the 64 hard variants, the benchmark introduces a hardness proxy based on the average number of python_exec calls, interpreted as reasoning–action cycles (Szeider, 1 Feb 2026). The average is approximately 7.7 calls, with the easiest hard problem at 3.7 calls and the hardest at 26 calls, specifically DNA sequence assembly (Szeider, 1 Feb 2026). The paper also records total solving time, token usage, and, in comparative settings, tool call counts (Szeider, 1 Feb 2026).
The hardest hard problems by average execution count include Problem 47: DNA Sequence Assembly (OPT, TEMP, DEFAULT, RESOURCE) – 26.0 executions, Problem 46: Metroidvania Generation – 21.0, Problem 01: Who Is Killer – 20.0, Problem 09: Nonogram Solver (SPATIAL) – 17.2, and Problem 61: Historical Counterfactual – 13.7 (Szeider, 1 Feb 2026). The easiest hard problems include Problem 27: Queens Domination (SPATIAL + OPT) – 3.7, Problem 12: Zebra Puzzle – 3.7, Problem 31: Network Flow – 4.0, and Problem 58: Exam Scheduling – 4.0 (Szeider, 1 Feb 2026).
The benchmark also categorizes each python_exec call into eight activity types: Data Setup, Initial Model, Rule Addition, Logic Verification, Error Correction, Full Solve attempts, Logic Refinement, and Solution Formatting (Szeider, 1 Feb 2026). For harder problems, Logic Verification grows in proportion, and Error Correction remains under 10% of activity even for hard cases (Szeider, 1 Feb 2026). The average time per execution decreases from 22 s for easier problems to 15 s for the hardest, while the input/output token ratio increases from 24:1 to 45:1 (Szeider, 1 Feb 2026). The paper interprets this as evidence that later stages involve more focused, shorter responses combined with heavier contextual conditioning (Szeider, 1 Feb 2026).
A further comparison examines two Model Context Protocol (MCP) configurations on the 10 hardest ASP-Bench problems. ipython-mcp achieves 30/30 successful runs, with 16.0 average tool calls and 216 s average time, while mcp-solver-asp achieves 27/30 successful runs, with 47.2 average tool calls and 297 s average time (Szeider, 1 Feb 2026). The benchmark’s interpretation is that full Python control appears more efficient and robust, whereas the item-wise symbolic interface is safer and more transparent but requires more calls and yields slightly lower success on the hardest problems (Szeider, 1 Feb 2026).
6. Relation to ASPER, benchmark ambiguity, and research significance
The term “Asper-Bench” can be confused with the earlier ASPER work, formally titled “Bridging Logic and Learning: A Neural-Symbolic Approach for Enhanced Reasoning in Neural Models (ASPER)” (Machot, 2023). That paper introduces a deliberately simple neural-symbolic method that uses an ASP solver to structure both the data and the loss for training a small neural network on reasoning-heavy tasks, instantiated on 9×9 Sudoku (Machot, 2023). The evaluation uses extremely small training sets such as 12 puzzles, difficulty levels 0.1, 0.3, 0.6, 0.8, 3-fold cross-validation, and loss variants Standard only, Standard + Expert, Standard + Constraints, and All Combined (Machot, 2023). The paper does not explicitly name a benchmark “Asper-Bench,” but the description states that the Sudoku-based evaluation setup can naturally function as such (Machot, 2023).
This distinction matters because the two works target different problem formulations. ASPER evaluates whether a shallow neural network trained with ASP-derived constraint and expert losses can learn reasoning under hard logical constraints from very few examples (Machot, 2023). ASP-Bench, by contrast, evaluates whether a system can translate natural language specifications into ASP programs and solve them semantically correctly (Szeider, 1 Feb 2026). The former is a neural-symbolic learning setup; the latter is a benchmark for NL→logic program synthesis.
The relationship is nevertheless conceptually close. Both rely on Answer Set Programming as an explicit symbolic substrate, and both foreground settings where symbolic structure is essential rather than incidental. ASPER uses ASP to generate or verify structured data and define loss terms for learning (Machot, 2023), while ASP-Bench uses ASP as the representational target of natural-language formalization (Szeider, 1 Feb 2026). This suggests a broader neurosymbolic trajectory in which ASP functions both as a source of formal constraints for learning and as a target language for automated modeling.
ASP-Bench’s broader significance lies in its validator-based semantics, its multidimensional characterization of reasoning difficulty, and its demonstration that a feedback-driven agentic approach can fully automate NL→ASP translation across a broad range of problems under the reported conditions (Szeider, 1 Feb 2026). Its limitations are also explicit: the benchmark currently contains 64 base problems, is built around clingo, uses relatively precise technical English descriptions, and does not directly assess the elegance or efficiency of the generated ASP encodings beyond correctness and optimality (Szeider, 1 Feb 2026). Future directions identified in the benchmark description include reducing reasoning–action cycles and token usage, evaluating alternative architectures and models, designing “very hard” problems, and extending evaluation toward runtime-oriented encoding quality metrics (Szeider, 1 Feb 2026).
