ASPBench: Evaluating ASP Problem Solving

• ASPBench is a benchmark that evaluates answer set programming by combining synthetic and real-world programs for advanced ASP reasoning.
• It employs both symbolic and textual representations to assess three specific tasks: entailment, answer set verification, and answer set computation.
• Empirical findings highlight that while LLMs perform well on entailment and verification, they struggle with complete stable-model construction.

ASPBench is a benchmark for evaluating how well LLMs can solve Answer Set Programming (ASP) problems. It was introduced to address a gap in prior evaluations, which were described as relying on overly simplified programs, limited operator coverage, and predominantly single-answer-set settings. ASPBench combines synthetic and real-world ASP programs, provides both symbolic and textual representations, and evaluates three ASP-specific tasks—ASP entailment, answer set verification, and answer set computation—thereby targeting increasingly difficult forms of ASP reasoning rather than generic logical inference alone (Ren et al., 26 Jul 2025).

1. Scope, motivation, and positioning

ASPBench was designed around the claim that earlier ASP-related LLM evaluations were too narrow. The benchmark paper states that prior work often used overly simple logic programs, usually did not support strong negation, default negation, disjunction, or multiple answer sets, and lacked tasks specifically designed around ASP semantics. ASPBench therefore targets a broader fragment of ASP and frames evaluation around stable-model reasoning rather than surface logical consistency alone (Ren et al., 26 Jul 2025).

A defining characteristic of the benchmark is that it is tailored to ASP solving as such. The paper contrasts it with datasets such as $\delta$-NLI, ProofWriter, ruletaker, LogicNMR, generics-exemplars, and LogicBench, arguing that these resources are limited in one or more of the following dimensions: they are often textual only, commonly assume a single answer set, do not support the full ASP feature set, or do not evaluate all three ASP-specific tasks. ASPBench is presented as differing by supporting any predicate arity, symbolic and textual representations, strong negation, default negation, disjunction, constraints, multiple answer sets, and dedicated evaluation for ASP entailment (ASE), answer set verification (ASV), and answer set computation (ASC).

This positioning is significant because ASP introduces semantic commitments that are not captured by ordinary textual entailment benchmarks. In particular, stable-model computation requires handling negation-as-failure, model minimality, and potentially multiple answer sets. ASPBench treats those properties as first-class evaluation targets rather than incidental complications.

2. Construction pipeline and representational design

ASPBench is built from both synthetic ASP programs generated automatically through a graph-based pipeline and real-world ASP programs collected from public sources and expanded for diversity. The synthetic construction process has three stages: ASP Graph Construction, ASP Rule Generation, and ASPBench Construction (Ren et al., 26 Jul 2025).

The benchmark’s core formal object is an ASP Graph, defined as a directed graph with rule nodes and predicate nodes. Edge types encode operations and are labeled P, SN, DN, and SN+DN, corresponding to positive, strong negation, default negation, and the combination of strong and default negation. The graph is built in layers. A rule graph is first created as a DAG with a single sink node; predicate nodes are then inserted between rules to form a rule-predicate graph; finally, operation types are assigned probabilistically to obtain a rule-predicate-operation graph.

This graph is converted into concrete ASP rules by assigning variables and arities, ensuring safety, repairing invalid rules via negation flips, and validating syntax and solvability with DLV2. The resulting symbolic programs are then textualized through templates and an LLM-based proofreading step using gpt-4o-mini. The benchmark therefore exposes models to two parallel input regimes: a symbolic form close to executable ASP syntax and a textual form intended to probe whether linguistic phrasing changes reasoning performance.

The synthetic programs are complemented by 47 classic ASP problems, grouped into four categories: Basic Logic Reasoning (BLR), Combinatorial Search / Optimization (CSO), Constraint Satisfaction / Scheduling (CSS), and Mathematical / Set Problems (MSP). These were scaled up and converted to P-style format to reduce memorization and increase diversity.

3. Task suite and ASP semantics

ASPBench evaluates three tasks of increasing difficulty. These tasks are not generic prompting templates; they are defined around specific ASP semantics and solver-like behaviors (Ren et al., 26 Jul 2025).

Task	Objective	Synthetic samples
ASE	infer the truth status of a ground atom under a unique answer set	1,000
ASV	judge whether a candidate set of ground atoms is a valid answer set	1,000
ASC	construct one valid answer set of a program	1,000

In ASP entailment (ASE), the input is an ASP program $P$ and a ground atom $a$, and the output is one of true, false, or unknown, where true means $a \in S$, false means $-a \in S$, and unknown means neither appears, for the unique answer set $S$. ASE is restricted to programs with no disjunction and exactly one answer set, making it the simplest task in the suite.

In Answer set verification (ASV), the input is a program $P$ and a candidate set of ground atoms $C$, and the model must decide whether $C$ is a valid answer set of $P$. Programs in ASV may include disjunction and are guaranteed to have at least one answer set. Negative examples are constructed by taking a correct answer set and applying exactly one small corruption: Flip negation, Delete fact, Add modified fact with changed constants, or Add modified fact with changed predicate.

In Answer set computation (ASC), the input is only the program $P$0, and the required output is one valid answer set. This is described in the benchmark paper as the core ASP solving task because it requires full constructive reasoning rather than query answering or candidate checking.

The paper states the standard ASP rule form as

$P$1

Given a candidate set $P$2, the Gelfond–Lifschitz reduct $P$3 is formed by deleting rules whose body contains not α with $P$4 and then removing all remaining default-negated literals. A set $P$5 is an answer set iff it is a minimal model of $P$6. This formalism matters because ASV and ASC are explicitly meant to test whether models can operate under stable-model semantics rather than only mimic surface entailment patterns.

4. Dataset composition and evaluation protocol

The synthetic component of ASPBench contains 1,000 samples for each of ASE, ASV, and ASC. Average program sizes are reported as 10.72 rules and 9.77 facts for ASE, 15.71 rules and 14.36 facts for ASV, and 15.47 rules and 14.23 facts for ASC. Predicate arity ranges from 0 to 3, and ASE has a reported max chain length of 1–7 (Ren et al., 26 Jul 2025).

The benchmark also characterizes the syntactic classes of its programs. Approximate shares are similar across tasks: Positive programs account for about 9–10%, Stratified programs about 35%, and HCF programs about 43–45%. The average number of answer sets is 1.00 for ASE, 3.33 for ASV, and 3.44 for ASC. Predicate naming styles are roughly balanced across P-style, Related concept words, and Random concept words.

Evaluation uses macro-F1 for ASE and ASV, and Exact Match (EM) for ASC. For ASC, a prediction is counted correct if it exactly matches any one ground-truth answer set. The paper notes two implementation details that are methodologically relevant: model output format is not constrained during inference, and raw outputs are post-processed into JSON using gpt-4o-mini. For ASC, an additional alignment step maps predicted facts to ground-truth sets. The benchmark evaluates 14 LLMs, divided into general LLMs and reasoning-optimized LLMs, including deepseek-r1, o3-mini, o4-mini, and gemini-2.5-flash-thinking.

This design implies that ASPBench is not only testing final task success but also sensitivity to input representation, predicate naming, syntactic class, and answer-set multiplicity. A plausible implication is that it functions as both a benchmark and a diagnostic instrument for failure analysis in neurosymbolic reasoning.

5. Empirical findings and error patterns

ASPBench reports a pronounced hierarchy of difficulty across its three tasks. Average performance is 0.672 on symbolic ASE and 0.704 on textual ASE, 0.645 on symbolic ASV and 0.649 on textual ASV, but only 0.240 on symbolic ASC and 0.224 on textual ASC. The benchmark paper interprets this as evidence that current LLMs can perform relatively well on entailment and verification yet remain much weaker at full stable-model construction (Ren et al., 26 Jul 2025).

Among the strongest models, deepseek-r1 reports 0.976 / 0.977 on ASE across symbolic/textual inputs, 0.873 / 0.809 on ASV, and 0.817 / 0.297 on ASC. o3-mini reports 0.982 / 0.984 on ASE, 0.859 / 0.818 on ASV, and 0.600 / 0.531 on ASC. o4-mini reports 0.800 / 0.823 on ASV and 0.645 / 0.604 on ASC. The paper nonetheless emphasizes that even strong reasoning-oriented models struggle with ASC.

Performance drops sharply on the real-world ASP programs. Reported averages are 0.576 for ASV and 0.086 for ASC, substantially below the corresponding synthetic-task performance. This suggests that synthetic competence does not transfer cleanly to scaled or more structurally varied ASP instances.

The paper also reports several fine-grained effects. For ASE and ASV, textual input is slightly better on average than symbolic input; for ASC, symbolic input is slightly better. Models perform better when predicates have lexical meaning than when they are pure P-style identifiers, with RanW and RelW generally outperforming P-style. Positive programs are much easier than programs with negation or recursion; on ASC, Positive programs boost EM from about 19.4% to 56.1%. The benchmark authors interpret this as evidence that LLMs lack robust iterative and fixed-point reasoning.

Error analysis is equally central. In ASE, the dominant error is not a direct true/false reversal but overproduction of unknown: the gold proportion of unknown is 26.6%, while model outputs raise it to 47.2%. In ASV, failures are described as a completeness blind spot and weak sensitivity to answer-set count. In ASC, performance collapses as models move from checking a candidate answer set to constructing one from scratch, especially when there are three or more answer sets. The case studies attribute errors to incorrect variable unification, missing intermediate subgoals, hallucinated or misquoted rules, misunderstanding of the Gelfond–Lifschitz reduct, incomplete answer sets, wrong variable grounding, and confusion between “cannot derive” and ASP’s negation-as-failure semantics.

6. Related benchmarks, naming ambiguity, and research significance

A recurrent source of confusion in the literature is the proximity between ASPBench and ASP-Bench. The former, introduced in “Can LLMs Solve ASP Problems? Insights from a Benchmarking Study (Extended Version),” evaluates whether LLMs can solve ASP problems given symbolic or textual ASP instances and is centered on ASE, ASV, and ASC (Ren et al., 26 Jul 2025). By contrast, ASP-Bench, introduced in “ASP-Bench: From Natural Language to Logic Programs,” is a benchmark for translating natural-language specifications into ASP programs. It contains 128 natural language problem instances, organized as 64 base problems with easy and hard variants, and uses reference validators that check semantic correctness rather than exact string match (Szeider, 1 Feb 2026).

The distinction is substantive. ASPBench evaluates ASP solving, including answer-set semantics, multiple answer sets, and constructive stable-model generation. ASP-Bench evaluates natural-language-to-ASP translation, systematic coverage of ASP features such as choice rules, aggregates, and optimization, and a workflow in which an agent iteratively refines models using solver feedback. Both benchmarks are relevant to neurosymbolic engineering, but they target different stages of the ASP pipeline: one tests solving given a formal program, while the other tests synthesis from natural-language specifications.

Within that landscape, ASPBench is significant because it isolates a precise claim about current LLM capability: models are considerably stronger on entailment and verification than on answer set computation, which the paper treats as the core of ASP solving. The benchmark also suggests several methodological directions. Its results favor reasoning-optimized models over general LLMs, indicate benefits from lexical grounding in predicate names, and point toward hybrid symbolic-neural architectures or ASP-specific methods as likely future directions. This suggests that progress in ASP reasoning may depend less on generic scaling alone than on mechanisms that explicitly support stable-model construction.