LLMEval-Logic: Chinese Logical Reasoning Benchmark

• LLMEval-Logic is a benchmark that rigorously assesses rule-governed natural language reasoning through solver-backed formal verification and expert-developed rubrics.
• It employs a three-stage pipeline of forward authoring, layered normalization, and Z3 verification to ensure accurate natural-to-formal translations.
• It features two paired subsets—Base and Hard—to measure answer accuracy and reasoning fidelity using adversarial hardening and structured rubric assessments.

Searching arXiv for the benchmark and closely related logic-evaluation work to ground the article in the cited literature. LLMEval-Logic is a solver-verified Chinese benchmark for evaluating logical reasoning in LLMs under realistic natural-language conditions and adversarial difficulty control. It was introduced to address three limitations attributed to prior logical-reasoning benchmarks: reverse-templated item generation from sampled formulas, coarse or unaudited formal annotations, and rapid saturation by frontier models (Zhang et al., 19 May 2026). The benchmark is built from realistic situational scenarios, pairs natural-language items with reference formalizations in a solver-friendly formal language, verifies annotated answers with Z3, and evaluates natural-to-formal translation with expert-developed rubrics. It is released as two paired subsets—a Base subset for single-question evaluation and a Hard subset for adversarially hardened, multi-step closed-world reasoning—thereby positioning itself as both an answer-accuracy benchmark and a faithfulness benchmark for formalization (Zhang et al., 19 May 2026).

1. Conceptual scope and motivation

LLMEval-Logic targets rule-governed natural-language reasoning in settings where conclusions must follow strictly from stated premises. Its stated motivation is that many existing datasets allow models to exploit template artifacts or surface cues rather than perform genuine inference, especially when items are generated by first sampling formulas and then rendering them into natural language (Zhang et al., 19 May 2026). The benchmark therefore adopts forward authoring from realistic scenarios such as eligibility rules, scheduling, and institutional procedures, rather than reverse templating from symbolic logic.

The benchmark’s objectives are threefold. First, it seeks semantic grounding in realistic scenarios. Second, it introduces fine-grained audit of natural-to-formal translations through expert-developed rubrics that decompose faithfulness into logical relation, stated constraint, and query alignment atoms. Third, it aims for sustained difficulty through adversarial hardening designed to foil shallow pattern matching and to require enumeration, counterfactual recomputation, branching, and coreference tracking (Zhang et al., 19 May 2026).

This design places LLMEval-Logic within a broader shift in logic evaluation. DivLogicEval emphasizes controlled propositional inference in diverse natural language and introduces a metric, PartialCircular, to reduce random-guess inflation (Chung et al., 19 Sep 2025). RLMEval extends evaluation to research-level Lean theorem proving and proof autoformalization, showing that progress on curated benchmarks does not transfer cleanly to research-level formal reasoning (Poiroux et al., 29 Oct 2025). MAPLE, although developed for mathematical reasoning, similarly argues that answer accuracy alone is insufficient because it obscures intermediate reasoning flaws (Roy et al., 21 May 2025). Taken together, these works suggest an emerging evaluation agenda in which benchmark realism, formal verification, and reasoning-process assessment are treated as distinct but complementary requirements.

2. Construction pipeline and formal verification

LLMEval-Logic is constructed through a three-stage pipeline: forward authoring, expert auditing with layered normalization, and Z3-based formal verification (Zhang et al., 19 May 2026). In the forward-authoring stage, contributors with prior logic training write Chinese reasoning items embedded in realistic situational backdrops and provide both the natural-language text and a first-pass formalization in a solver-friendly “Formal Language” JSON.

The expert-auditing stage reviews clarity, natural-language–formal-language faithfulness, concept stability, and correctness of variable and predicate mapping. This auditing is coupled to a four-layer normalization procedure. At the lexical layer, punctuation, whitespace, and logical glyphs such as \land and \lor are unified. At the syntactic layer, formulas are parsed and checked for well-formedness, including operator scope and arity, using a Lark grammar. At the semantic-alignment layer, each symbol’s role, arity, and natural-language gloss are verified. At the type-and-parameter layer, the pipeline distinguishes propositional from first-order items, applies deterministic renaming, and regularizes parameters (Zhang et al., 19 May 2026). The result is a benchmark in which formal annotations are not merely attached metadata but auditable objects subjected to structured normalization.

Formal verification is performed with Z3. Each normalized item contains a premise set $\Sigma$ and one or more query formulas $\varphi$, and each query is assigned one of three task types in the Base subset: possible$(\varphi)$, which checks $\mathrm{Sat}(\Sigma \cup \{\varphi\})$; necessary$(\varphi)$, which checks $\mathrm{Unsat}(\Sigma \cup \{\neg \varphi\})$, i.e. $\Sigma \vDash \varphi$; and enumerate_models$(\ldots)$, which enumerates all satisfying assignments under a closed-world interpretation (Zhang et al., 19 May 2026). Items whose annotated answers are not solver-certified are rejected and repaired until every released item passes Z3.

A representative Base item formalizes an orchestration scenario with boolean parameters for sections such as trumpet, horn, alto horn, and tuba. The premise set includes constraints such as $H \rightarrow U$ and $A \leftrightarrow (T \wedge \neg H)$, and the query uses enumerate_models(T,H,A,U). Z3 enumerates the single assignment $\varphi$0 (Zhang et al., 19 May 2026). The example is significant because it illustrates the benchmark’s core commitment: natural-language realism is paired with explicit solver-backed semantics rather than left at the level of informal plausibility.

3. Dataset organization and logical coverage

LLMEval-Logic is released in two paired subsets with distinct evaluative roles (Zhang et al., 19 May 2026). The Base subset contains 246 single-question items and 1,400 rubric atoms in total, approximately 5.7 per item across logical relation, stated constraint, and query alignment. Its question types are possible, necessary, and enumerate_models. The Hard subset contains 190 adversarially hardened items and 938 sub-questions, with an average of 4.94 sub-questions per item and a range of 2–8. It adds two extra closed-world question types: unique_solution, which asks whether there is exactly one model, and has_alternative, which asks whether there is more than one.

The benchmark covers both propositional and first-order structure. The listed propositional constructs are $\varphi$1, $\varphi$2, $\varphi$3, $\varphi$4, and $\varphi$5. For first-order items it includes quantified formulas such as $\varphi$6 and $\varphi$7. It also includes set-valued outputs such as enumerate_models, count_models, projection of subsets, uniqueness checks, and alternative-solution checks. In addition to logical operators, the benchmark explicitly incorporates branching, distractor premises, counterfactual flips, and alias/coreference variation (Zhang et al., 19 May 2026).

The paired-subset design encodes two different notions of benchmark difficulty. Base measures whether a model can correctly encode and answer a single logically grounded item. Hard measures whether the model can preserve correctness across multi-step sub-question sequences under closed-world reasoning. This suggests that LLMEval-Logic treats “logical reasoning” not as a unitary capability but as a composite of encoding fidelity, model-space control, and robustness under perturbation. That interpretation is consistent with the observed divergence between Base and Hard rankings reported in the benchmark (Zhang et al., 19 May 2026).

4. Adversarial hardening and expert rubric design

A distinctive feature of LLMEval-Logic is its closed-loop adversarial hardening pipeline for the Hard subset (Zhang et al., 19 May 2026). The workflow contains five agent roles. The Decider diagnoses easy solution paths and emits a hardening blueprint covering six strategies. Two Proposal sub-agents rewrite the background and the questions according to that blueprint. The Review stage audits design completeness and well-posedness before model answering. The Answering stage uses an ensemble of LLMs to answer and produce reasoning traces. The Verification stage checks answer correctness, reasoning validity, and whether shallow-path leaks remain, and it can trigger repair or return the item to the proposal stage.

The six structural hardening strategies are add_branching, add_distractor_premise, change_question_to_set_output, add_uncertainty_or_multi_answer, add_counterfactual_variant, and alias_and_coreference_variation (Zhang et al., 19 May 2026). Each strategy must demonstrably alter the closed candidate space or the reasoning path. Of 254 candidates, 224 completed the workflow and 190 final items were released. This procedural filtering is important because it makes “hardness” an audited property of reasoning structure rather than a subjective claim.

The benchmark’s rubric system serves a different but complementary function. For each Base item, experts decompose faithfulness into three rubric groups. logical_relation checks whether the formalization correctly uses operators such as $\varphi$8, $\varphi$9, $(\varphi)$0, $(\varphi)$1, $(\varphi)$2, $(\varphi)$3, and $(\varphi)$4. stated_constraint checks domains, type restrictions, and boundary conditions. query_alignment checks whether the formal query matches the natural-language question (Zhang et al., 19 May 2026). Each rubric atom is paired with a natural-language criterion and a Z3-checkable statement, and the gold formalization must pass all rubric atoms before the item enters the benchmark.

The grading guidelines impose both a sufficient-strength and a not-too-strong requirement. A candidate formalization must include the required relation or constraint, but it must not introduce unjustified strengthening, such as replacing $(\varphi)$5 with $(\varphi)$6 unless warranted (Zhang et al., 19 May 2026). Query alignment allows equivalent enumeration, counting, or projection transformations under closed-world semantics. In the fixed-symbol setting, a Z3 premise-equivalence gate automatically passes logical_relation and stated_constraint when the candidate premise set is logically equivalent; remaining differences are then soft-reviewed by an LLM to permit defensible alternative readings (Zhang et al., 19 May 2026). This combination of symbolic and rubric-based evaluation addresses a recurrent benchmark problem: a formalization can yield the right answer for the wrong reasons, or satisfy local rubric criteria while remaining globally mismatched. LLMEval-Logic explicitly treats these as separate failure modes.

5. Evaluation methodology and empirical results

LLMEval-Logic evaluates 14 frontier LLMs, including think-enabled variants and “no-think” or “low-think” variants of Gemini 3.1 Pro, Claude Opus 4.6, GPT-5.4 Pro, Qwen 3.5 Plus, Kimi K2.5, Hy3 preview, and Seed 2.0 Pro (Zhang et al., 19 May 2026). The reported metrics are Base Item Accuracy, Hard Item Accuracy, Hard Sub-Question Accuracy, and formalization accuracy over the Base subset measured by Z3 execution success versus reference, rubric pass-rate over logical relation plus stated constraint plus query alignment atoms, and a joint Z3+Rubric signal.

The scoring protocol uses LLM-as-Judge with gpt-5.1-chat for semantic scoring of answers and rubric atoms. Results are reported over three independent runs as mean ± standard deviation. Inter-judge validation with Claude Opus 4.6 and Gemini 3.1 Pro yields Cohen’s $(\varphi)$7, described as “almost perfect” (Zhang et al., 19 May 2026). The inclusion of inter-judge validation is notable because it addresses a standard concern in model-graded benchmarks: whether grading variance overwhelms the signal of interest.

The main findings indicate that forward-authored Hard items remain substantially difficult. Average Base accuracy is approximately 65%, whereas Hard accuracy is approximately 23%, a drop of approximately 42 percentage points. The best Hard Item Accuracy is 37.5%, achieved by Gemini 3.1 Pro (Zhang et al., 19 May 2026). The benchmark also reports that Z3 and rubric signals are complementary. Z3 correctness can overestimate faithfulness because over-constrained formalizations may accidentally produce the same answer, whereas positive-only rubric checks may pass extra or mis-specified constraints; their conjunction is therefore presented as a stricter proxy (Zhang et al., 19 May 2026).

Providing reference symbols in a “fixed” setting improves formalization performance but does not close the gap to perfect faithfulness. One reported example is that GPT-5.4 Pro no-think increases its joint Z3+Rubric score from 32.9% to 60.16% under the fixed setting (Zhang et al., 19 May 2026). This result suggests that symbol grounding is a significant but incomplete component of the natural-to-formal translation problem.

The reported error patterns include modal quantifier tracking failures, incomplete model enumeration, counterfactual recomputation failures, projection-level errors on nested set-families, and evidence-provenance and validity-tier breakdowns (Zhang et al., 19 May 2026). These patterns align with observations from related benchmarks. DivLogicEval reports hallucinations and misinterpretation of conditional versus factual statements as common failure modes in natural-language propositional reasoning (Chung et al., 19 Sep 2025). RLMEval finds that brute-force sampling alone does not resolve failures involving long dependency chains, missing intermediate lemmas, and research-level context retrieval (Poiroux et al., 29 Oct 2025). A plausible implication is that logical weakness in current LLMs is not confined to a single representation regime; it appears in solver-grounded natural language, controlled propositional inference, and proof-assistant-mediated formal mathematics alike.

6. Relation to adjacent logic-evaluation benchmarks

LLMEval-Logic can be situated relative to at least three neighboring evaluation paradigms represented in recent work. DivLogicEval is a multiple-choice machine reading comprehension benchmark designed to isolate classical propositional logic reasoning by deriving each question from sound propositional logic expressed in natural language (Chung et al., 19 Sep 2025). Its construction uses symbolic proposition sampling, natural-language instantiation with SNLI/MNLI sentences, and counterintuitive composition to prevent reliance on surface plausibility. It also introduces PartialCircular, defined by

$(\varphi)$8

to combine consistency across cyclic option permutations with an entropy-based confidence penalty (Chung et al., 19 Sep 2025). Compared with DivLogicEval, LLMEval-Logic moves beyond multiple-choice entailment toward solver-checked closed-world reasoning and explicit natural-to-formal faithfulness.

RLMEval operates in a substantially different regime: research-level mathematics formalized in Lean 4 (Poiroux et al., 29 Oct 2025). It evaluates neural theorem proving from formal statements alone and proof autoformalization from informal proof sketches plus formal statements. Its core metric is pass@k, defined as the fraction of theorems for which at least one of $(\varphi)$9 proof attempts type-checks in Lean. On 613 theorems from six Lean Blueprint projects, the best model reaches 10.3% in proof autoformalization at pass@128 (Poiroux et al., 29 Oct 2025). Relative to RLMEval, LLMEval-Logic is less concerned with full proof synthesis and more concerned with audited natural-language formalization and solver-grounded inference over realistic scenarios. The two are complementary: one measures end-to-end formal proof construction, the other measures premise encoding, query alignment, and model-space reasoning under symbolic verification.

MAPLE proposes a reasoning-process metric for mathematical solutions by combining error rate, redundancy, and validity into a misalignment score (Roy et al., 21 May 2025). Its formal definition uses a penalty-weighted, log-smoothed error frequency

$\mathrm{Sat}(\Sigma \cup \{\varphi\})$0

and then computes

$\mathrm{Sat}(\Sigma \cup \{\varphi\})$1

where $\mathrm{Sat}(\Sigma \cup \{\varphi\})$2 is redundancy and $\mathrm{Sat}(\Sigma \cup \{\varphi\})$3 is validity (Roy et al., 21 May 2025). MAPLE’s relevance to LLMEval-Logic lies less in domain overlap than in evaluative philosophy: both reject final-answer accuracy as a sufficient measure of reasoning quality. LLMEval-Logic operationalizes this via Z3 plus rubrics; MAPLE does so via error labeling, redundancy, and validity.

This comparison suggests a useful taxonomy. DivLogicEval prioritizes controlled propositional isolation and bias-resistant multiple-choice scoring. RLMEval prioritizes proof-search realism in a formal proof assistant. MAPLE prioritizes diagnosis of intermediate reasoning flaws. LLMEval-Logic prioritizes realistic scenario authoring, solver-certified answers, and expert-audited natural-to-formal faithfulness (Zhang et al., 19 May 2026, Chung et al., 19 Sep 2025, Poiroux et al., 29 Oct 2025, Roy et al., 21 May 2025).

7. Significance, limitations, and implications for logic evaluation

LLMEval-Logic’s principal significance lies in combining three elements that are often separated in benchmark design: realistic forward-authored natural language, solver-backed verification, and explicit grading of natural-to-formal faithfulness (Zhang et al., 19 May 2026). The benchmark therefore evaluates not only whether a model reaches the correct conclusion, but also whether it constructs an appropriate formal representation of the scenario and whether that representation matches the intended query under closed-world semantics.

Its results show that even strong contemporary models remain far from saturating adversarially hardened logical reasoning in Chinese. The best Hard Item Accuracy is 37.5%, and the highest joint Z3+Rubric formalization score reported under reference-symbol assistance is 60.16% (Zhang et al., 19 May 2026). These figures matter because they indicate residual weakness at two levels: inference over the induced model space and faithful translation from natural language into formal constraints.

Several limitations are implicit in the benchmark’s own design. The use of LLM-as-Judge introduces a semantic scoring layer that, while validated with high inter-judge agreement, remains distinct from direct symbolic verification (Zhang et al., 19 May 2026). The benchmark is also language-specific, being designed for Chinese, though its rubric and hardening framework are explicitly presented as reusable in other languages or domains. A plausible implication is that cross-lingual transfer of logical ability cannot be assumed from strong English-only results on neighboring benchmarks.

The benchmark also contributes methodological guidance for future “LLMEval-Logic” suites. DivLogicEval concludes that a tightly controlled, highly diverse natural-language dataset with bias-mitigating metrics can reveal substantial headroom in formal reasoning (Chung et al., 19 Sep 2025). RLMEval recommends research-level tasks, dual evaluation modes, controlled context access, pass@k reporting, and annual versioning to mitigate data leakage (Poiroux et al., 29 Oct 2025). MAPLE suggests extending beyond terminal accuracy to assess the structure and validity of reasoning chains (Roy et al., 21 May 2025). LLMEval-Logic synthesizes many of these impulses in a single benchmark by combining realistic item design, symbolic certification, adversarial hardening, and rubric-based formalization assessment (Zhang et al., 19 May 2026).

The benchmark is publicly released with Base and Hard subsets, forward-authoring and adversarial-hardening code, formalizations, rubrics, and audit traces, and items are stored in JSON with natural-language text, formal-language parameters, premises, questions, and rubric checklist atoms (Zhang et al., 19 May 2026). In practical terms, it offers a reference design for evaluating logic not merely as answer selection, but as a pipeline spanning interpretation, formalization, inference, and robustness.