---
title: 'LLMEval-Logic: Chinese Logical Reasoning Benchmark'
url: https://www.emergentmind.com/topics/llmeval-logic
type: topic
---

# LLMEval-Logic: Chinese Logical Reasoning Benchmark

Searching arXiv for the benchmark and closely related logic-evaluation work to ground the article in the cited literature.
LLMEval-Logic is a solver-verified Chinese benchmark for evaluating logical reasoning in large language models under realistic natural-language conditions and adversarial difficulty control. It was introduced to address three limitations attributed to prior logical-reasoning benchmarks: reverse-templated item generation from sampled formulas, coarse or unaudited formal annotations, and rapid saturation by frontier models [2605.19597]. The benchmark is built from realistic situational scenarios, pairs natural-language items with reference formalizations in a solver-friendly formal language, verifies annotated answers with Z3, and evaluates natural-to-formal translation with expert-developed rubrics. It is released as two paired subsets—a Base subset for single-question evaluation and a Hard subset for adversarially hardened, multi-step closed-world reasoning—thereby positioning itself as both an answer-accuracy benchmark and a faithfulness benchmark for formalization [2605.19597].

## 1. Conceptual scope and motivation

LLMEval-Logic targets rule-governed natural-language reasoning in settings where conclusions must follow strictly from stated premises. Its stated motivation is that many existing datasets allow models to exploit template artifacts or surface cues rather than perform genuine inference, especially when items are generated by first sampling formulas and then rendering them into natural language [2605.19597]. The benchmark therefore adopts forward authoring from realistic scenarios such as eligibility rules, scheduling, and institutional procedures, rather than reverse templating from symbolic logic.

The benchmark’s objectives are threefold. First, it seeks semantic grounding in realistic scenarios. Second, it introduces fine-grained audit of natural-to-formal translations through expert-developed rubrics that decompose faithfulness into logical relation, stated constraint, and query alignment atoms. Third, it aims for sustained difficulty through adversarial hardening designed to foil shallow pattern matching and to require enumeration, counterfactual recomputation, branching, and coreference tracking [2605.19597].

This design places LLMEval-Logic within a broader shift in logic evaluation. DivLogicEval emphasizes controlled propositional inference in diverse natural language and introduces a metric, PartialCircular, to reduce random-guess inflation [2509.15587]. RLMEval extends evaluation to research-level Lean theorem proving and proof autoformalization, showing that progress on curated benchmarks does not transfer cleanly to research-level formal reasoning [2510.25427]. MAPLE, although developed for mathematical reasoning, similarly argues that answer accuracy alone is insufficient because it obscures intermediate reasoning flaws [2505.15623]. Taken together, these works suggest an emerging evaluation agenda in which benchmark realism, formal verification, and reasoning-process assessment are treated as distinct but complementary requirements.

## 2. Construction pipeline and formal verification

LLMEval-Logic is constructed through a three-stage pipeline: forward authoring, expert auditing with layered normalization, and Z3-based formal verification [2605.19597]. In the forward-authoring stage, contributors with prior logic training write Chinese reasoning items embedded in realistic situational backdrops and provide both the natural-language text and a first-pass formalization in a solver-friendly “Formal Language” JSON.

The expert-auditing stage reviews clarity, natural-language–formal-language faithfulness, concept stability, and correctness of variable and predicate mapping. This auditing is coupled to a four-layer normalization procedure. At the lexical layer, punctuation, whitespace, and logical glyphs such as `\land` and `\lor` are unified. At the syntactic layer, formulas are parsed and checked for well-formedness, including operator scope and arity, using a Lark grammar. At the semantic-alignment layer, each symbol’s role, arity, and natural-language gloss are verified. At the type-and-parameter layer, the pipeline distinguishes propositional from first-order items, applies deterministic renaming, and regularizes parameters [2605.19597]. The result is a benchmark in which formal annotations are not merely attached metadata but auditable objects subjected to structured normalization.

Formal verification is performed with Z3. Each normalized item contains a premise set $\Sigma$ and one or more query formulas $\varphi$, and each query is assigned one of three task types in the Base subset: possible$(\varphi)$, which checks $\mathrm{Sat}(\Sigma \cup \{\varphi\})$; necessary$(\varphi)$, which checks $\mathrm{Unsat}(\Sigma \cup \{\neg \varphi\})$, i.e. $\Sigma \vDash \varphi$; and enumerate_models$(\ldots)$, which enumerates all satisfying assignments under a closed-world interpretation [2605.19597]. Items whose annotated answers are not solver-certified are rejected and repaired until every released item passes Z3.

A representative Base item formalizes an orchestration scenario with boolean parameters for sections such as trumpet, horn, alto horn, and tuba. The premise set includes constraints such as $H \rightarrow U$ and $A \leftrightarrow (T \wedge \neg H)$, and the query uses `enumerate_models(T,H,A,U)`. Z3 enumerates the single assignment $H=U=\text{true},\,T=A=\text{false}$ [2605.19597]. The example is significant because it illustrates the benchmark’s core commitment: natural-language realism is paired with explicit solver-backed semantics rather than left at the level of informal plausibility.

## 3. Dataset organization and logical coverage

LLMEval-Logic is released in two paired subsets with distinct evaluative roles [2605.19597]. The Base subset contains 246 single-question items and 1,400 rubric atoms in total, approximately 5.7 per item across logical relation, stated constraint, and query alignment. Its question types are possible, necessary, and enumerate_models. The Hard subset contains 190 adversarially hardened items and 938 sub-questions, with an average of 4.94 sub-questions per item and a range of 2–8. It adds two extra closed-world question types: unique_solution, which asks whether there is exactly one model, and has_alternative, which asks whether there is more than one.

The benchmark covers both propositional and first-order structure. The listed propositional constructs are $\neg A$, $A \wedge B$, $A \vee B$, $A \rightarrow B$, and $A \leftrightarrow B$. For first-order items it includes quantified formulas such as $\forall x\,P(x)\rightarrow Q(x)$ and $\exists x\,P(x)$. It also includes set-valued outputs such as enumerate_models, count_models, projection of subsets, uniqueness checks, and alternative-solution checks. In addition to logical operators, the benchmark explicitly incorporates branching, distractor premises, counterfactual flips, and alias/coreference variation [2605.19597].

The paired-subset design encodes two different notions of benchmark difficulty. Base measures whether a model can correctly encode and answer a single logically grounded item. Hard measures whether the model can preserve correctness across multi-step sub-question sequences under closed-world reasoning. This suggests that LLMEval-Logic treats “logical reasoning” not as a unitary capability but as a composite of encoding fidelity, model-space control, and robustness under perturbation. That interpretation is consistent with the observed divergence between Base and Hard rankings reported in the benchmark [2605.19597].

## 4. Adversarial hardening and expert rubric design

A distinctive feature of LLMEval-Logic is its closed-loop adversarial hardening pipeline for the Hard subset [2605.19597]. The workflow contains five agent roles. The Decider diagnoses easy solution paths and emits a hardening blueprint covering six strategies. Two Proposal sub-agents rewrite the background and the questions according to that blueprint. The Review stage audits design completeness and well-posedness before model answering. The Answering stage uses an ensemble of LLMs to answer and produce reasoning traces. The Verification stage checks answer correctness, reasoning validity, and whether shallow-path leaks remain, and it can trigger repair or return the item to the proposal stage.

The six structural hardening strategies are add_branching, add_distractor_premise, change_question_to_set_output, add_uncertainty_or_multi_answer, add_counterfactual_variant, and alias_and_coreference_variation [2605.19597]. Each strategy must demonstrably alter the closed candidate space or the reasoning path. Of 254 candidates, 224 completed the workflow and 190 final items were released. This procedural filtering is important because it makes “hardness” an audited property of reasoning structure rather than a subjective claim.

The benchmark’s rubric system serves a different but complementary function. For each Base item, experts decompose faithfulness into three rubric groups. **logical_relation** checks whether the formalization correctly uses operators such as $\rightarrow$, $\leftrightarrow$, $\neg$, $\wedge$, $\vee$, $\forall$, and $\exists$. **stated_constraint** checks domains, type restrictions, and boundary conditions. **query_alignment** checks whether the formal query matches the natural-language question [2605.19597]. Each rubric atom is paired with a natural-language criterion and a Z3-checkable statement, and the gold formalization must pass all rubric atoms before the item enters the benchmark.

The grading guidelines impose both a sufficient-strength and a not-too-strong requirement. A candidate formalization must include the required relation or constraint, but it must not introduce unjustified strengthening, such as replacing $\to$ with $\leftrightarrow$ unless warranted [2605.19597]. Query alignment allows equivalent enumeration, counting, or projection transformations under closed-world semantics. In the fixed-symbol setting, a Z3 premise-equivalence gate automatically passes logical_relation and stated_constraint when the candidate premise set is logically equivalent; remaining differences are then soft-reviewed by an LLM to permit defensible alternative readings [2605.19597]. This combination of symbolic and rubric-based evaluation addresses a recurrent benchmark problem: a formalization can yield the right answer for the wrong reasons, or satisfy local rubric criteria while remaining globally mismatched. LLMEval-Logic explicitly treats these as separate failure modes.

## 5. Evaluation methodology and empirical results

LLMEval-Logic evaluates 14 frontier LLMs, including think-enabled variants and “no-think” or “low-think” variants of Gemini 3.1 Pro, Claude Opus 4.6, GPT-5.4 Pro, Qwen 3.5 Plus, Kimi K2.5, Hy3 preview, and Seed 2.0 Pro [2605.19597]. The reported metrics are Base Item Accuracy, Hard Item Accuracy, Hard Sub-Question Accuracy, and formalization accuracy over the Base subset measured by Z3 execution success versus reference, rubric pass-rate over logical relation plus stated constraint plus query alignment atoms, and a joint Z3+Rubric signal.

The scoring protocol uses LLM-as-Judge with gpt-5.1-chat for semantic scoring of answers and rubric atoms. Results are reported over three independent runs as mean ± standard deviation. Inter-judge validation with Claude Opus 4.6 and Gemini 3.1 Pro yields Cohen’s $\kappa \in [0.873,0.922]$, described as “almost perfect” [2605.19597]. The inclusion of inter-judge validation is notable because it addresses a standard concern in model-graded benchmarks: whether grading variance overwhelms the signal of interest.

The main findings indicate that forward-authored Hard items remain substantially difficult. Average Base accuracy is approximately 65%, whereas Hard accuracy is approximately 23%, a drop of approximately 42 percentage points. The best Hard Item Accuracy is 37.5%, achieved by Gemini 3.1 Pro [2605.19597]. The benchmark also reports that Z3 and rubric signals are complementary. Z3 correctness can overestimate faithfulness because over-constrained formalizations may accidentally produce the same answer, whereas positive-only rubric checks may pass extra or mis-specified constraints; their conjunction is therefore presented as a stricter proxy [2605.19597].

Providing reference symbols in a “fixed” setting improves formalization performance but does not close the gap to perfect faithfulness. One reported example is that GPT-5.4 Pro no-think increases its joint Z3+Rubric score from 32.9% to 60.16% under the fixed setting [2605.19597]. This result suggests that symbol grounding is a significant but incomplete component of the natural-to-formal translation problem.

The reported error patterns include modal quantifier tracking failures, incomplete model enumeration, counterfactual recomputation failures, projection-level errors on nested set-families, and evidence-provenance and validity-tier breakdowns [2605.19597]. These patterns align with observations from related benchmarks. DivLogicEval reports hallucinations and misinterpretation of conditional versus factual statements as common failure modes in natural-language propositional reasoning [2509.15587]. RLMEval finds that brute-force sampling alone does not resolve failures involving long dependency chains, missing intermediate lemmas, and research-level context retrieval [2510.25427]. A plausible implication is that logical weakness in current LLMs is not confined to a single representation regime; it appears in solver-grounded natural language, controlled propositional inference, and proof-assistant-mediated formal mathematics alike.

## 6. Relation to adjacent logic-evaluation benchmarks

LLMEval-Logic can be situated relative to at least three neighboring evaluation paradigms represented in recent work. DivLogicEval is a multiple-choice machine reading comprehension benchmark designed to isolate classical propositional logic reasoning by deriving each question from sound propositional logic expressed in natural language [2509.15587]. Its construction uses symbolic proposition sampling, natural-language instantiation with SNLI/MNLI sentences, and counterintuitive composition to prevent reliance on surface plausibility. It also introduces PartialCircular, defined by
$$
\mathrm{PC}
\;=\;
\frac{c}{4}
\;\times\;
\Bigl(
1 + \sum_{i=1}^{4} p(o_i)\,\log_{4} p(o_i)
\Bigr),
$$
to combine consistency across cyclic option permutations with an entropy-based confidence penalty [2509.15587]. Compared with DivLogicEval, LLMEval-Logic moves beyond multiple-choice entailment toward solver-checked closed-world reasoning and explicit natural-to-formal faithfulness.

RLMEval operates in a substantially different regime: research-level mathematics formalized in Lean 4 [2510.25427]. It evaluates neural theorem proving from formal statements alone and proof autoformalization from informal proof sketches plus formal statements. Its core metric is pass@k, defined as the fraction of theorems for which at least one of $k$ proof attempts type-checks in Lean. On 613 theorems from six Lean Blueprint projects, the best model reaches 10.3% in proof autoformalization at pass@128 [2510.25427]. Relative to RLMEval, LLMEval-Logic is less concerned with full proof synthesis and more concerned with audited natural-language formalization and solver-grounded inference over realistic scenarios. The two are complementary: one measures end-to-end formal proof construction, the other measures premise encoding, query alignment, and model-space reasoning under symbolic verification.

MAPLE proposes a reasoning-process metric for mathematical solutions by combining error rate, redundancy, and validity into a misalignment score [2505.15623]. Its formal definition uses a penalty-weighted, log-smoothed error frequency
$$
e =
\frac{\sum_{l \in L} w_l \cdot \log(1 + f_l)}
{\sum_{l \in L} w_l},
$$
and then computes
$$
\mathrm{MAPLE\_score} = \tanh(e \cdot v / r),
$$
where $r$ is redundancy and $v$ is validity [2505.15623]. MAPLE’s relevance to LLMEval-Logic lies less in domain overlap than in evaluative philosophy: both reject final-answer accuracy as a sufficient measure of reasoning quality. LLMEval-Logic operationalizes this via Z3 plus rubrics; MAPLE does so via error labeling, redundancy, and validity.

This comparison suggests a useful taxonomy. DivLogicEval prioritizes controlled propositional isolation and bias-resistant multiple-choice scoring. RLMEval prioritizes proof-search realism in a formal proof assistant. MAPLE prioritizes diagnosis of intermediate reasoning flaws. LLMEval-Logic prioritizes realistic scenario authoring, solver-certified answers, and expert-audited natural-to-formal faithfulness [2605.19597; 2509.15587; 2510.25427; 2505.15623].

## 7. Significance, limitations, and implications for logic evaluation

LLMEval-Logic’s principal significance lies in combining three elements that are often separated in benchmark design: realistic forward-authored natural language, solver-backed verification, and explicit grading of natural-to-formal faithfulness [2605.19597]. The benchmark therefore evaluates not only whether a model reaches the correct conclusion, but also whether it constructs an appropriate formal representation of the scenario and whether that representation matches the intended query under closed-world semantics.

Its results show that even strong contemporary models remain far from saturating adversarially hardened logical reasoning in Chinese. The best Hard Item Accuracy is 37.5%, and the highest joint Z3+Rubric formalization score reported under reference-symbol assistance is 60.16% [2605.19597]. These figures matter because they indicate residual weakness at two levels: inference over the induced model space and faithful translation from natural language into formal constraints.

Several limitations are implicit in the benchmark’s own design. The use of LLM-as-Judge introduces a semantic scoring layer that, while validated with high inter-judge agreement, remains distinct from direct symbolic verification [2605.19597]. The benchmark is also language-specific, being designed for Chinese, though its rubric and hardening framework are explicitly presented as reusable in other languages or domains. A plausible implication is that cross-lingual transfer of logical ability cannot be assumed from strong English-only results on neighboring benchmarks.

The benchmark also contributes methodological guidance for future “LLMEval-Logic” suites. DivLogicEval concludes that a tightly controlled, highly diverse natural-language dataset with bias-mitigating metrics can reveal substantial headroom in formal reasoning [2509.15587]. RLMEval recommends research-level tasks, dual evaluation modes, controlled context access, pass@k reporting, and annual versioning to mitigate data leakage [2510.25427]. MAPLE suggests extending beyond terminal accuracy to assess the structure and validity of reasoning chains [2505.15623]. LLMEval-Logic synthesizes many of these impulses in a single benchmark by combining realistic item design, symbolic certification, adversarial hardening, and rubric-based formalization assessment [2605.19597].

The benchmark is publicly released with Base and Hard subsets, forward-authoring and adversarial-hardening code, formalizations, rubrics, and audit traces, and items are stored in JSON with natural-language text, formal-language parameters, premises, questions, and rubric checklist atoms [2605.19597]. In practical terms, it offers a reference design for evaluating logic not merely as answer selection, but as a pipeline spanning interpretation, formalization, inference, and robustness.

Source: https://www.emergentmind.com/topics/llmeval-logic