LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening

Abstract: Evaluating LLMs on natural-language logical reasoning is essential because rule-governed tasks require conclusions to follow strictly from stated premises. Many existing logical-reasoning benchmarks are generated by templating natural-language items from sampled formulas, provide only coarse or unaudited formal annotations, and are now quickly saturated by frontier reasoning models. We present LLMEval-Logic, a Chinese logical reasoning benchmark built from realistic situational scenarios. Its pipeline forward-authors and expert-audits natural-language items together with their reference formalizations, verifies annotated answers with Z3, constructs expert rubrics for natural-to-formal grading, and hardens selected items through a closed-loop adversarial workflow. The benchmark is released in two paired subsets: a 246-item Base subset shipped with 1,400 expert-developed rubric atoms, and a 190-item Hard subset with 938 multi-step sub-questions over closed model spaces. Evaluating 14 frontier LLMs on LLMEval-Logic reveals substantial gaps in current models: the best model reaches only 37.5% Hard Item Accuracy, and even with reference symbols the highest joint Z3+Rubric formalization score among evaluated models reaches only 60.16%. Our benchmark is publicly available at https://github.com/llmeval/LLMEval-Logic.

• The paper introduces a Chinese benchmark that leverages Z3 solver verification and a closed-loop adversarial hardening process to challenge LLMs on logical reasoning tasks.
• It employs a multi-stage pipeline with expert audits, natural-to-formal translations, and detailed rubric evaluations to ensure fine-grained assessment.
• Empirical results reveal notable performance drops between Base and Hard subsets, emphasizing the need for improved semantic translation and multi-step reasoning in LLMs.

LLMEval-Logic: A Solver-Verified Chinese Benchmark for LLM Logical Reasoning with Adversarial Hardening

Motivation and Problem Formulation

Rigorous evaluation of LLM reasoning under natural language is critical for deployment in contexts with non-negotiable logical correctness requirements. Current benchmarks for logical reasoning frequently suffer from three key limitations: (1) template-based item generation leading to distributional artifacts, (2) insufficient or coarse-grained auditing of natural-to-formal translations, and (3) lack of discriminative headroom due to rapid saturation of model performance on extant datasets. The "LLMEval-Logic: A Solver-Verified Chinese Benchmark for Logical Reasoning of LLMs with Adversarial Hardening" (2605.19597) paper addresses these gaps by developing a Chinese benchmark that (i) forward-authors realistic, scenario-based logical reasoning tasks, (ii) pairs them with Z3-verified formalizations and decompositional rubrics, and (iii) applies a closed-loop adversarial hardening protocol to sharpen discrimination on frontier models.

Figure 1: The LLMEval-Logic construction pipeline—spanning forward-authored items, expert audits, Z3 solver verification, rubric construction, and adversarial hardening, resulting in challenging Base and Hard subsets evaluated along answer and formalization axes.

Dataset Construction and Benchmarking Protocol

LLMEval-Logic’s pipeline begins with 521 human-authored reasoning items, authored in Chinese by contributors with formal logic training and grounded in intricate, real-world inspired scenarios (e.g., eligibility rules, institutional procedures, scheduling). Each item is meticulously mapped to a formal representation (PL or FOL), which undergoes expert review and a four-stage normalization pipeline (lexical, syntactic, semantic alignment, and type/parameter consistency). Verified items are certified for logical correctness using the Z3 SMT solver, separating items into three types: possible (satisfiability), necessary (entailment), and enumerate_models (explicit model enumeration). Each item is paired with expert-developed rubric atoms that independently audit logical-relation fidelity, stated-constraint preservation, and query alignment at a fine semantic granularity.

A central feature is the adversarial hardening pipeline applied to items that are not challenging enough for current LLMs. This closed-loop process involves multi-role agents (Decider, Proposal, Review, Answering, Verification) applying strategies such as branching, distractors, uncertainty, counterfactuals, set-valued outputs, and alias/coreference variation—ensuring items are not only superficially altered but force genuinely new computation over closed candidate spaces. The resulting benchmark is split into a 246-item Base subset (single-question, rubric-paired) and a 190-item Hard subset comprising 938 multi-step sub-questions, with each Hard item requiring all sub-questions correct for full credit.

Figure 2: Per-family Item Accuracy transition from Base to Hard subsets, detailing dramatic accuracy drop and highlighting reasoning-specific family weaknesses.

Evaluation Metrics and Protocols

LLMEval-Logic implements a dual-axis evaluation: (1) answer accuracy (item/sub-question matching for LLM responses), and (2) formalization accuracy, computed with Z3 execution and rubric-matching for model-produced formalizations. Evaluation uses both "free" (model picks all formal symbols) and "fixed" (reference symbols provided) variants. All answer and rubric judgments are performed by LLM-as-judge protocols (primarily GPT-5.1-chat), with robust inter-judge agreement validation across multiple models (Cohen’s $\kappa$ $>0.87$).

Empirical Findings

Robust Shortcut Resistance and Discriminative Difficulty

Forward-authored, scenario-based items exhibit high resistance to template-based shortcuts. Across 14 SOTA LLMs, mean Item Accuracy is $65.1\%$ on Base but plummets to $22.9\%$ on Hard, with the leading model scoring only $37.5\pm3.8\%$ on Hard. The adversarial hardening protocol ensures that accuracy does not saturate even among cutting-edge models, as Hard items systematically stress global candidate set maintenance, multi-axis uncertainty, and recomputation under counterfactuals.

Fine-Grained Audit of Natural-to-Formal Translation

The benchmark’s rubric system yields a much stricter and more diagnostic audit of model-generated formalizations than answer-only or Z3-only signals. Notably, models often produce formalizations that pass Z3 execution (i.e., derive the correct answer), yet fail the rubric audit due to missing logical relations or constraints irrelevant to a discrete query. The best non-gold formalization scores (joint Z3+Rubric) peak at $60.16\%$ even when provided reference symbols, a sharp indicator that model outputs still fail to encode source semantics robustly. The gold reference achieves $100\%$ by construction, underscoring the difficulty of the NL$\to$FOL mapping for current LLMs.

Figure 3: Z3 vs. Rubric accuracy scatter—models’ improvement with fixed symbols is evident, but no model approaches gold-level unity.

Substantial Model and Protocol-Dependent Variance

The shift from Base to Hard ranking is non-monotonic: top Base performers (e.g., Seed 2.0 Pro, Hy3 preview) experience severe rank drops on Hard, while others (e.g., Claude Opus 4.6) comparatively improve, demonstrating that single-question metrics are not predictive for adversarial, closed-world reasoning. Disabling "thinking" in model prompting (removal of explicit reasoning steps) disproportionately affects open-weight Chinese models, accentuating the necessity of step-level reasoning on these diagnostic tasks.

Error Analysis

Taxonomy of error reveals: (i) modal quantifier confusion (mixing existential/possibilistic with universal/necessary levels), (ii) incomplete model enumeration, (iii) failure to propagate counterfactual updates through the entire reasoning chain, and (iv) projection and provenance errors when recomputation is required across entity or set families. In Hard items, these failures are magnified due to stringent scoring criteria (all sub-questions must be correct), and high rates of local partial credit (solving individual sub-questions but failing global consistency).

Implications and Future Directions

LLMEval-Logic surfaces nontrivial empirical gaps and demonstrates that modern LLMs, even at top scale, remain far from robust performance on realistic, scenario-grounded logical reasoning under natural language in structurally rich settings. The results show that high answer accuracy on single questions is not a reliable signal for robust model-based reasoning in adversarial multi-branch scenarios. The decoupled Z3 and rubric signals suggest complementary future directions: models must evolve not just in surface pattern matching or shallow CoT but in genuinely faithful semantic translation and global candidate-space maintenance.

From a practical standpoint, the benchmark's approach and evaluation protocols can generalize to other domains (e.g., legal, clinical, regulatory audit), and the adversarial hardening pipeline provides a blueprint for sustaining benchmark difficulty as models advance. Theoretically, this work directly engages with the gap between language-based and formal-symbolic reasoning, with implications for developing hybrid neuro-symbolic architectures and targeted RLHF for reasoning subskills.

Conclusion

LLMEval-Logic establishes a new standard for logical reasoning evaluation in LLMs by combining forward-authored Chinese scenarios, Z3 solver verification, fine-grained semantic rubrics, and an adversarial hardening framework. It exposes significant limitations in current LLMs' ability to perform robust logical reasoning and maintain formal alignment across both single-query and adversarially recomposed multi-query tasks. This benchmark and its methodology present a comprehensive diagnostic environment and a robust challenge set for the next generation of LLM research in logical reasoning (2605.19597).