LogicBench: Multi-Domain Logical Benchmark

Updated 8 July 2026

LogicBench is a designation for multiple distinct benchmarks assessing logical reasoning in natural language, multimodal contexts, and quantum circuit synthesis.
In LLM evaluations, it features around 1,520 binary QA and 500 multiple-choice QA samples covering 25 inference patterns, exposing gaps between human and model performance.
For quantum synthesis, LogicBench provides benchmark circuits that quantify qubit counts, T-counts, and runtime trade-offs, aiding efficient quantum design.

LogicBench is a designation used for several distinct benchmarking resources in contemporary computing research. In arXiv literature, the name has been applied to a natural-language benchmark for evaluating the logical reasoning ability of LLMs, a multimodal benchmark for diagnosing logical blindspots in vision-LLMs, and a benchmark suite associated with quantum logic synthesis via LUT-based Hierarchical Reversible Logic Synthesis. These uses share an emphasis on systematic evaluation of “logic,” but they operationalize that term in materially different ways: formal inference in text, logical structure in image-text alignment, and resource-cost assessment for synthesized logic circuits (Parmar et al., 2024, Zhou et al., 15 Aug 2025, Soeken et al., 2017).

1. Major usages of the name

The most widely defined recent usage is LogicBench as a natural language question-answering dataset for logical reasoning in LLMs. A second, unrelated usage defines LogicBench as a large-scale benchmark for logical understanding in vision-LLMs across images, videos, anomaly detection, and medical diagnostics. An earlier usage in quantum computing associates LogicBench with benchmark circuits and mapping statistics produced by the LHRS framework, where the emphasis is on qubit counts, $T$ -count, and runtime rather than linguistic inference (Parmar et al., 2024, Zhou et al., 15 Aug 2025, Soeken et al., 2017).

Usage	Domain	Core object of evaluation
LogicBench	LLM reasoning	Natural-language logical inference
LogicBench	Vision-language modeling	Logical structure in vision-language pairs
LogicBench	Quantum logic synthesis	Qubit/ $T$ -count/runtime trade-offs

This multiplicity is important for interpretation. A reference to “LogicBench” in recent literature does not, by itself, identify a single canonical dataset or metric family.

2. LogicBench for natural-language logical reasoning

In the LLM literature, LogicBench was introduced to address the underassessment of logical reasoning relative to other reasoning skills. The benchmark is a synthetic, natural language QA dataset in which each instance focuses on the use of a single inference rule. It covers 25 reasoning patterns spanning propositional logic, first-order logic, and non-monotonic logic. The propositional component includes Modus Ponens, Modus Tollens, Hypothetical Syllogism, Disjunctive Syllogism, Constructive Dilemma, Destructive Dilemma, Bidirectional Dilemmas, Material Implication, and Commutation. The first-order component extends these patterns with quantifiers and includes Existential Generalization and Universal Instantiation. The non-monotonic component covers default reasoning with irrelevance, default reasoning with open domain, reasoning about exceptions and priorities, and unknown expectations. Two task formats are used per sample: Binary QA, asking whether the conclusion is entailed, and Multiple Choice QA, asking for the correct conclusion among four options (Parmar et al., 2024).

The dataset construction follows a three-stage process. First, diverse ontologies and their negations are generated via GPT-3.5 prompts. Second, templates convert these materials into logical narratives instantiating a target inference rule. Third, context-question-answer triplets are formed for BQA and MCQA, including semantic and negated variants for BQA and distractors for MCQA. The benchmark comprises approximately 1,520 BQA and 500 MCQA unique evaluations, matching the survey characterization of 2,020 instances. Evaluation was conducted in zero-shot chain-of-thought settings without in-context exemplars and with three stylistically different prompt variants, using GPT-4, ChatGPT, Gemini-Pro, Llama-2-7B-Chat, and Mistral-7B-Instruct. The reported result is that existing LLMs do not fare well on LogicBench, especially on complex reasoning and negations, and sometimes overlook contextual information necessary for correct inference. Human accuracy is reported at about 86%, while model performance is substantially lower; for example, GPT-4 achieves an average of 63.98% on propositional-logic BQA for the affirmative label class. The benchmark also highlights polarity effects, with models better at rejecting incorrect conclusions than affirming correct ones (Parmar et al., 2024).

Representative rules are expressed in the paper using standard formal notation, for example Modus Ponens as $((p \rightarrow q)\land p)\rightarrow q$ , Modus Tollens as $((p \rightarrow q)\land \neg q)\rightarrow \neg p$ , Hypothetical Syllogism in first-order form as $(\forall x ((p(x)\rightarrow q(x)) \land (q(x)\rightarrow r(x)))) \rightarrow (p(a) \rightarrow r(a))$ , and Existential Generalization as $P(a) \rightarrow \exists x\ P(x)$ . This design deliberately isolates inference patterns rather than testing broad mixed reasoning.

3. Benchmark ecology, survey treatment, and semantic robustness

Subsequent benchmark surveys place LogicBench within the “Logical” category of reasoning benchmarks and describe it as targeting 25 reasoning patterns across propositional, first-order, and non-monotonic reasoning. In that survey treatment, LogicBench is characterized as a controlled, largely synthetic or model-authored benchmark with automated evaluation via accuracy, and as part of a broader move away from narrow evaluations centered only on modus ponens or modus tollens. The survey also associates LogicBench with the broader challenge of compositional generalization, process faithfulness, and contamination-resistant evaluation in logical reasoning research (Ni et al., 21 Aug 2025).

A later robustness framework, GSM-SEM, applies stochastic, answer-preserving semantic augmentation to LogicBench. For LogicBench, GSM-SEM adapts prompts so that generated questions preserve the multiple-choice structure and the reasoning chain while changing thematic content, entities, attributes, or relationships. The reported out-of-domain experiment uses a 160-sample first-order logic subset. Accuracy remains effectively stable under these semantic perturbations: Llama-4-Scout changes from 92.81% to 92.26%, while Gemini-2.5-Flash changes from 85.00% to 87.62%. The reported performance drop rate is therefore approximately zero. The paper interprets this as evidence that LogicBench, at present, is relatively robust to semantic variation and less affected by contamination than older, more widely circulated benchmarks. A plausible implication is that LogicBench is currently measuring logical reasoning more than test-set memorization, although the same paper argues that live semantic augmentation remains useful as benchmarks age (Singh et al., 8 May 2026).

This later treatment is significant because it distinguishes two properties that are often conflated: logical difficulty and benchmark fragility. LogicBench is presented as difficult for LLMs in the original study, yet relatively stable under answer-preserving semantic rewrites in later robustness analysis.

4. LogicBench for vision-LLMs

In multimodal research, LogicBench denotes a separate benchmark designed to diagnose logical blindspots in vision-LLMs. This LogicBench contains over 50,000 vision-language pairs across 9 logical categories and 4 scenarios: images, videos, anomaly detection, and medical diagnostics. The logical categories are Conjunction, Disjunction, Negation, Contrast, Comparison, Condition, Causality, Temporality, and Inclusion. The scenarios draw from MSCOCO and CC12M for natural images, MSRVTT for videos, DADA-2000 for anomaly videos, and Open-i for radiology. The benchmark supports two tasks: logic-aware retrieval and logical multiple-choice question answering. Positive samples are filtered from human-annotated datasets via syntactic parsing with spaCy and regex, while negative samples are generated by multiple LLMs—Qwen 2.5-max, DeepSeek-V3, Gemini-2.5-pro, GPT-4.1, and LLaMA 3.3—to create fluent but logically corrupted alternatives, followed by human review (Zhou et al., 15 Aug 2025).

The reported evaluation reveals a large gap between human and model performance. Human baselines are approximately 96% on Image MCQ, 94% on Video MCQ, 94% on Anomaly MCQ, and 87% on Medical MCQ. Baseline VLMs are reported at only 40–60% on MCQ tasks. Weaknesses are concentrated in Causality, Temporality, Conditionality, and Negation; for OpenAI CLIP-B/L, the paper reports approximately 20% on Causality, 19–20% on Temporality, 27–32% on Conditionality, and 37–41% on Negation. The benchmark therefore targets failures not reducible to object recognition or lexical matching.

The same paper introduces LogicCLIP, a training framework intended to improve performance on this multimodal LogicBench. LogicCLIP expands MSCOCO into 475,624 training samples, with 25% positives and 75% hard negatives, and optimizes a combined objective consisting of coarse-grained CLIP contrastive loss, a fine-grained multiple-choice loss, and a logical structure-aware loss:

$L_{\mathrm{Total}} = \alpha L_{\mathrm{CLIP}} + \beta L_{\mathrm{MC}} + \gamma L_{\mathrm{Logic}},$

with $\alpha=4$ , $\beta=2$ , and $\gamma=1$ in experiments. On Image MCQ, the paper reports gains such as OpenAI CLIP-B from 38.6% to 81.9%, OpenAI CLIP-L from 36.9% to 83.9%, NegCLIP from 55.3% to 85.7%, and COCO retrieval improvements such as $T$ 0 from 30.4% to 42.5% for CLIP-B. In this context, LogicBench functions both as a diagnostic benchmark and as a training target for logic-aware multimodal representation learning (Zhou et al., 15 Aug 2025).

5. LogicBench in quantum logic synthesis

In quantum computing, LogicBench refers to benchmark circuits and mapping statistics produced within the LUT-based Hierarchical Reversible Logic Synthesis framework. LHRS maps classical combinational logic into quantum circuits in the Clifford+ $T$ 1 gate set by first synthesizing a $T$ 2-LUT network, then mapping LUTs to single-target reversible gates, and finally decomposing those gates into Clifford+ $T$ 3 implementations. The framework explicitly trades off qubit count against $T$ 4-count by varying LUT size $T$ 5 and ancilla management strategies. Within this context, LogicBench is described as the first systematic benchmarking initiative for quantum logic synthesis, analogous to classical logic synthesis benchmark suites, and as a way to report qubits, $T$ 6-count, and runtime for synthesized arithmetic components (Soeken et al., 2017).

The benchmark role is tightly connected to scientific cost estimation. LHRS is reported to synthesize IEEE-compliant floating-point networks up to double precision, including add, multiply, divide, and square root components. The produced LogicBench statistics are intended to let quantum algorithm designers estimate the arithmetic cost that would otherwise remain hidden in high-level algorithm descriptions. The paper emphasizes that automatic, global resynthesis of composed functions can outperform naive composition of separately synthesized modules. For a 32-bit floating-point $T$ 7 circuit, varying $T$ 8 from 6 to 16 yields qubit counts from approximately 4000 to approximately 1700 and $T$ 9-gates from approximately $((p \rightarrow q)\land p)\rightarrow q$ 0 to $((p \rightarrow q)\land p)\rightarrow q$ 1. For a 32-bit Gaussian function $((p \rightarrow q)\land p)\rightarrow q$ 2, direct composition is reported at 6,355 qubits and 8,960,228 $((p \rightarrow q)\land p)\rightarrow q$ 3 gates, whereas global resynthesis reduces this to 6,283 qubits and 1,850,001 $((p \rightarrow q)\land p)\rightarrow q$ 4 gates. Here, LogicBench is not a reasoning benchmark in the NLP sense; it is a resource-estimation benchmark for reversible and quantum circuit synthesis (Soeken et al., 2017).

This use materially broadens the semantic scope of the name. “Logic” refers not to logical inference over propositions, but to synthesized Boolean logic networks and their realization costs on quantum hardware.

The recurring appearance of the name “LogicBench” in later literature shows that these resources are not isolated artifacts. The LLM benchmark is explicitly incorporated into later benchmark surveys and into structured-inference systems. MetaJuLS, for example, evaluates LLM-constrained generation on LogicBench and GSM8K-Constrained by formulating structured inference as adaptive constraint propagation. In the reported LogicBench-constrained setting, MetaJuLS reaches 98.2% constraint satisfaction and 79.4% accuracy at 79 ms per sequence, while the combination of speculative decoding and MetaJuLS reaches 98.5% constraint satisfaction, 79.5% accuracy, and 67 ms per sequence. The same paper reports adaptation in 5–10 gradient steps, or 5–15 seconds, after meta-training on related tasks (Shihab et al., 31 Dec 2025).

At the same time, the name should not be conflated with adjacent but distinct logic-centric resources. ChaosBench-Logic evaluates LLM logical and symbolic reasoning over chaotic dynamical systems using a first-order logic ontology with 11 predicates and 621 questions over 30 systems; its focus is scientific reasoning under an axiom system rather than the single-rule natural-language setting of LogicBench (Thomas, 5 Jan 2026). LogicCBMs, by contrast, are a model family that adds differentiable logic modules to concept bottleneck models; they include synthetic “LogicBench” tasks such as XOR/2XOR and CLEVR-Logic to test concept composition, but these are experimental datasets internal to a neurosymbolic learning paper rather than the standalone LLM or VLM benchmarks that also bear the name (Vemuri et al., 8 Dec 2025).

A common misconception is therefore that LogicBench denotes one benchmark and one metric regime. The literature instead supports a more precise characterization: LogicBench is a reused research label for multiple benchmark constructions whose only broad commonality is a concern with logical structure. In one line of work it tests rule-governed deduction in natural language; in another it measures logical sensitivity in multimodal alignment; in another it tabulates synthesis costs for logic networks realized as quantum circuits. The name is shared, but the underlying tasks, supervision, metrics, and scientific objectives are not.