DivLogicEval: Logic-Centered Benchmark

• DivLogicEval is a benchmark that isolates logical reasoning by using natural sentences to express classical propositional logic with diverse, counterintuitive structures.
• It transforms symbolic logic problems into multiple-choice reading comprehension questions through a two-stage pipeline, ensuring that full multi-proposition reasoning is required.
• The PartialCircular metric improves evaluation stability by combining correctness over cyclic permutations with an entropy-based confidence measure, mitigating bias in LLM assessments.

DivLogicEval is a benchmark for evaluating logical reasoning in LLMs by targeting logical reasoning itself rather than a mixture of logic, commonsense, retrieval, or other latent skills. It is introduced as a classical logic benchmark consisting of natural sentences composed of diverse statements in a counterintuitive way, with the stated goal of combining logic-centeredness with language diversity. The benchmark is paired with a new evaluation metric, PartialCircular, intended to mitigate the influence of bias and randomness inherent in LLMs, and its experiments are used to assess how much logical reasoning is actually required for successful performance (Chung et al., 19 Sep 2025).

1. Definition and evaluative rationale

The central motivation for DivLogicEval is a critique of existing “logical reasoning” benchmarks. The benchmark paper argues that datasets such as ReClor and LogiQA2 are not cleanly logic-centered, because the correct answer can often be inferred using commonsense or pretraining shortcuts rather than formal reasoning. The paper further reports that when GPT-3.5 is explicitly prompted to avoid logical reasoning, performance on ReClor and LogiQA2 can improve rather than degrade, which is presented as evidence that those benchmarks do not isolate logical reasoning cleanly (Chung et al., 19 Sep 2025).

DivLogicEval is designed to satisfy two goals simultaneously: logic-centeredness and language diversity. In the paper’s comparison, ReClor and LogiQA2 are characterized as more diverse but less logic-centered, while RuleTaker, LogicNLI, FOLIO, RobustLR, and PrOntoQA-OOD are characterized as more logic-centered but less diverse. DivLogicEval is intended to be strong on both dimensions. Its underlying conception is that classical propositional logic should be expressed in natural language with varied sentence forms, so that successful performance depends on tracking logical structure rather than exploiting narrow templates or familiar discourse patterns (Chung et al., 19 Sep 2025).

A common source of confusion is the similarity of the name to other logic-related evaluation frameworks. DivLogicEval is a benchmark for logical reasoning evaluation in LLMs, whereas “LogicEval” refers to a framework for evaluating automated repair techniques for logical vulnerabilities in software (Rashid et al., 14 Apr 2026). The two systems address different problem domains despite the lexical overlap.

2. Benchmark design and construction pipeline

DivLogicEval is a multiple-choice machine reading comprehension dataset with four options and exactly one correct answer. Each instance contains three parts: content, question, and options. The construction pipeline has two stages: first, generate a symbolic propositional-logic problem that can be verified by an external logic solver; second, transform the symbolic problem into natural language by replacing logical variables with diverse natural sentences and combining them with counterintuitive connectives (Chung et al., 19 Sep 2025).

The symbolic content is built from propositional variables A–H, sampled in a controlled way. Each instance uses two to three logic atom units sampled iteratively, and the sampling probability of variables decreases when a variable has already been selected many times. The paper specifies three implication-style rules used to assemble the content propositions: $o$8 The appendix gives a probability rule for selecting variable $i$: $o$9 where $o$ is the vector of occurrence counts for the eight variables in one instance, and $i \in (0,8)$. The stated purpose is to reduce repetition and avoid repeated inference shortcuts (Chung et al., 19 Sep 2025).

Options are constructed from all possible pairs of variables with an inference relation, the single atom variables, and their negations. An external logic validator partitions these candidates into the entailment group $e$ and the non-entailment group $n$. To ensure that the instance requires the full multi-proposition context, candidates directly derivable from a single proposition are filtered out from the entailment set. This design choice is meant to suppress trivial local derivations and force reasoning across the entire content (Chung et al., 19 Sep 2025).

3. Question types and natural-language realization

DivLogicEval uses three question types, explicitly inspired by GMAT-style logic questions. In 3c1e, the content fails to imply three options and implies one option. In 3e1c, the content implies three options and fails to imply one option. In Missing Premise, the content is modified so that a necessary proposition is removed; the removed proposition becomes the correct option, and the other three options are drawn from non-entailments. The paper highlights the Missing Premise type because it directly tests whether a model can identify what premise is needed to make a conclusion follow (Chung et al., 19 Sep 2025).

The natural-language realization is based on what the paper calls “diverse statements in a counterintuitive way.” Logical variables are instantiated with simple natural sentences sourced from SNLI and MNLI. The paper also states that GPT-3.5 is used to generate natural-language templates for the three inference rules and to grammar-check the resulting text. Diversity is attributed to four mechanisms: each logic variable is instantiated with a natural sentence from SNLI/MNLI; repeated variables may be instantiated with different sentences, including semantically similar or contradictory ones; GPT-3.5 is used to diversify the templates that connect the statements; and sentence negation is applied to increase variety (Chung et al., 19 Sep 2025).

The “counterintuitive” aspect is defined in the benchmark paper as preserving formal logical validity while making the natural-language surface form unusual or surprising, so that the problem cannot be solved by simple template matching or by relying on familiar discourse patterns. A plausible implication is that DivLogicEval attempts to shift the evaluation burden from lexical familiarity to explicit structure-tracking over propositions.

The dataset organization reported in the paper is summarized below.

Component	Value
Total instances	12,589
3c1e	4,196
3e1c	4,195
Missing Premise	4,198
Polished human-reviewed test set	900
Vocabulary size	6,748

These figures are used in the paper to support the claim that DivLogicEval is substantially more diverse than templated classical-logic datasets while remaining logic-centered (Chung et al., 19 Sep 2025).

4. Evaluation protocol and PartialCircular

The benchmark introduces a new metric, PartialCircular (PC), alongside Accuracy (ACC) and Circular (CIR). Accuracy scores only the original instance. Circular evaluates all cyclic permutations of the options and counts an instance as correct only if all mutants are answered correctly. The paper argues that ACC can vary widely across runs and that CIR can be overly brittle, because slight inconsistency across option orders can cause the entire instance to be marked wrong (Chung et al., 19 Sep 2025).

PartialCircular still evaluates all cyclic mutants, but assigns a partial score based on correctness and confidence concentration. If an instance has four cyclic mutants, $c$ is the number of mutants answered correctly, and $p(\cdot)$ is the empirical frequency distribution of the model’s predicted options over the mutants, then the per-instance score is: $i \in (0,8)$0 The paper interprets this as the fraction of correctly answered mutants multiplied by an entropy-based term that penalizes uncertain, high-entropy option distributions. An example is given in which the predictions are $o_1, o_1, o_3, o_4$ and the ground truth is $o_1$; then $c=2$, $o$0, $o$1, $o$2, and PartialCircular equals 0.125 (Chung et al., 19 Sep 2025).

The appendix also defines a parameterized variant: $i \in (0,8)$1 with $o$3 yielding the original PC, $o$4 reducing to plain accuracy $o$5, and $o$6 giving a hybrid between rewarding correctness and penalizing randomness. The paper states that the original metric ranges between 0 and 1 (Chung et al., 19 Sep 2025).

The empirical motivation for the new metric is stability under repeated runs. For Gemini-Pro over five runs, the paper reports coefficient of variation values of ACC 3.3, CIR 6.3, and PC 3.1, and uses these numbers to argue that PC is more stable than CIR (Chung et al., 19 Sep 2025).

5. Empirical results and diagnostic findings

The evaluated models include Mixtral (mixtral-8x7B-instruct-v0.1), LLaMA 3.3 (llama-3.3-70b-instruct), Qwen 2.5 (qwen-2.5-72b-instruct), Gemini (gemini-1.5-pro), GPT-3.5 (gpt-3.5-turbo), GPT-4 (gpt-4-1106-preview), GPT-4o (gpt-4o-2024-05-13), and o1-preview (o1-preview-2024-09-12), evaluated in zero-shot and 3-shot settings. The prompt format reported in the paper is: “You need to answer in the form of Answer: <A/B/C/D> without explanation” (Chung et al., 19 Sep 2025).

The main reported finding is that o1-preview performs best by a large margin. The paper reports zero-shot overall ACC 51.3, 3-shot overall ACC 60.6, and 3-shot overall PC 51.2 for o1-preview, while also arguing that substantial headroom remains. Among non-o1 models, Qwen 2.5 and LLaMA 3.3 are described as strongest overall, followed by GPT-4o, GPT-4, and Mixtral, with GPT-3.5 weakest. The paper also states that ACC alone makes it difficult to distinguish models except for o1-preview, whereas CIR and PC yield clearer rankings (Chung et al., 19 Sep 2025).

Several model-specific results are explicitly listed. For example, the paper reports GPT-4o 3-shot at ACC 35.8, CIR 16.1, PC 25.6; GPT-4 0-shot at ACC 32.2, CIR 12.3, PC 22.1; GPT-3.5 0-shot at ACC 29.6, CIR 3.7, PC 13.3; LLaMA 3.3 3-shot at ACC 30.6, CIR 17.7, PC 24.9; and Qwen 2.5 0-shot at ACC 33.5, CIR 19.2, PC 26.7 (Chung et al., 19 Sep 2025).

Performance varies by question type. The paper states that o1-preview is especially good at identifying the non-entailing option, GPT-4o and GPT-4 do better on the Missing Premise type, and many models are better at locating entailing options than non-entailing ones. Few-shot prompting helps unevenly: o1-preview improves the most under 3-shot prompting, while other models improve modestly or may decline (Chung et al., 19 Sep 2025).

A particularly important diagnostic experiment is the “NoLR” prompt ablation. GPT-3.5 is instructed: “Please try your best to answer correctly without performing any logical reasoning.” The reported outcome is that NoLR hurts DivLogicEval but improves ReClor and LogiQA2. The paper gives the following DivLogicEval numbers: ACC 32.2 to 28.6, CIR 16.4 to 15.8, and PC 6.3 to 5.2; for ReClor, ACC 57.3 to 59.8; and for LogiQA2, ACC 51.9 to 53.2. This is presented as evidence that DivLogicEval depends more directly on logical reasoning than those alternative benchmarks (Chung et al., 19 Sep 2025).

The paper also introduces a symbolic variant, s-DivLogicEval, and reports that GPT-4-turbo performs substantially better on symbolic expressions than on the natural-language version: DivLogicEval 0-shot ACC 32.2 versus s-DivLogicEval 0-shot ACC 39.1, and DivLogicEval 3-shot ACC 27.2 versus s-DivLogicEval 3-shot ACC 38.1. The intended interpretation is that the natural-language version is difficult because the model must interpret unusual sentence composition rather than because of symbolic-formatting artifacts (Chung et al., 19 Sep 2025).

6. Diversity, distribution, human performance, and limitations

A substantial part of the benchmark’s case rests on diversity and distributional comparison. The paper reports a test-set vocabulary size of 6,748 and compares KL divergence against a Wikipedia subset. Among the classical logic benchmarks listed, DivLogicEval has the lowest KL divergence at 1.87, compared with RuleTaker at 4.29, LogicNLI at 4.62, FOLIO at 2.77, RobustLR at 6.28, and PrOntoQA-OOD at 6.24. ReClor and LogiQA2 have lower KL values, 1.44 and 1.32 respectively, but are not characterized as logic-centered in the same way (Chung et al., 19 Sep 2025).

The paper also presents human-performance and contamination evidence. Four graduate students, pre-screened on symbolic logic ability, solved 60 test samples and achieved 86.7% accuracy, which the paper uses to argue that the benchmark is solvable by humans with logical competence and is not intrinsically ambiguous. Exact-match contamination rate with GPT-3.5 is reported as only 0.2%. In an additional analysis, further pretraining LLaMA2 on SNLI and MNLI improves SNLI by +12.7 but DivLogicEval by only +1.2, which is used to argue that performance cannot be explained primarily by memorization of source corpora (Chung et al., 19 Sep 2025).

The benchmark paper explicitly notes several limitations. DivLogicEval is synthesized from existing benchmarks and source corpora, which may still introduce grammatical issues. GPT-assisted grammar correction and manual review are needed, especially for the test set. The number of logical variables and implication rules is limited in the current construction. The pipeline is described as extensible to larger $o$7, more variables, and more rules, but the paper states that additional grammar checking and human review would then be needed (Chung et al., 19 Sep 2025).

Taken together, these features position DivLogicEval as a benchmark whose distinctive claim is not merely hardness, but evaluative specificity. It is meant to be hard because the instances are logic-centered and linguistically varied, not because they are obscure or underdetermined. This suggests that DivLogicEval should be read less as a general reasoning leaderboard and more as an instrument for probing whether natural-language logical structure is being processed as logic rather than approximated through superficial cues.