---
title: Logical-CommonSenseQA Benchmark
url: https://www.emergentmind.com/topics/logical-commonsenseqa
type: topic
---

# Logical-CommonSenseQA Benchmark

LOGICAL-COMMONSENSEQA is a benchmark that recasts commonsense question answering as a compositional plausibility judgment over logical pairs of atomic statements, systematically probing the ability of large language models (LLMs) to perform logical reasoning beyond surface-level plausibility estimation. Unlike classic benchmarks that rely on single-label or binary (yes/no) evaluation, LOGICAL-COMMONSENSEQA requires models to determine how two independent, human-validated propositions compose under explicit logical operators: AND (conjunction), OR (disjunction), and NEITHER/NOR (joint negation), with plausibility defined quantitatively.

## 1. Formal Task Definition and Logical Operators

LOGICAL-COMMONSENSEQA is structured around pairs of atomic statements $S_1$, $S_2$, each representing an independent, plausible or implausible commonsense proposition. The core technical object is the plausibility function $P(\cdot)$, mapping each atomic statement $S$ to a normalized score in $[0,1]$ encoding the degree to which it aligns with commonsense world knowledge.

Plausibility composition is governed by three logical operators, each with explicit semantics:

- **Conjunction (AND):** 
  $$P(S_1 \,\mathbf{AND}\, S_2) = \min(P(S_1), P(S_2))$$
  Both $S_1$ and $S_2$ must be independently plausible; joint plausibility is capped by the less typical statement.

- **Disjunction (OR):** 
  $$P(S_1 \,\mathbf{OR}\, S_2) = \max(P(S_1), P(S_2))$$
  Partial plausibility holds if either $S_1$ or $S_2$ is plausible.

- **Negation (NEITHER/NOR):**
  $$P(\mathbf{NEITHER}\;S_1\;\mathbf{NOR}\;S_2) = 1 - \max(P(S_1), P(S_2))$$
  Joint implausibility requires both statements to lack plausibility.

Each instance presents a multiple-choice question with four labeled options (A–D), each an unordered pair $(S_1, S_2)$ under a specified operator. The benchmark preserves the original CommonsenseQA multiple-choice format but reinterprets selection as an operator-conditioned joint inference task [2601.16504].

## 2. Dataset Construction and Validation Methodology

LOGICAL-COMMONSENSEQA is derived by automated extension and transformation of the CommonsenseQA dataset:

- **Candidate Generation:** 5,000 CommonsenseQA items are selected. GPT-4o-mini is used to generate 4–6 diverse atomic answer candidates per question, mixing both plausible and implausible alternatives.
- **Refinement and Pruning:** Logically incoherent or trivial options are filtered out. Three plausible and four implausible candidate statements per question are retained, ensuring multi-step reasoning is required.
- **Deterministic Logical Pairing:** All unordered pairs of refined atomic options are formed and programmatically labeled with one of the three operators (AND, OR, NEITHER/NOR), with an additional MIXED condition—per-instance random assignment of operators.

The benchmark totals 19,996 instances, stratified equally among operators:

| Operator         | Instances | Train   | Dev   | Test   |
|------------------|-----------|---------|-------|--------|
| AND              | 4,999     | 2,999   | 1,500 | 500    |
| OR               | 4,999     | 2,999   | 1,500 | 500    |
| NEITHER/NOR      | 4,999     | 2,999   | 1,500 | 500    |
| MIXED            | 4,999     | 2,999   | 1,500 | 500    |

Human validation is performed on 5% of Stage 2 items (250 questions) using an “awareness–consensus” protocol: each atomic option is separately judged for personal belief and perceived social agreement, with adjudication of annotator disagreement. Accuracy of assigned labels on the human-validated subset:

- **AND:** 89.2%
- **OR:** 96.4%
- **NEITHER/NOR:** 73.6%
- **MIXED:** 88.4%

Inter-annotator agreement is moderate (Cohen’s κ = 0.49) [2601.16504].

## 3. Benchmark Instances and Plausibility Semantics

The benchmark instances exemplify logical composition over human-centric commonsense scenarios:

- **Conjunctive Example (AND):**
  - Q: “Sammy wanted to go where the people were. Where might he go?”
  - A: “local events AND social venues” (both independently plausible)

- **Disjunctive Example (OR):**
  - Q: “Where might Sammy go if he just wanted a public space?”
  - A: “local events OR empty parks” (at least one plausible)

- **Negation Example (NEITHER/NOR):**
  - Q: “Sammy wanted a quiet spot away from crowds. Which is not plausible?”
  - A: “NEITHER quiet retreats NOR empty parks” (both jointly implausible)

The formal semantics enforce operator-conditional selection based on joint plausibility $P(S_1 ⊕ S_2)$ matching the targeted constraint.

## 4. Evaluation Paradigms and Empirical Findings

Model evaluations probe both zero-shot and targeted adaptation settings:

- **Zero-Shot Prompting:** Decoder-only models (LLaMA-3.3-70B, LLaMA-3.1-8B, Qwen2.5-7B) are given only the question and composite options.
- **Few-Shot Prompting:** Same LLMs, with 1–3 in-context demonstrations instantiated for the operator under test.
- **Chain-of-Thought (CoT) Prompting:** For LLaMA-3.1-8B, includes explicit reasoning steps.
- **Supervised Fine-Tuning:** Encoder-only (DeBERTa-v3-base) and encoder–decoder (Flan-T5-base, Entailer-11B) models trained on the complete logical dataset.

All decoding uses temperature 0.0, top-p = 0.9; outputs must be one of {A, B, C, D}.

Macro-F1 scores on the human-validated test subset (zero-shot):

| Model           | AND   | OR    | NEITHER/NOR | MIXED |
|-----------------|-------|-------|-------------|-------|
| LLaMA-3.3-70B   | 80.9  | 70.9  | 13.4        | 53.0  |
| LLaMA-3.1-8B    | 71.9  | 62.2  | 13.1        | 41.8  |
| Qwen2.5-7B      | 79.6  | 68.9  | 12.9        | 53.2  |

With supervised fine-tuning:

| Model             | AND   | OR    | NEITHER/NOR | MIXED |
|-------------------|-------|-------|-------------|-------|
| Flan-T5-base      | 92.8  | 92.4  | 89.2        | 89.6  |
| DeBERTa-v3-base   | 87.6  | 87.2  | 84.8        | 82.4  |

A pronounced asymmetry is observed: LLMs perform well on conjunction and moderately on disjunction, but exhibit sharp degradation on negation (NEITHER/NOR)—even with chain-of-thought prompting. Supervised models close the gap across operators but error analysis reveals persistent single-statement dominance and systematic confusion in negation scenarios [2601.16504].

## 5. Error Analysis and Reasoning Pathologies

Error clusters illustrate fundamental model limitations:

- **Single-Statement Dominance (AND/OR):** The model frequently selects the pair containing the most plausible atomic statement, disregarding required compositionality.
- **Negation Inversion (NEITHER/NOR):** Under joint negation, models incorrectly pick pairs containing plausible atoms, failing to internalize the operator’s semantics—consistent across prompting regimes.
- **Intermediate MIXED Difficulty:** When operators are assigned randomly (MIXED condition), performance drops but remains above pure negation, which collapses reasoning quality.

These error patterns highlight the difference between surface plausibility modeling and operator-grounded logical composition.

## 6. Benchmark Context and Connections

LOGICAL-COMMONSENSEQA builds on the classic CommonsenseQA single-label framework but diverges through its joint, operator-conditioned inference task. In contrast to binary yes/no QA formats as exemplified in CommonsenseQA 2.0 [2201.05320], which probe logical skills but lack multi-way compositionality, LOGICAL-COMMONSENSEQA formalizes and quantifies joint plausibility. Benchmark connection points include:

- **KEAR’s external attention paradigm [2112.03254]:** Incorporates external knowledge in standard QA including CommonsenseQA, but is not designed for compositional operator inference.
- **COM² [2403.07398]:** Demonstrates training on multi-hop logical queries sampled from knowledge graphs using conjunction, intersection, negation, and projection, with structured natural language verbalization. This approach suggests that extending LOGICAL-COMMONSENSEQA to richer operator sets (e.g. union, set-difference, temporal, multi-hop chains) is tractable via knowledge graph sampling and LLM verbalization.
- **Contrast to SCoRE [2503.06218, not available]:** LOGICAL-COMMONSENSEQA emphasizes plausibility-level logical composition rather than multi-hop scenario chains or long-chain reasoning.

## 7. Implications and Prospective Extensions

LOGICAL-COMMONSENSEQA foregrounds a critical limitation in contemporary LLMs: strong performance in conjunctive/disjunctive settings is driven by pattern-matching over commonsense priors, while true logical composition, especially involving negation, is not reliably internalized. The benchmark's operator-specific performance stratification clarifies this gap at scale.

Recommended technical extensions include:

- Expansion of the operator set to exclusive OR, implication, temporal/causal logic, and universal/existential quantification.
- Integration of explicit operator-centric supervision and auxiliary losses targeting operator semantics.
- Neural-symbolic architectures maintaining operator-specific representations.
- Enabling generative composition: requiring models to generate new atomic statements and compose them logically.

A plausible implication is that training protocols leveraging multi-hop, multi-operator data sampled from structured knowledge graphs—as advocated in COM² [2403.07398]—could address compositional deficits observed in LOGICAL-COMMONSENSEQA.

LOGICAL-COMMONSENSEQA thus provides a controlled, validated framework for joint-plausibility evaluation, serving both as an empirical diagnostic for model compositionality and a template for advancing the logical expressiveness of future commonsense QA systems [2601.16504].

Source: https://www.emergentmind.com/topics/logical-commonsenseqa