NeuBAROCO: Bilingual Logical Bias Benchmark

• NeuBAROCO is a bilingual benchmark that evaluates deductive reasoning and cognitive biases in syllogistic tasks across English and Japanese.
• It operationalizes classical categorical syllogisms into both NLI and multiple‐choice formats to assess entailment, contradiction, and neutral inferences.
• Evaluation with models like GPT-3.5 and GPT-4 reveals structured weaknesses on bias-sensitive cases, highlighting differences between human and model reasoning.

NeuBAROCO is a bilingual benchmark for evaluating logical reasoning in natural language, with a particular emphasis on whether LLMs exhibit the same systematic error patterns documented in human syllogistic reasoning. It was introduced as a dataset derived from BAROCO, a Japanese collection of logical-inference questions originally designed for psychological experiments, and it was later analyzed in an expanded form that combined multiple-choice and natural language inference formulations in English and Japanese (Ando et al., 2023, Ozeki et al., 2024). Across these studies, NeuBAROCO is defined less as a generic logic dataset than as a cognitively grounded evaluation framework: it tests deductive validity, but it also operationalizes belief bias, conversion errors, and atmosphere effects, thereby linking LLM evaluation to the psychology of human deduction.

1. Origins and research motivation

NeuBAROCO originates in BAROCO, a Japanese syllogism problem set created for psychological and behavioral-genetic studies of human reasoning. The benchmark inherits that experimental lineage directly: the original questions were designed to assess human logical abilities, including belief-bias effects, and NeuBAROCO reformulates them for contemporary NLP and LLM evaluation (Ando et al., 2023).

The original motivation was twofold. First, NeuBAROCO was created to measure whether LLMs can carry out syllogistic reasoning, a canonical form of deductive inference with a long history in logic, psychology, and cognitive science. Second, it was intended to test whether model errors resemble human reasoning biases rather than arising as unstructured failure. This positioning distinguishes NeuBAROCO from prior synthetic logical benchmarks that emphasize validity alone. The benchmark instead preserves human-origin questions, adds bias-sensitive annotations, and supports cross-lingual evaluation in English and Japanese (Ozeki et al., 2024).

The 2024 study makes the cognitive-science orientation explicit by treating syllogisms as formally simple but psychologically rich. They can be represented in monadic predicate logic, yet decades of work show that humans frequently misreason on them in characteristic ways. NeuBAROCO therefore functions simultaneously as a logic benchmark and as a probe of cognitively meaningful failure modes (Ozeki et al., 2024).

2. Formal task design and dataset organization

NeuBAROCO is built around standard categorical syllogistic forms. The benchmark uses the four classical sentence types:

$A:\ \text{\sf All } S \text{\sf\ are } P$

$E:\ \text{\sf No } S \text{\sf\ are } P$

$I:\ \text{\sf Some } S \text{\sf\ are } P$

$O:\ \text{\sf Some } S \text{\sf\ are not } P$

The 2024 paper further gives their logical readings as:

• $A$: $\forall x (S(x) \to P(x))$, equivalently $S \subseteq P$
• $E$: $\forall x (S(x) \to \neg P(x))$, equivalently $S \cap P = \emptyset$
• $E:\ \text{\sf No } S \text{\sf\ are } P$0: $E:\ \text{\sf No } S \text{\sf\ are } P$1, equivalently $E:\ \text{\sf No } S \text{\sf\ are } P$2
• $E:\ \text{\sf No } S \text{\sf\ are } P$3: $E:\ \text{\sf No } S \text{\sf\ are } P$4, equivalently $E:\ \text{\sf No } S \text{\sf\ are } P$5 (Ozeki et al., 2024)

A syllogism consists of two premises and a conclusion or hypothesis. In the NLI formulation, each instance contains two premises, one conclusion, and one label: entailment, contradiction, or neutral. The benchmark also includes a multiple-choice formulation closer to the original BAROCO experiments, where two premises are paired with five candidate conclusions corresponding to the four categorical forms plus a “none of them” option (Ando et al., 2023, Ozeki et al., 2024).

The published papers describe two benchmark states: an earlier NLI-focused introduction and an expanded later form.

Version described in the literature	Task format	Reported size
Initial NeuBAROCO formulation (Ando et al., 2023)	NLI-style inference problems	375 inference problems
Expanded NeuBAROCO form (Ozeki et al., 2024)	95 multiple-choice problems; 790 NLI problems	95 MC, 790 NLI

For the initial 375-problem version, the reported NLI label distribution is 122 entailment, 71 contradiction, and 182 neutral. That version also distinguishes 318 basic syllogisms from 57 extended syllogisms, where the extended cases include Boolean inferences with conjunction or disjunction inside terms and hypothetical syllogisms with conditionals (Ando et al., 2023).

For the expanded 790-instance NLI form, the reported label distribution is 254 entailment, 188 contradiction, and 348 neutral. The multiple-choice side comprises 80 main problems plus 15 additional examples and practice problems. The paper notes a structural property of the released multiple-choice set: the correct answer is always an entailed conclusion rather than “none of them,” which makes the task easier than three-way NLI classification (Ozeki et al., 2024).

NeuBAROCO is bilingual. BAROCO was originally Japanese; every problem was translated into English using DeepL and then manually checked and edited so that the English matched categorical-sentence patterns. The English quantifiers were also normalized: all or every for A-type, no for E-type, and some, a certain, or one of for existential I/O forms. The authors deliberately avoided a/an where it could invite a generic reading (Ando et al., 2023).

3. Bias annotations and reasoning phenomena

A central property of NeuBAROCO is that it is not labeled only by truth relation. It also marks problems by psychologically motivated sources of difficulty.

The first annotation family concerns belief bias. In the 2023 paper, content is categorized as symbolic, consistent, inconsistent, or others; in the 2024 expanded study, the corresponding terminology is symbolic, congruent, incongruent, and others (Ando et al., 2023, Ozeki et al., 2024). In the earlier version, the counts are 95 symbolic, 167 consistent, 102 inconsistent, and 11 others. In the expanded version, the counts are 98 symbolic, 404 congruent, 238 incongruent, and 50 others. The governing idea is unchanged: symbolic items use abstract terms such as A, B, C; congruent or consistent items align with commonsense belief; incongruent or inconsistent items contain premises or conclusions that conflict with common sense.

The canonical belief-inconsistent entailment example is:

• Some animals are human beings.
• All animals are tomatoes.
• Some humans are tomatoes.

This is logically valid but semantically bizarre, and it is used to probe whether models reject valid inferences because the content is unbelievable (Ando et al., 2023).

The second annotation family concerns conversion errors. NeuBAROCO operationalizes classic illicit conversions such as treating All A are B as All B are A, or Some A are not B as Some B are not A. The 2023 paper reports 70 conversion-labeled problems, after expanding the original BAROCO stock of such cases; the 2024 expanded study reports 66 conversion cases (Ando et al., 2023, Ozeki et al., 2024). These cases are typically gold-labeled neutral, but they can appear entailing if one reverses a premise improperly.

The third annotation family concerns atmosphere effects, a heuristic bias in which the polarity and quantity of the premises bias the expected form of the conclusion. The benchmark reproduces two classical principles: if one or both premises are negative, the conclusion tends to be expected as negative; if one or both premises are particular, the conclusion tends to be expected as particular. The 2023 paper reports 104 atmosphere-labeled problems, whereas the expanded 2024 version reports 345 atmosphere cases (Ando et al., 2023, Ozeki et al., 2024).

The 2023 paper also notes that BAROCO annotation assumes existential import for universal statements. This is a technically important detail because it affects validity judgments in categorical syllogistics (Ando et al., 2023).

Taken together, these annotations make NeuBAROCO a benchmark of both logical form and bias susceptibility. This suggests that the dataset is intended to distinguish between failure to compute entailment relations and failure induced by content plausibility, illicit quantifier reversal, or surface quantifier “mood.”

4. Empirical findings on LLM reasoning

The initial NeuBAROCO evaluation tested RoBERTa (roberta-large-mnli), BART (facebook/bart-large-mnli), and GPT-3.5 (gpt-3.5-turbo), with Japanese results reported for GPT-3.5. Overall English accuracies were 34.67 for RoBERTa, 35.20 for BART, and 51.73 for GPT-3.5; Japanese GPT-3.5 reached 48.27 (Ando et al., 2023).

The most striking label-level result in that study is the weakness on neutral examples. RoBERTa and BART are near-collapse on neutral English cases, at 0.55 and 2.75, respectively. GPT-3.5 performs much better on entailment than on contradiction or neutral: in English, 79.51 on entailment, 38.03 on contradiction, and 38.46 on neutral; in Japanese, 80.33, 54.93, and 24.18 (Ando et al., 2023). The benchmark therefore exposed a systematic tendency to handle definite support more reliably than underdetermination.

The expanded 2024 study added GPT-4, compared zero-shot and 3-shot prompting, and evaluated both multiple-choice and 790-problem NLI. On the 80-problem multiple-choice task, English accuracies were 53.75 for GPT-3.5 and 83.75 for GPT-4, while Japanese accuracies were 42.50 and 95.00, respectively. The paper also reports a Human Japanese multiple-choice score of 53.00 from the BAROCO experiment with 440 participants, but explicitly cautions against overinterpreting this comparison because human-model comparison methodology is unresolved and the multiple-choice format structurally favors entailment identification (Ozeki et al., 2024).

The NLI results are more diagnostic. In English zero-shot NLI, GPT-3.5 achieved 49.75 overall and GPT-4 71.77; the corresponding Japanese figures were 40.00 and 70.38. Few-shot prompting improved GPT-4 to 77.47 in English and 78.61 in Japanese. Yet even GPT-4 remained far weaker on neutral than on entailment or contradiction. In English zero-shot, GPT-4 scored 85.04 on entailment, 93.62 on contradiction, and 50.29 on neutral; in Japanese zero-shot, the corresponding numbers were 87.40, 95.74, and 44.25. Few-shot prompting improved neutral to 61.78 in English and 63.51 in Japanese, but neutral remained the hardest label (Ozeki et al., 2024).

Bias-specific subsets exhibit the expected degradations. In the 2023 results, every evaluated model performed worst on inconsistent content relative to symbolic or consistent content. In the 2024 expanded results, incongruent items are harder than congruent or symbolic items, and conversion and atmosphere cases are much harder than the overall average. For example, English zero-shot GPT-4 scored 61.76 on incongruent items, 40.91 on conversion, and 50.99 on atmosphere, against 71.77 overall; Japanese zero-shot GPT-4 scored 60.92, 43.94, and 38.12, against 70.38 overall (Ozeki et al., 2024).

These results were interpreted as evidence that model failures are structured rather than random. A plausible implication is that NeuBAROCO is most informative not on easy entailments, but on underdetermined or bias-sensitive cases where commonsense belief, asymmetrical quantification, or premise polarity can distort formal reasoning.

5. Cross-lingual and diagnostic analyses

NeuBAROCO’s bilingual design is integral rather than incidental. The Japanese source was translated into English to form a parallel corpus, allowing the same problem structures to be evaluated across a typologically different language pair. The 2023 paper states that Japanese GPT-3.5 showed “strikingly the same tendency” as English GPT-3.5: strong entailment performance, lower contradiction and neutral performance, and poor results on belief-inconsistent, conversion, and atmosphere subsets, with Japanese doing better on contradiction but worse on atmosphere (Ando et al., 2023).

The 2024 study added a diagnostic Translate-and-Explain task intended to separate semantic interpretation from downstream reasoning. It is a 1-shot Chain-of-Thought prompt with three stages: Translation, Reasoning, and Answer. The authors compared three settings: Explanation only, Predicate logic translation + explanation, and Set theory translation + explanation (Ozeki et al., 2024).

On the 90-problem Translate-and-Explain task, translation accuracy was often much higher than final answer accuracy. In English, GPT-4 reached 96.67 on predicate-logic translation and 92.22 on set-theory translation, while its corresponding final answer accuracies were 83.33 and 80.00. In Japanese, GPT-4 reached 95.56 on predicate-logic translation and 76.66 on set-theory translation, with answer accuracies of 75.56 and 74.44. GPT-3.5 showed the same qualitative pattern, though with substantially weaker Japanese translation performance, especially for set theory (Ozeki et al., 2024).

The authors use this gap to argue that the main bottleneck is often the reasoning process rather than mere sentence interpretation. That conclusion is supported by the error analysis. Reported failures include treating “All animals are tomatoes” as set identity rather than subset, mis-scoping negation in “A certain police officer is not a public servant,” confusing existential conjunction with implication in Japanese “Some A are B,” and confusing O-type and E-type in set-theoretic translation (Ozeki et al., 2024).

This diagnostic layer is one of NeuBAROCO’s more distinctive contributions. It shifts the benchmark from simple input-output scoring toward process-level analysis, while still preserving the underlying syllogistic structure.

6. Benchmark landscape, comparative positioning, and later extensions

NeuBAROCO has become a reference point in later work on Japanese reasoning benchmarks. The 2025 paper introducing BIS Reasoning 1.0 explicitly presents NeuBAROCO as the closest predecessor for Japanese syllogistic belief-bias evaluation, but also argues that it is not a dedicated benchmark for belief-inconsistent reasoning at scale. That paper states that NeuBAROCO’s Japanese subset contains fewer than 800 examples for the NLI task and under 100 for the multiple-choice format, and that it “does not exclusively target belief-inconsistent reasoning.” In experiments, BIS reports complementary evaluation on over 300 belief-inconsistent syllogistic reasoning samples from the NeuBAROCO benchmark (Nguyen et al., 8 Jun 2025).

The BIS comparison is informative because it frames NeuBAROCO as broader in bias coverage but less concentrated on the specific case of logically valid conclusions that conflict with common belief. On the NeuBAROCO belief-inconsistent subset used there, GPT-4o reached 94.01, whereas on BIS it reached 79.54; the same table reports large discrepancies for Claude models, which performed much better on NeuBAROCO than on BIS. The BIS authors interpret this as evidence that task formulation and dataset characteristics strongly affect conclusions about model robustness (Nguyen et al., 8 Jun 2025).

A later development extends the name NeuBAROCO beyond classical syllogistics. The 2025 paper “Normative Reasoning in LLMs: A Comparative Benchmark from Logical and Modal Perspectives” describes a benchmark positioned as an extension of prior NeuBAROCO work on syllogistic reasoning and human-like reasoning biases. In that setting, the name is repurposed for a comparative framework on deontic and epistemic modal reasoning, with both single-premise deontic-logic tasks and multi-premise syllogistic tasks (Ozeki et al., 30 Oct 2025). This indicates a conceptual continuity—cognitively informed logical benchmarking with explicit attention to bias and formal patterning—even though the target phenomena broaden from categorical syllogisms to modal reasoning.

In the benchmark landscape, NeuBAROCO is therefore best understood as an influential bridge resource. It links psychological experiments on human reasoning to bilingual LLM evaluation; it exposes not only logical weakness but structured, human-like bias; and it has served both as a precursor to more focused Japanese belief-inconsistency benchmarks and as a foundation for later benchmark expansion into modal normative reasoning (Ozeki et al., 2024, Nguyen et al., 8 Jun 2025, Ozeki et al., 30 Oct 2025).