---
title: 'NeuBAROCO: Bilingual Logical Bias Benchmark'
url: https://www.emergentmind.com/topics/neubaroco
type: topic
---

# NeuBAROCO: Bilingual Logical Bias Benchmark

NeuBAROCO is a bilingual benchmark for evaluating logical reasoning in natural language, with a particular emphasis on whether large language models exhibit the same systematic error patterns documented in human syllogistic reasoning. It was introduced as a dataset derived from BAROCO, a Japanese collection of logical-inference questions originally designed for psychological experiments, and it was later analyzed in an expanded form that combined multiple-choice and natural language inference formulations in English and Japanese [2306.12567][2408.04403]. Across these studies, NeuBAROCO is defined less as a generic logic dataset than as a cognitively grounded evaluation framework: it tests deductive validity, but it also operationalizes belief bias, conversion errors, and atmosphere effects, thereby linking LLM evaluation to the psychology of human deduction.

## 1. Origins and research motivation

NeuBAROCO originates in BAROCO, a Japanese syllogism problem set created for psychological and behavioral-genetic studies of human reasoning. The benchmark inherits that experimental lineage directly: the original questions were designed to assess human logical abilities, including belief-bias effects, and NeuBAROCO reformulates them for contemporary NLP and LLM evaluation [2306.12567].

The original motivation was twofold. First, NeuBAROCO was created to measure whether LLMs can carry out syllogistic reasoning, a canonical form of deductive inference with a long history in logic, psychology, and cognitive science. Second, it was intended to test whether model errors resemble human reasoning biases rather than arising as unstructured failure. This positioning distinguishes NeuBAROCO from prior synthetic logical benchmarks that emphasize validity alone. The benchmark instead preserves human-origin questions, adds bias-sensitive annotations, and supports cross-lingual evaluation in English and Japanese [2408.04403].

The 2024 study makes the cognitive-science orientation explicit by treating syllogisms as formally simple but psychologically rich. They can be represented in monadic predicate logic, yet decades of work show that humans frequently misreason on them in characteristic ways. NeuBAROCO therefore functions simultaneously as a logic benchmark and as a probe of cognitively meaningful failure modes [2408.04403].

## 2. Formal task design and dataset organization

NeuBAROCO is built around standard categorical syllogistic forms. The benchmark uses the four classical sentence types:

$$
A:\ \text{\sf All } S \text{\sf\ are } P
$$

$$
E:\ \text{\sf No } S \text{\sf\ are } P
$$

$$
I:\ \text{\sf Some } S \text{\sf\ are } P
$$

$$
O:\ \text{\sf Some } S \text{\sf\ are not } P
$$

The 2024 paper further gives their logical readings as:

- $A$: $\forall x (S(x) \to P(x))$, equivalently $S \subseteq P$
- $E$: $\forall x (S(x) \to \neg P(x))$, equivalently $S \cap P = \emptyset$
- $I$: $\exists x (S(x) \wedge P(x))$, equivalently $S \cap P \neq \emptyset$
- $O$: $\exists x (S(x) \wedge \neg P(x))$, equivalently $S \setminus P \neq \emptyset$ [2408.04403]

A syllogism consists of two premises and a conclusion or hypothesis. In the NLI formulation, each instance contains two premises, one conclusion, and one label: **entailment**, **contradiction**, or **neutral**. The benchmark also includes a multiple-choice formulation closer to the original BAROCO experiments, where two premises are paired with five candidate conclusions corresponding to the four categorical forms plus a “none of them” option [2306.12567][2408.04403].

The published papers describe two benchmark states: an earlier NLI-focused introduction and an expanded later form.

| Version described in the literature | Task format | Reported size |
|---|---|---|
| Initial NeuBAROCO formulation [2306.12567] | NLI-style inference problems | 375 inference problems |
| Expanded NeuBAROCO form [2408.04403] | 95 multiple-choice problems; 790 NLI problems | 95 MC, 790 NLI |

For the initial 375-problem version, the reported NLI label distribution is **122 entailment**, **71 contradiction**, and **182 neutral**. That version also distinguishes **318 basic syllogisms** from **57 extended syllogisms**, where the extended cases include Boolean inferences with conjunction or disjunction inside terms and hypothetical syllogisms with conditionals [2306.12567].

For the expanded 790-instance NLI form, the reported label distribution is **254 entailment**, **188 contradiction**, and **348 neutral**. The multiple-choice side comprises **80 main problems plus 15 additional examples and practice problems**. The paper notes a structural property of the released multiple-choice set: the correct answer is always an entailed conclusion rather than “none of them,” which makes the task easier than three-way NLI classification [2408.04403].

NeuBAROCO is bilingual. BAROCO was originally Japanese; every problem was translated into English using **DeepL** and then manually checked and edited so that the English matched categorical-sentence patterns. The English quantifiers were also normalized: **all** or **every** for A-type, **no** for E-type, and **some**, **a certain**, or **one of** for existential I/O forms. The authors deliberately avoided **a/an** where it could invite a generic reading [2306.12567].

## 3. Bias annotations and reasoning phenomena

A central property of NeuBAROCO is that it is not labeled only by truth relation. It also marks problems by psychologically motivated sources of difficulty.

The first annotation family concerns **belief bias**. In the 2023 paper, content is categorized as **symbolic**, **consistent**, **inconsistent**, or **others**; in the 2024 expanded study, the corresponding terminology is **symbolic**, **congruent**, **incongruent**, and **others** [2306.12567][2408.04403]. In the earlier version, the counts are **95 symbolic**, **167 consistent**, **102 inconsistent**, and **11 others**. In the expanded version, the counts are **98 symbolic**, **404 congruent**, **238 incongruent**, and **50 others**. The governing idea is unchanged: symbolic items use abstract terms such as A, B, C; congruent or consistent items align with commonsense belief; incongruent or inconsistent items contain premises or conclusions that conflict with common sense.

The canonical belief-inconsistent entailment example is:

- Some animals are human beings.
- All animals are tomatoes.
- Some humans are tomatoes.

This is logically valid but semantically bizarre, and it is used to probe whether models reject valid inferences because the content is unbelievable [2306.12567].

The second annotation family concerns **conversion errors**. NeuBAROCO operationalizes classic illicit conversions such as treating **All A are B** as **All B are A**, or **Some A are not B** as **Some B are not A**. The 2023 paper reports **70 conversion-labeled problems**, after expanding the original BAROCO stock of such cases; the 2024 expanded study reports **66 conversion cases** [2306.12567][2408.04403]. These cases are typically gold-labeled **neutral**, but they can appear entailing if one reverses a premise improperly.

The third annotation family concerns **atmosphere effects**, a heuristic bias in which the polarity and quantity of the premises bias the expected form of the conclusion. The benchmark reproduces two classical principles: if one or both premises are negative, the conclusion tends to be expected as negative; if one or both premises are particular, the conclusion tends to be expected as particular. The 2023 paper reports **104 atmosphere-labeled problems**, whereas the expanded 2024 version reports **345 atmosphere cases** [2306.12567][2408.04403].

The 2023 paper also notes that BAROCO annotation assumes **existential import** for universal statements. This is a technically important detail because it affects validity judgments in categorical syllogistics [2306.12567].

Taken together, these annotations make NeuBAROCO a benchmark of both logical form and bias susceptibility. This suggests that the dataset is intended to distinguish between failure to compute entailment relations and failure induced by content plausibility, illicit quantifier reversal, or surface quantifier “mood.”

## 4. Empirical findings on LLM reasoning

The initial NeuBAROCO evaluation tested **RoBERTa** (`roberta-large-mnli`), **BART** (`facebook/bart-large-mnli`), and **GPT-3.5** (`gpt-3.5-turbo`), with Japanese results reported for GPT-3.5. Overall English accuracies were **34.67** for RoBERTa, **35.20** for BART, and **51.73** for GPT-3.5; Japanese GPT-3.5 reached **48.27** [2306.12567].

The most striking label-level result in that study is the weakness on **neutral** examples. RoBERTa and BART are near-collapse on neutral English cases, at **0.55** and **2.75**, respectively. GPT-3.5 performs much better on entailment than on contradiction or neutral: in English, **79.51** on entailment, **38.03** on contradiction, and **38.46** on neutral; in Japanese, **80.33**, **54.93**, and **24.18** [2306.12567]. The benchmark therefore exposed a systematic tendency to handle definite support more reliably than underdetermination.

The expanded 2024 study added **GPT-4**, compared zero-shot and 3-shot prompting, and evaluated both multiple-choice and 790-problem NLI. On the 80-problem multiple-choice task, English accuracies were **53.75** for GPT-3.5 and **83.75** for GPT-4, while Japanese accuracies were **42.50** and **95.00**, respectively. The paper also reports a **Human** Japanese multiple-choice score of **53.00** from the BAROCO experiment with **440 participants**, but explicitly cautions against overinterpreting this comparison because human-model comparison methodology is unresolved and the multiple-choice format structurally favors entailment identification [2408.04403].

The NLI results are more diagnostic. In English zero-shot NLI, GPT-3.5 achieved **49.75** overall and GPT-4 **71.77**; the corresponding Japanese figures were **40.00** and **70.38**. Few-shot prompting improved GPT-4 to **77.47** in English and **78.61** in Japanese. Yet even GPT-4 remained far weaker on neutral than on entailment or contradiction. In English zero-shot, GPT-4 scored **85.04** on entailment, **93.62** on contradiction, and **50.29** on neutral; in Japanese zero-shot, the corresponding numbers were **87.40**, **95.74**, and **44.25**. Few-shot prompting improved neutral to **61.78** in English and **63.51** in Japanese, but neutral remained the hardest label [2408.04403].

Bias-specific subsets exhibit the expected degradations. In the 2023 results, every evaluated model performed worst on **inconsistent** content relative to symbolic or consistent content. In the 2024 expanded results, **incongruent** items are harder than **congruent** or **symbolic** items, and **conversion** and **atmosphere** cases are much harder than the overall average. For example, English zero-shot GPT-4 scored **61.76** on incongruent items, **40.91** on conversion, and **50.99** on atmosphere, against **71.77** overall; Japanese zero-shot GPT-4 scored **60.92**, **43.94**, and **38.12**, against **70.38** overall [2408.04403].

These results were interpreted as evidence that model failures are structured rather than random. A plausible implication is that NeuBAROCO is most informative not on easy entailments, but on underdetermined or bias-sensitive cases where commonsense belief, asymmetrical quantification, or premise polarity can distort formal reasoning.

## 5. Cross-lingual and diagnostic analyses

NeuBAROCO’s bilingual design is integral rather than incidental. The Japanese source was translated into English to form a parallel corpus, allowing the same problem structures to be evaluated across a typologically different language pair. The 2023 paper states that Japanese GPT-3.5 showed “strikingly the same tendency” as English GPT-3.5: strong entailment performance, lower contradiction and neutral performance, and poor results on belief-inconsistent, conversion, and atmosphere subsets, with Japanese doing better on contradiction but worse on atmosphere [2306.12567].

The 2024 study added a diagnostic **Translate-and-Explain** task intended to separate semantic interpretation from downstream reasoning. It is a **1-shot Chain-of-Thought** prompt with three stages: **Translation**, **Reasoning**, and **Answer**. The authors compared three settings: **Explanation only**, **Predicate logic translation + explanation**, and **Set theory translation + explanation** [2408.04403].

On the **90-problem Translate-and-Explain task**, translation accuracy was often much higher than final answer accuracy. In English, GPT-4 reached **96.67** on predicate-logic translation and **92.22** on set-theory translation, while its corresponding final answer accuracies were **83.33** and **80.00**. In Japanese, GPT-4 reached **95.56** on predicate-logic translation and **76.66** on set-theory translation, with answer accuracies of **75.56** and **74.44**. GPT-3.5 showed the same qualitative pattern, though with substantially weaker Japanese translation performance, especially for set theory [2408.04403].

The authors use this gap to argue that the main bottleneck is often the reasoning process rather than mere sentence interpretation. That conclusion is supported by the error analysis. Reported failures include treating “All animals are tomatoes” as **set identity** rather than **subset**, mis-scoping negation in “A certain police officer is not a public servant,” confusing existential conjunction with implication in Japanese “Some A are B,” and confusing O-type and E-type in set-theoretic translation [2408.04403].

This diagnostic layer is one of NeuBAROCO’s more distinctive contributions. It shifts the benchmark from simple input-output scoring toward process-level analysis, while still preserving the underlying syllogistic structure.

## 6. Benchmark landscape, comparative positioning, and later extensions

NeuBAROCO has become a reference point in later work on Japanese reasoning benchmarks. The 2025 paper introducing **BIS Reasoning 1.0** explicitly presents NeuBAROCO as the closest predecessor for Japanese syllogistic belief-bias evaluation, but also argues that it is not a dedicated benchmark for belief-inconsistent reasoning at scale. That paper states that NeuBAROCO’s Japanese subset contains **fewer than 800 examples for the NLI task and under 100 for the multiple-choice format**, and that it “does not exclusively target belief-inconsistent reasoning.” In experiments, BIS reports complementary evaluation on **over 300 belief-inconsistent syllogistic reasoning samples from the NeuBAROCO benchmark** [2506.06955].

The BIS comparison is informative because it frames NeuBAROCO as broader in bias coverage but less concentrated on the specific case of logically valid conclusions that conflict with common belief. On the NeuBAROCO belief-inconsistent subset used there, **GPT-4o** reached **94.01**, whereas on BIS it reached **79.54**; the same table reports large discrepancies for Claude models, which performed much better on NeuBAROCO than on BIS. The BIS authors interpret this as evidence that task formulation and dataset characteristics strongly affect conclusions about model robustness [2506.06955].

A later development extends the name NeuBAROCO beyond classical syllogistics. The 2025 paper **“Normative Reasoning in Large Language Models: A Comparative Benchmark from Logical and Modal Perspectives”** describes a benchmark positioned as an extension of prior NeuBAROCO work on syllogistic reasoning and human-like reasoning biases. In that setting, the name is repurposed for a comparative framework on **deontic** and **epistemic** modal reasoning, with both single-premise deontic-logic tasks and multi-premise syllogistic tasks [2510.26606]. This indicates a conceptual continuity—cognitively informed logical benchmarking with explicit attention to bias and formal patterning—even though the target phenomena broaden from categorical syllogisms to modal reasoning.

In the benchmark landscape, NeuBAROCO is therefore best understood as an influential bridge resource. It links psychological experiments on human reasoning to bilingual LLM evaluation; it exposes not only logical weakness but structured, human-like bias; and it has served both as a precursor to more focused Japanese belief-inconsistency benchmarks and as a foundation for later benchmark expansion into modal normative reasoning [2408.04403][2506.06955][2510.26606].

Source: https://www.emergentmind.com/topics/neubaroco