---
title: 'FOLIO: NL Reasoning & FOL Translation Benchmark'
url: https://www.emergentmind.com/topics/folio
type: topic
---

# FOLIO: NL Reasoning & FOL Translation Benchmark

FOLIO most commonly denotes a benchmark for **natural language reasoning with first-order logic (FOL)**: an expert-written, human-annotated dataset in which a set of premises is paired with one or more conclusions, and each conclusion is labeled as **True**, **False**, or **Unknown** relative to the premises [2209.00840]. In subsequent work on large language models, the same three-way task is often operationalized as **True**, **False**, or **Uncertain**, with FOLIO treated as a testbed for faithful logical reasoning rather than open-ended question answering [2406.17663]. In a separate line of graph-minor research, **Folio** also denotes a bounded rooted-minor summary of a graph, formalized as \(d\text{-}\mathsf{folio}\) and \((k,d)\text{-}\mathsf{folio}\) [2605.14902]. The dominant contemporary usage, however, is the logical-reasoning benchmark introduced in 2022.

## 1. Definition, scope, and representational form

The original benchmark presents each natural-language story \(S\) as a set of premises
\[
P = \{p_1, p_2, ..., p_n\}
\]
and conclusions
\[
H = \{h_1, h_2, ..., h_m\},
\]
together with a parallel FOL story \(SF\) consisting of
\[
PF = \{pf_1, pf_2, ..., pf_n\}
\]
and
\[
HF = \{hf_1, hf_2, ..., hf_m\},
\]
where each \(pf_i\) and \(hf_i\) is intended to be semantically and logically aligned with its natural-language counterpart [2209.00840]. The task is to determine whether each conclusion is **True**, **False**, or **Unknown**. In later LLM work, the same operational semantics are usually described as entailment, contradiction, or non-entailment, and the label set is often rendered as **True**, **False**, or **Uncertain** [2406.17663].

FOLIO was introduced because earlier reasoning benchmarks either did not isolate logical reasoning from other skills, used synthetic language, had limited logical complexity and diversity, or lacked formal symbolic annotations [2209.00840]. The benchmark is therefore defined simultaneously as an **NL reasoning task** and an **NL-FOL translation task**. Its central design premise is that natural-language reasoning should be formally checkable.

The paper’s dataset counts are reported in two slightly different ways. The abstract states that FOLIO consists of **1,430 examples (unique conclusions)** paired with **487 sets of premises**, while the main statistics table reports **487 stories**, **2,407 premises**, and **1,435 conclusions** [2209.00840]. The train/validation/test split is **1,001 / 203 / 226** examples [2209.00840].

## 2. Construction, annotation, and formal semantics

FOLIO was built through a six-stage manual process: **WikiLogic collection**, **HybLogic collection**, **NL quality control**, **FOL quality control**, **NL-FOL alignment**, and **FOL verification** [2209.00840]. Total annotation effort is reported as **980 man-hours** [2209.00840]. Annotators were selected for native or near-native English ability, formal education or self-study in first-order logic or semantic parsing, NLP expertise for language review, and FOL expertise for formal review [2209.00840].

The benchmark combines two construction sources. **WikiLogic** uses random Wikipedia pages as topical seeds and yields **304 stories**, **1,353 premises**, **753 conclusions**, vocabulary **3,250**, average sentence length **8.50 words**, reasoning depth **1–5**, and **51** distinct ASTs [2209.00840]. **HybLogic** uses combinations of the **24 valid syllogisms** among **256** syllogism types and yields **183 stories**, **1,054 premises**, **682 conclusions**, vocabulary **1,902**, average sentence length **11.52 words**, reasoning depth **5–8**, and **25** ASTs [2209.00840]. The combined dataset reports vocabulary **4,351**, average sentence length **9.86 words**, readability spanning **0–14 grade**, and **76** distinct ASTs [2209.00840].

The FOL side adopts standard AI-style first-order logic, including
\[
\lnot,\ \land,\ \lor,\ \rightarrow,\ \forall,\ \exists,\ =
\]
with temporal and modal logic explicitly out of scope [2209.00840]. Translation conventions aim to preserve semantics and sentence structure, avoid unnecessary semantic decomposition, ignore tense, normalize plural verb forms, and primarily use one-place and two-place predicates, with \(n\)-place predicates where needed [2209.00840]. A representative example formalizes the premise “All people who regularly drink coffee are dependent on caffeine” as
\[
\forall x (Drinks(x) \rightarrow Dependent(x)),
\]
and “No one who jokes about being addicted to caffeine is unaware that caffeine is a drug” as
\[
\forall x (Jokes(x) \rightarrow (\neg Unaware(x)))
\]
[2305.12295].

Logical correctness is enforced by both human review and automatic verification. The original release used an FOL inference engine from the Stanford CS221 logic assignment framework, together with a Python semantic parser that converts the dataset’s notation into the prover’s input format [2209.00840]. Later papers make the label semantics fully explicit: **True** iff the premises entail the conclusion, **False** iff the premises entail its negation or the conclusion contradicts the premises, and **Uncertain/Unknown** otherwise [2603.02788].

## 3. Benchmark structure, difficulty, and early results

FOLIO was designed to be harder and more natural than prior synthetic logic datasets. The paper reports that the **mode** reasoning depth is **4**, and **28.7\%** of examples require **five or more** reasoning steps [2209.00840]. It also reports that GPT-4 performs much better on shallow examples than on those with reasoning depth **4–7**, and that on the harder HybLogic subset GPT-4 reaches only **53.10\%** [2209.00840]. This suggests that FOLIO’s difficulty is driven not only by label uncertainty but by multi-step compositional structure.

Human and model baselines further underline that difficulty. The majority baseline on the test set is **38.5\%**, because the 226 test examples contain **87 true**, **78 false**, and **61 unknown** conclusions [2209.00840]. Fully supervised fine-tuning yields **62.1** for RoBERTa-large and **65.9** for Flan-T5-Large, while GPT-4 reaches **64.2** with 8-shot NL prompting, **68.9** with CoT, **69.5** with CoT + self-consistency, and **70.0** with Tree-of-Thought [2209.00840]. The paper also reports **95.98\%** expert human accuracy, **61.82\%** non-expert human accuracy, and about **64.16\%** for GPT-4 few-shot prompting, leaving a large expert–model gap [2209.00840].

The original paper’s broader conclusion is that plain LLM prompting does not solve FOLIO well, particularly on **False** and **Unknown** cases, and that logic-specific methods are materially stronger [2209.00840]. This later became the benchmark’s defining role: a stress test for whether a system can formalize and reason, rather than paraphrase and guess.

## 4. Methodological ecosystem and solver-backed systems

A large fraction of subsequent work on FOLIO treats it as a benchmark for **semantic parsing plus formal reasoning**. The dominant pattern is to translate natural-language premises and conclusions into a symbolic representation, then use a theorem prover, SMT solver, ASP solver, or proof assistant to determine entailment, contradiction, or non-entailment [2305.12295].

Representative systems illustrate several distinct paradigms:

| System | Reported FOLIO result | Setup note |
|---|---:|---|
| Logic-LM [2305.12295] | **78.92** | GPT-4, Prover9, full 204-example test set |
| Logic-LM++ [2407.02514] | **84.80** | GPT-4, semantic backtracking over symbolic refinements |
| LeanReasoner [2403.13312] | **82.6%** | full 204-example test set, Lean proof search |
| LLM-ARC [2406.17663] | **88.32%** | validation set, ASP + test generation + iterative repair |
| Auto-formalization agent [2603.02788] | **86.70%** | cleaned FOLIO validation set, Z3Py under assessor protocol |
| LogicAgent [2509.24765] | **79.90** | Qwen2.5-32B, semiotic-square-guided FOL reasoning |
| MoT [2505.15817] | **78.9** | Gemma-2-9B-It, multi-modal reasoning without external solver |

These numbers are not directly interchangeable, because the papers use different splits, cleaned subsets, or evaluation protocols. What they do show is methodological convergence around FOLIO as a benchmark for **faithful logical reasoning under natural-language input**.

The first wave emphasized external theorem proving. **Logic-LM** translates premises and hypotheses into FOL, uses **Prover9**, and maps **ENTAILMENT / CONTRADICTION / CONTINGENT** to **True / False / Uncertain**; on GPT-4 it reports **78.92** on FOLIO [2305.12295]. **LINC** similarly uses an LLM as a semantic parser plus **Prover9**, then majority-votes over multiple parses, reaching **72.5%** on a cleaned 182-example FOLIO validation set for GPT-4 and outperforming CoT for smaller models [2310.15164]. **LeanReasoner** instead casts the task into Lean theorem proving, turns each question into positive and negative theorem targets, and reports **82.6%** on the full 204-example FOLIO test set after mathlib pretraining and fine-tuning on **27** FOLIO proofs [2403.13312].

A second wave focused on **iterative repair and verification**. **Logic-LM++** adds pairwise semantic comparison and backtracking to symbolic refinement and reports **84.80** on FOLIO with GPT-4 [2407.02514]. **LLM-ARC** reframes FOLIO as an Actor–Critic loop in **Answer Set Programming**, where the Actor writes ASP plus semantic tests and the Critic runs **Clingo**, compiles the program, executes tests, and returns structured feedback; the trained system reaches **88.32%** on the validation set and is presented as a new state of the art [2406.17663]. **Agentified Assessment of Logical Reasoning Agents** evaluates a Z3Py auto-formalization agent on a cleaned validation split and reports **86.70%** versus **73.89%** for a chain-of-thought baseline, with the largest gain on **FALSE** cases, from **44.26%** to **77.05%** [2603.02788].

A third line explores prompting-only or hybrid prompting approaches. **GReaTer** treats FOLIO as a three-way natural-language reasoning task and optimizes prompts for small open models using gradients over generated reasoning trajectories; it reports **69.1** or **68.5** for Gemma-2-9B-it and **62.6** for Llama-3-8B-Instruct, depending on the table [2412.09722]. **MoT** trains a single model across natural language, code, and truth-table modalities, then majority-votes across modalities at inference time; on FOLIO it raises Gemma-2-9B-It from **69.5** to **78.9** and Qwen2.5-7B-Instruct from **71.9** to **78.3** [2505.15817]. **LogicAgent** adds semiotic-square-guided multi-perspective deduction, existential import checks, and a three-valued decision scheme, reporting **79.90** on FOLIO and an ablation drop to **34.31** when FOL input is removed [2509.24765]. By contrast, **Symbolic-Aided CoT** is explicitly non-iterative and tool-free; on FOLIO it improves GPT-4 CoT but underperforms stronger iterative or solver-assisted systems, and on several open models it is slightly below ordinary CoT [2508.12425].

## 5. Benchmark reliability, annotation repair, and evaluation disputes

Later work shows that FOLIO’s formal layer is itself a research topic. A 2026 audit of the **FOLIO validation** split reports **107 / 275 = 38.9%** incorrect FOL formalizations, including **29 / 275 = 10.5%** syntactic errors, **78 / 275 = 28.3%** semantic errors, and **45 / 275 = 16.4%** ambiguous natural-language sentences; it also reports **17** corrected NLI labels, stated as about **8.4%** [2606.02837]. The same paper argues that correcting these errors changes benchmark conclusions materially: testing **Gemma~4 31B-it**, **Qwen3-30B-A3B**, and **GPT-4o-mini** against corrected ground truth yields FOLIO translation-accuracy gains of **+21.8**, **+11.0**, and **+12.7** points, respectively, on all instances [2606.02837].

A related 2026 paper builds a **solver-verified and repaired split of FOLIO** for agent evaluation and reports that the original dataset exhibits “potential label errors and misalignments between natural language and formal annotations due to the complexity of semantic parsing” [2603.02788]. Its cleaning pipeline uses **Vampire**, a critique agent, a refiner agent, and iterative re-verification. On the validation set of **203** examples, it reports **154 (75.9%)** verified directly, **10 (4.9%)** verified after automatic repair, and **39 (19.2%)** remaining problematic [2603.02788]. The paper then evaluates on what it calls the cleaned validation set comprising **203 examples after label verification**, a wording that leaves the exact handling of the 39 problematic cases somewhat unclear [2603.02788].

These findings do not invalidate FOLIO as a benchmark, but they do shift its interpretation. A plausible implication is that FOLIO should be treated not only as a reasoning benchmark, but also as a benchmark for **annotation quality, ontology discipline, and NL-to-FOL faithfulness**. That implication is reinforced by **MATP**, which uses theorem proving to audit LLM-generated reasoning chains and finds on FOLIO both a much lower execution accuracy for NL2FOL translation (**70.88**) and a much higher evaluation failure rate (**31.92%**) than on more synthetic datasets [2512.23511].

## 6. Alternative technical meaning: \((k,d)\)-Folio in graph-minor theory

In structural graph theory, **Folio** denotes a different object altogether. A rooted graph \((G,\mathcal R)\) has
\[
\mathsf{detail}(G,\mathcal R)\coloneqq \max\{\, |V(G)\setminus \mathcal R|,\ |E(G)|\,\},
\]
and its \(d\)-folio is the set of rooted graphs of detail at most \(d\) that occur as rooted minors of \((G,\mathcal R)\) [2605.14902]. For an annotated graph \((G,R)\), the paper defines the **\((k,d)\)-folio** as the union of these \(d\)-folios over all ordered multisubsets of \(R\) of size \(k\) [2605.14902].

This graph-theoretic Folio is the abstraction used to unify **\(k\)-Disjoint Paths**, **Rooted Minor Checking**, and bounded rooted pattern enumeration [2605.14902]. The main theorem in the cited work states that if an annotated graph \((G,R)\) has bidimensionality at most \(b\) and
\[
\mathsf{tw}(G)>\beta(k,b,d),
\]
then there exists a vertex outside \(R\) that is **strongly irrelevant** for \((k,d)\text{-}\mathsf{folio}(G,R)\), with
\[
\beta(k,b,d)\in \mathbf{poly}(k)\cdot 2^{\mathbf{poly}(b+d)}
\]
[2605.14902]. The same paper reports that the classical linkage function satisfies
\[
\ell(k)\in 2^{\mathbf{poly}(k)}
\]
[2605.14902].

This usage is conceptually unrelated to the natural-language benchmark. The shared name reflects a common idea of a bounded summary of what a structure “contains,” but the mathematical objects, tasks, and proof techniques are entirely different.

FOLIO therefore occupies two distinct positions in current research. As a benchmark, it is a formally grounded, expert-written testbed for natural-language deductive reasoning, NL-to-FOL translation, and solver-backed evaluation. As a graph-minor abstraction, Folio is a rooted-minor summary operator central to irrelevant-vertex theorems and fixed-parameter algorithms. In contemporary machine-reasoning literature, the first sense dominates, and its significance lies precisely in the tension it exposes between fluent language modeling and faithful logical inference.

Source: https://www.emergentmind.com/topics/folio