FOLIO: NL Reasoning & FOL Translation Benchmark

• FOLIO is a natural language benchmark that pairs expert-written premises with conclusions labeled as True, False, or Unknown to evaluate logical reasoning.
• It integrates NL-to-FOL translation with formal entailment checks, constructed via a six-stage annotation process using diverse story sources.
• The benchmark tests methods like theorem proving and iterative repair, highlighting gaps between large language models and expert reasoning accuracy.

FOLIO most commonly denotes a benchmark for natural language reasoning with first-order logic (FOL): an expert-written, human-annotated dataset in which a set of premises is paired with one or more conclusions, and each conclusion is labeled as True, False, or Unknown relative to the premises (Han et al., 2022). In subsequent work on LLMs, the same three-way task is often operationalized as True, False, or Uncertain, with FOLIO treated as a testbed for faithful logical reasoning rather than open-ended question answering (Kalyanpur et al., 2024). In a separate line of graph-minor research, Folio also denotes a bounded rooted-minor summary of a graph, formalized as $d\text{-}\mathsf{folio}$ and $(k,d)\text{-}\mathsf{folio}$ (Cavallaro et al., 14 May 2026). The dominant contemporary usage, however, is the logical-reasoning benchmark introduced in 2022.

1. Definition, scope, and representational form

The original benchmark presents each natural-language story $S$ as a set of premises

$P = \{p_1, p_2, ..., p_n\}$

and conclusions

$H = \{h_1, h_2, ..., h_m\},$

together with a parallel FOL story $SF$ consisting of

$PF = \{pf_1, pf_2, ..., pf_n\}$

and

$HF = \{hf_1, hf_2, ..., hf_m\},$

where each $pf_i$ and $hf_i$ is intended to be semantically and logically aligned with its natural-language counterpart (Han et al., 2022). The task is to determine whether each conclusion is True, False, or Unknown. In later LLM work, the same operational semantics are usually described as entailment, contradiction, or non-entailment, and the label set is often rendered as True, False, or Uncertain (Kalyanpur et al., 2024).

FOLIO was introduced because earlier reasoning benchmarks either did not isolate logical reasoning from other skills, used synthetic language, had limited logical complexity and diversity, or lacked formal symbolic annotations (Han et al., 2022). The benchmark is therefore defined simultaneously as an NL reasoning task and an NL-FOL translation task. Its central design premise is that natural-language reasoning should be formally checkable.

The paper’s dataset counts are reported in two slightly different ways. The abstract states that FOLIO consists of 1,430 examples (unique conclusions) paired with 487 sets of premises, while the main statistics table reports 487 stories, 2,407 premises, and 1,435 conclusions (Han et al., 2022). The train/validation/test split is 1,001 / 203 / 226 examples (Han et al., 2022).

2. Construction, annotation, and formal semantics

FOLIO was built through a six-stage manual process: WikiLogic collection, HybLogic collection, NL quality control, FOL quality control, NL-FOL alignment, and FOL verification (Han et al., 2022). Total annotation effort is reported as 980 man-hours (Han et al., 2022). Annotators were selected for native or near-native English ability, formal education or self-study in first-order logic or semantic parsing, NLP expertise for language review, and FOL expertise for formal review (Han et al., 2022).

The benchmark combines two construction sources. WikiLogic uses random Wikipedia pages as topical seeds and yields 304 stories, 1,353 premises, 753 conclusions, vocabulary 3,250, average sentence length 8.50 words, reasoning depth 1–5, and 51 distinct ASTs (Han et al., 2022). HybLogic uses combinations of the 24 valid syllogisms among 256 syllogism types and yields 183 stories, 1,054 premises, 682 conclusions, vocabulary 1,902, average sentence length 11.52 words, reasoning depth 5–8, and 25 ASTs (Han et al., 2022). The combined dataset reports vocabulary 4,351, average sentence length 9.86 words, readability spanning 0–14 grade, and 76 distinct ASTs (Han et al., 2022).

The FOL side adopts standard AI-style first-order logic, including

$(k,d)\text{-}\mathsf{folio}$0

with temporal and modal logic explicitly out of scope (Han et al., 2022). Translation conventions aim to preserve semantics and sentence structure, avoid unnecessary semantic decomposition, ignore tense, normalize plural verb forms, and primarily use one-place and two-place predicates, with $(k,d)\text{-}\mathsf{folio}$1-place predicates where needed (Han et al., 2022). A representative example formalizes the premise “All people who regularly drink coffee are dependent on caffeine” as

$(k,d)\text{-}\mathsf{folio}$2

and “No one who jokes about being addicted to caffeine is unaware that caffeine is a drug” as

$(k,d)\text{-}\mathsf{folio}$3

(Pan et al., 2023).

Logical correctness is enforced by both human review and automatic verification. The original release used an FOL inference engine from the Stanford CS221 logic assignment framework, together with a Python semantic parser that converts the dataset’s notation into the prover’s input format (Han et al., 2022). Later papers make the label semantics fully explicit: True iff the premises entail the conclusion, False iff the premises entail its negation or the conclusion contradicts the premises, and Uncertain/Unknown otherwise (Ni et al., 3 Mar 2026).

3. Benchmark structure, difficulty, and early results

FOLIO was designed to be harder and more natural than prior synthetic logic datasets. The paper reports that the mode reasoning depth is 4, and 28.7\% of examples require five or more reasoning steps (Han et al., 2022). It also reports that GPT-4 performs much better on shallow examples than on those with reasoning depth 4–7, and that on the harder HybLogic subset GPT-4 reaches only 53.10\% (Han et al., 2022). This suggests that FOLIO’s difficulty is driven not only by label uncertainty but by multi-step compositional structure.

Human and model baselines further underline that difficulty. The majority baseline on the test set is 38.5\%, because the 226 test examples contain 87 true, 78 false, and 61 unknown conclusions (Han et al., 2022). Fully supervised fine-tuning yields 62.1 for RoBERTa-large and 65.9 for Flan-T5-Large, while GPT-4 reaches 64.2 with 8-shot NL prompting, 68.9 with CoT, 69.5 with CoT + self-consistency, and 70.0 with Tree-of-Thought (Han et al., 2022). The paper also reports 95.98\% expert human accuracy, 61.82\% non-expert human accuracy, and about 64.16\% for GPT-4 few-shot prompting, leaving a large expert–model gap (Han et al., 2022).

The original paper’s broader conclusion is that plain LLM prompting does not solve FOLIO well, particularly on False and Unknown cases, and that logic-specific methods are materially stronger (Han et al., 2022). This later became the benchmark’s defining role: a stress test for whether a system can formalize and reason, rather than paraphrase and guess.

4. Methodological ecosystem and solver-backed systems

A large fraction of subsequent work on FOLIO treats it as a benchmark for semantic parsing plus formal reasoning. The dominant pattern is to translate natural-language premises and conclusions into a symbolic representation, then use a theorem prover, SMT solver, ASP solver, or proof assistant to determine entailment, contradiction, or non-entailment (Pan et al., 2023).

Representative systems illustrate several distinct paradigms:

System	Reported FOLIO result	Setup note
Logic-LM (Pan et al., 2023)	78.92	GPT-4, Prover9, full 204-example test set
Logic-LM++ (Kirtania et al., 2024)	84.80	GPT-4, semantic backtracking over symbolic refinements
LeanReasoner (Jiang et al., 2024)	82.6%	full 204-example test set, Lean proof search
LLM-ARC (Kalyanpur et al., 2024)	88.32%	validation set, ASP + test generation + iterative repair
Auto-formalization agent (Ni et al., 3 Mar 2026)	86.70%	cleaned FOLIO validation set, Z3Py under assessor protocol
LogicAgent (Zhang et al., 29 Sep 2025)	79.90	Qwen2.5-32B, semiotic-square-guided FOL reasoning
MoT (Zheng et al., 21 May 2025)	78.9	Gemma-2-9B-It, multi-modal reasoning without external solver

These numbers are not directly interchangeable, because the papers use different splits, cleaned subsets, or evaluation protocols. What they do show is methodological convergence around FOLIO as a benchmark for faithful logical reasoning under natural-language input.

The first wave emphasized external theorem proving. Logic-LM translates premises and hypotheses into FOL, uses Prover9, and maps ENTAILMENT / CONTRADICTION / CONTINGENT to True / False / Uncertain; on GPT-4 it reports 78.92 on FOLIO (Pan et al., 2023). LINC similarly uses an LLM as a semantic parser plus Prover9, then majority-votes over multiple parses, reaching 72.5% on a cleaned 182-example FOLIO validation set for GPT-4 and outperforming CoT for smaller models (Olausson et al., 2023). LeanReasoner instead casts the task into Lean theorem proving, turns each question into positive and negative theorem targets, and reports 82.6% on the full 204-example FOLIO test set after mathlib pretraining and fine-tuning on 27 FOLIO proofs (Jiang et al., 2024).

A second wave focused on iterative repair and verification. Logic-LM++ adds pairwise semantic comparison and backtracking to symbolic refinement and reports 84.80 on FOLIO with GPT-4 (Kirtania et al., 2024). LLM-ARC reframes FOLIO as an Actor–Critic loop in Answer Set Programming, where the Actor writes ASP plus semantic tests and the Critic runs Clingo, compiles the program, executes tests, and returns structured feedback; the trained system reaches 88.32% on the validation set and is presented as a new state of the art (Kalyanpur et al., 2024). Agentified Assessment of Logical Reasoning Agents evaluates a Z3Py auto-formalization agent on a cleaned validation split and reports 86.70% versus 73.89% for a chain-of-thought baseline, with the largest gain on FALSE cases, from 44.26% to 77.05% (Ni et al., 3 Mar 2026).

A third line explores prompting-only or hybrid prompting approaches. GReaTer treats FOLIO as a three-way natural-language reasoning task and optimizes prompts for small open models using gradients over generated reasoning trajectories; it reports 69.1 or 68.5 for Gemma-2-9B-it and 62.6 for Llama-3-8B-Instruct, depending on the table (Das et al., 2024). MoT trains a single model across natural language, code, and truth-table modalities, then majority-votes across modalities at inference time; on FOLIO it raises Gemma-2-9B-It from 69.5 to 78.9 and Qwen2.5-7B-Instruct from 71.9 to 78.3 (Zheng et al., 21 May 2025). LogicAgent adds semiotic-square-guided multi-perspective deduction, existential import checks, and a three-valued decision scheme, reporting 79.90 on FOLIO and an ablation drop to 34.31 when FOL input is removed (Zhang et al., 29 Sep 2025). By contrast, Symbolic-Aided CoT is explicitly non-iterative and tool-free; on FOLIO it improves GPT-4 CoT but underperforms stronger iterative or solver-assisted systems, and on several open models it is slightly below ordinary CoT (Nguyen et al., 17 Aug 2025).

5. Benchmark reliability, annotation repair, and evaluation disputes

Later work shows that FOLIO’s formal layer is itself a research topic. A 2026 audit of the FOLIO validation split reports 107 / 275 = 38.9% incorrect FOL formalizations, including 29 / 275 = 10.5% syntactic errors, 78 / 275 = 28.3% semantic errors, and 45 / 275 = 16.4% ambiguous natural-language sentences; it also reports 17 corrected NLI labels, stated as about 8.4% (Brunello et al., 1 Jun 2026). The same paper argues that correcting these errors changes benchmark conclusions materially: testing Gemma~4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini against corrected ground truth yields FOLIO translation-accuracy gains of +21.8, +11.0, and +12.7 points, respectively, on all instances (Brunello et al., 1 Jun 2026).

A related 2026 paper builds a solver-verified and repaired split of FOLIO for agent evaluation and reports that the original dataset exhibits “potential label errors and misalignments between natural language and formal annotations due to the complexity of semantic parsing” (Ni et al., 3 Mar 2026). Its cleaning pipeline uses Vampire, a critique agent, a refiner agent, and iterative re-verification. On the validation set of 203 examples, it reports 154 (75.9%) verified directly, 10 (4.9%) verified after automatic repair, and 39 (19.2%) remaining problematic (Ni et al., 3 Mar 2026). The paper then evaluates on what it calls the cleaned validation set comprising 203 examples after label verification, a wording that leaves the exact handling of the 39 problematic cases somewhat unclear (Ni et al., 3 Mar 2026).

These findings do not invalidate FOLIO as a benchmark, but they do shift its interpretation. A plausible implication is that FOLIO should be treated not only as a reasoning benchmark, but also as a benchmark for annotation quality, ontology discipline, and NL-to-FOL faithfulness. That implication is reinforced by MATP, which uses theorem proving to audit LLM-generated reasoning chains and finds on FOLIO both a much lower execution accuracy for NL2FOL translation (70.88) and a much higher evaluation failure rate (31.92%) than on more synthetic datasets (Zheng et al., 29 Dec 2025).

6. Alternative technical meaning: $(k,d)\text{-}\mathsf{folio}$4-Folio in graph-minor theory

In structural graph theory, Folio denotes a different object altogether. A rooted graph $(k,d)\text{-}\mathsf{folio}$5 has

$(k,d)\text{-}\mathsf{folio}$6

and its $(k,d)\text{-}\mathsf{folio}$7-folio is the set of rooted graphs of detail at most $(k,d)\text{-}\mathsf{folio}$8 that occur as rooted minors of $(k,d)\text{-}\mathsf{folio}$9 (Cavallaro et al., 14 May 2026). For an annotated graph $S$0, the paper defines the $S$1-folio as the union of these $S$2-folios over all ordered multisubsets of $S$3 of size $S$4 (Cavallaro et al., 14 May 2026).

This graph-theoretic Folio is the abstraction used to unify $S$5-Disjoint Paths, Rooted Minor Checking, and bounded rooted pattern enumeration (Cavallaro et al., 14 May 2026). The main theorem in the cited work states that if an annotated graph $S$6 has bidimensionality at most $S$7 and

$S$8

then there exists a vertex outside $S$9 that is strongly irrelevant for $P = \{p_1, p_2, ..., p_n\}$0, with

$P = \{p_1, p_2, ..., p_n\}$1

(Cavallaro et al., 14 May 2026). The same paper reports that the classical linkage function satisfies

$P = \{p_1, p_2, ..., p_n\}$2

(Cavallaro et al., 14 May 2026).

This usage is conceptually unrelated to the natural-language benchmark. The shared name reflects a common idea of a bounded summary of what a structure “contains,” but the mathematical objects, tasks, and proof techniques are entirely different.

FOLIO therefore occupies two distinct positions in current research. As a benchmark, it is a formally grounded, expert-written testbed for natural-language deductive reasoning, NL-to-FOL translation, and solver-backed evaluation. As a graph-minor abstraction, Folio is a rooted-minor summary operator central to irrelevant-vertex theorems and fixed-parameter algorithms. In contemporary machine-reasoning literature, the first sense dominates, and its significance lies precisely in the tension it exposes between fluent language modeling and faithful logical inference.