Fixing FOLIO and MALLS: Verified Annotations and an LLM-assisted Framework to Focus Human Relabeling

Abstract: Accurate translation from Natural Language to First-Order Logic (NL-to-FOL) underpins neurosymbolic AI systems and Natural Language Inference (NLI), making the quality of NL-to-FOL benchmarks essential -- yet these datasets have never been rigorously audited. Our first contribution is to present a systematic human inspection of the validation split of \textsf{FOLIO} and a subset of \textsf{MALLS} test instances, finding that approximately 39% and 36% of entries, respectively, contain incorrect FOL formalizations (i.e., ground truth labels), with additional rates of ambiguous NL sentences (16.4% and 48%) and incorrect NLI labels in \textsf{FOLIO} (8.4%). Our second contribution is to develop and release corrected ground truths for such datasets, showing that annotation errors distort model evaluation on a reference benchmark task: testing three state-of-the-art LLMs (Gemma~4 31B-it, Qwen3-30B-A3B, and GPT-4o-mini) with the corrected ground truths yields accuracy gains from +9 to +22 percentage points. Motivated by these findings, we propose an LLM-based framework to support humans in manual reviewing NL-to-FOL datasets. By directing reviewers toward the most error-prone instances, we empirically show that it is possible to achieve 90% dataset accuracy after reviewing fewer than 24% of instances, compared to over 70% required by unguided review. We release all human-verified annotations and the code for our framework.

• The paper uncovers substantial annotation errors and ambiguities in NL-to-FOL datasets, with error rates reaching up to 39%, which distort model evaluation outcomes.
• The paper introduces a novel LLM-assisted framework featuring dual pipelines that refine and regenerate formalizations to efficiently prioritize human review.
• The paper demonstrates that targeted human oversight guided by LLM verdicts can boost NL-to-FOL translation accuracy by up to 22 percentage points while reducing review effort.

Fixing NL-to-FOL Benchmarks: Verified Annotation and LLM-Assisted Oversight

Motivation and Background

The translation of natural language (NL) into First-Order Logic (FOL) is foundational for neurosymbolic AI, facilitating reasoning, formal verification, and natural language inference (NLI). However, NL-to-FOL datasets such as FOLIO and MALLS, pivotal for benchmarking autoformalization systems, lack systematic annotation audits. This paper identifies substantial annotation errors and ambiguities, quantifies their downstream impact on evaluation, and proposes an efficient LLM-assisted human curation framework to optimize review efforts.

Systematic Dataset Audit

A comprehensive manual inspection of FOLIO (275 validation instances) and MALLS (100 test instances) uncovers error rates of 39% and 36%, respectively, in FOL formalizations, with additional ambiguity rates of 16.4% (FOLIO) and 48% (MALLS) in NL sentences. FOLIO further includes mislabelled NLI labels (8.4%). These errors aggregate syntactic flaws (parenthesis mismatches, symbol typos, misuse), semantic mistranslations (quantifier scope, logical structure, entity relativization), and ambiguous NL constructs yielding multiple defensible interpretations.

Ontology extraction and correction are necessary due to absent explicit ontologies; automated symbol mapping, verified by annotators, is utilized. Annotation protocol involves dual-review and consensus resolution, ensuring robust error detection beyond LLM generative biases.

Impact on Model Evaluation

Annotation noise significantly distorts evaluation outcomes. Re-assessment of advanced LLMs (Gemma 4 31B-it, Qwen3-30B-A3B, GPT-4o-mini) against the revised ground truths produces accuracy increases from +9 to +22 percentage points in NL-to-FOL translation. The largest gains accrue to stronger models, indicating that annotation noise disproportionately penalizes semantically valid but non-reference formalizations.

Figure 1: Pipelines comparison across models and datasets (FOLIO, MALLS, GGC) under the AUC metric; Pipeline 1 yields higher scores and statistically significant improvement.

Figure 2: Accuracy-human effort curves on FOLIO and MALLS; LLM-assisted prioritization enables rapid accuracy gains with minimal human review.

Accuracy restricted to unambiguous instances provides a more precise assessment of model competence, separating translation errors from dataset-derived ambiguities.

LLM-Assisted Curation Framework

Manual oversight scales poorly. The paper introduces an LLM-assisted error prioritization framework comprising two pipelines:

1. Pipeline 1 (Direct V{content}R): The LLM judges and refines existing formalizations, issuing verdicts (\textsf{yes}, \textsf{no}, \textsf{?}) and proposed corrections.
2. Pipeline 2 (Regeneration + V{content}R): The LLM generates fresh formalizations before verdict assessment and refinement.

The key signal, empirically validated, is that the LLM rarely flags a correct formalization as wrong ($\sim$3% false negatives). Leveraging this, the framework prioritizes human review by verdict class: unparseable > ambiguous (\textsf{?}) > incorrect (\textsf{no}) > correct (\textsf{yes}).

Figure 3: Accuracy conditioned on the verdict issued by the judge; strong monotonicity justifies the prioritization order.

Figure 4: Pipeline comparison across models and datasets under the $T_{90}$ metric: fraction of data needing review for 90% accuracy.

Figure 5: Pipeline comparison across models and datasets under the $T_{95}$ metric.

Figure 6: Pipeline comparison across models and datasets under the AAG metric.

Experimental Results

Simulated human review tracks accuracy as a function of human effort. The framework achieves:

• FOLIO: 90% accuracy after reviewing just 24% of instances (versus 74% for random review; 63% for direct LLM regeneration).
• MALLS: 90% accuracy after reviewing 13% (versus 72%/38%).

The steep initial slope of the accuracy-human effort curve confirms the efficacy of verdict-based prioritization. Pipeline 1 demonstrates statistically significant gains over Pipeline 2. Gemma consistently outperforms other models, though baseline-dependent metrics (AAG) reveal nuances due to model-specific Green Baselines.

Figure 7: Model comparison across all datasets and metrics; Gemma achieves consistently higher accuracy and sample efficiency.

Prompting strategies and variants influence performance; feature-based decomposition and overly adversarial prompts underperform, while standard CoT and few-shot variants (B1-B3 with pv1/pv3) are robust.

Figure 8: Prompting strategies and variant comparison heatmap for Pipeline 1; blue-framed cells mark the highest performing group.

Figure 9: Prompt and variant comparison for Pipeline 2; performance landscape is flatter than Pipeline 1.

Figure 10: Oversight curve grid across models and datasets; Pipeline 1 universally dominates baselines for error concentration and review efficiency.

Practical and Theoretical Implications

High annotation noise in NL-to-FOL benchmarks invalidates prior evaluation claims and penalizes models capable of generating syntactically and semantically valid alternatives. Rigorous manual curation is impractical; the LLM-assisted framework serves as an efficient front-end filter for dataset development, formal verification, and runtime oversight scenarios. In contexts where annotation errors are rare (e.g., GGC), pipeline-induced degradation is negligible ($<5\%$).

The results motivate broader adoption of verification-centric curation, support benchmarking protocol revisions, and endorse downstream analyses accounting for ambiguity. The approach generalizes to other neurosymbolic or formal semantic tasks requiring human-in-the-loop correction.

Future Directions

Extensions include scalable application to training splits, iterative pipeline refinement for greater autonomy, automated ontology construction, and linguistic ambiguity detection prior to formalization. Further research could address transferability to other logical formalisms, task domains, and model architectures.

Conclusion

This work exposes critical annotation flaws in FOLIO and MALLS, revises ground truths, and quantifies the substantial impact on LLM evaluation. An LLM-assisted prioritization framework enables efficient human review, achieving high annotation quality with minimal effort. The method is robust, model-agnostic, and validated across multiple benchmarks and metrics, setting a new standard for dataset curation and evaluation reliability.