QMFOL: Benchmarking Large Language Model Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

Abstract: LLMs have made significant progress in reasoning, particularly in deductive reasoning, which is crucial for high-stakes decision-making. As models improve, evaluation benchmarks should evolve to keep pace. However, existing benchmarks lack fine-grained control over logical complexity and struggle to balance semantic diversity with logical consistency. To address these issues, we propose QMFOL, an automated framework for generating monadic first-order logic reasoning tasks with quantifiable and controllable complexity. It constructs formal logical structures using conjunction and disjunction patterns, enabling precise control over reasoning depth, width, label types, and distractors. These structures are then translated into natural language via LLMs, with logical consistency ensured through round-trip verification using an external prover. Based on our framework, we build QMFOLBench, a benchmark comprising 2880 instances with 960 configurations across diverse logical and semantic dimensions. Evaluations on six large reasoning models (LRMs) and two LLMs show that performance degrades and computational overhead increases with rising logical complexity. Models perform better on True-labeled tasks than on False or Unknown ones, and exhibit sensitivity to semantic variation. Overall, QMFOL offers a scalable and reliable approach for constructing deductive reasoning benchmarks with controllable complexity, enabling more precise evaluation of reasoning capabilities in modern LLMs.

• The paper introduces QMFOL as an automated framework for generating quantifiable MFOL tasks that precisely control logical complexity.
• It employs systematic logic construction, natural language translation with ATP verification, and parameterized distractor integration to evaluate model reasoning.
• Experimental evaluations reveal that while some models maintain high accuracy under complexity, increased distractors expose limitations in robustness and systematic bias.

QMFOL: Benchmarking LLM Reasoning via Quantifiable Monadic First-Order Logic Test Case Generation

Problem Statement and Motivation

The paper introduces QFOL, an automated framework for generating monadic first-order logic (MFOL) reasoning tasks with quantifiable and controllable complexity. The motivation arises from deficiencies in existing deductive reasoning benchmarks: limited granularity in logical complexity control and trade-offs between semantic diversity and logical consistency. These limitations hinder rigorous and scalable evaluation of LLMs and large reasoning models (LRMs). QFOL aims to resolve these through algorithmic construction of logical structures, parameterized by depth, width, label type, and distractors, and by translating formal logic into natural language with strict logical verification.

Figure 1: Data distribution of QFOLBench, spanning depth, width, label types, distractor counts, and semantic topics for a total of 960 configurations and 2880 instances.

Overview of QFOL Framework

QFOL targets MFOL, restricting predicates to unary forms, which enables explicit control of logical structures and preserves essential quantified reasoning. Logical rules are constructed algorithmically, leveraging conjunction and disjunction patterns for parameterized expansion of depth and width. Each reasoning task comprises a set of premises (rules and facts), a candidate conclusion, and a label (True, False, Unknown), mapping to entailment, contradiction, or independence.

The process includes:

• Logic Construction: Systematic expansion of base rule sets for specified depth and width, incorporation of distractor rules to challenge model robustness, and derivation of implicit rules for structured task generation.
• FOL2NL Translation: LLM-driven prompt engineering, mapping MFOL predicates to domain semantics and conversion to natural language statements. Logical consistency is enforced by round-trip translation (NL2FOL) and automated theorem proving (ATP) validation.
• Task Generation: Fine-grained control over dimensions (depth, width, distractors, label, topic) for benchmark construction.

  Figure 2: QFOL pipeline. Inputs (blue) include logical parameters and topics; outputs (pink) are validated NL tasks. $N_{i}$ denotes atomic or negated predicates.

QFOLBench Benchmark

QFOLBench is constructed from QFOL, containing 2880 MFOL tasks partitioned across four depth levels, four width levels, three label types, five distractor counts, and four semantic topics. Controlled partitioning enables multidimensional analysis—by depth, width, distractor, label, and topic—facilitating diagnostic evaluation of reasoning strategies and systematic identification of brittleness in LLMs/LRMs.

Experimental Evaluation

The paper presents a comprehensive evaluation of six LRMs and two LLMs on QFOLBench, assessing logical reasoning performance, robustness to distractors, and semantic dependence.

Logical Complexity and Model Performance

Performance metrics include macro and label-wise F1 scores, answer accuracy, computational overhead (time, token usage), and error analysis. Increasing depth and width yields non-linear performance degradation and increased overheads for most models, though Gemini-3.1-Pro and GPT-5.4-High maintain high accuracy (99.03% and 97.40%, respectively) across all complexity partitions.

Figure 3: Model accuracy, time overhead, and token overhead across depth-width subsets; Gemini-3.1-Pro and GPT-5.4-High demonstrate stable performance even at maximum complexity.

Models exhibit systematic bias toward True tasks, with more errors and misclassifications for False and Unknown labels. Error cases arise primarily from failure to identify contradictions or incomplete reasoning, with forward-only reasoning heuristics.

Robustness to Distractor Rules

Addition of distractor rules degrades performance in most models, supporting the claim that robustness is non-trivial even in controlled logical settings. Ablation studies show that the performance decline is caused by logical distractors—not context length—reinforcing the importance of reasoning over superficial signal integration.

Figure 4: Model answer changes across distractor subset variants; correct-to-wrong transitions predominate as distractor count increases.

Gemini-3.1-Pro shows minimal degradation, while Qwen3-32B exhibits the largest decline. Some models occasionally benefit from distractors, moving from wrong to correct answers, suggesting that distractors can prompt deeper reasoning but generally confound less robust architectures.

Semantic Dependence

Performance varies across semantic topics—even under identical logical structure—highlighting the persistent influence of domain familiarity and external knowledge leakage in LLMs/LRMs. Models perform best on University-related reasoning, with DeepSeek models underperforming on Mathematics, often incorporating external knowledge contrary to premises.

Implications and Future Directions

QFOL enables precise, scalable deductive reasoning benchmarking with multidimensional control and high logical consistency. The empirical evaluation surfaces systematic biases (label-type, robustness, domain sensitivity) in SOTA models, supporting the need for richer and more rigorous benchmarks.

Practically, QFOLBench offers nuanced performance diagnostics for evolving LLMs/LRMs and exposes brittleness otherwise masked by coarse benchmarks. Theoretically, the MFOL-centric construction provides an extendable foundation for benchmarks with higher-arity predicates, more complex logical relations, and additional reasoning dimensions.

Future directions include:

• Extension to full FOL (higher-arity, relational predicates).
• Expansion of distractor strategies and semantic topics.
• Integration of richer task configurations facilitating longitudinal evaluation of logical reasoning advancements.
• Investigation into coupled reasoning-translation fidelity, optimizing prompt-based conversion for diverse model architectures.

Conclusion

QFOL establishes a rigorous methodology for deductive reasoning benchmarks via automated MFOL task generation, multidimensional parameterization, and provable logical consistency. QFOLBench enables systematic evaluation of state-of-the-art LLMs and LRMs, uncovering detailed reasoning behaviors, robustness limitations, and semantic dependencies. The methodology is extendable for future research in logical benchmarking, fostering deeper analysis and advancement of LLM reasoning capabilities (2606.20227).