Know Your Limits : On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning

Abstract: LLMs achieve strong performance on reasoning tasks, but whether this reflects faithful logical inference or heuristic approximation remains unclear. We study this question in legal entailment by comparing three paradigms, including pure LLM classification, LLM-based Formal Reasoning, and solver-based Formal Reasoning using the Z3 SMT solver, on a re-annotated subset of ContractNLI across five LLMs. Our re-annotation reveals a systematic and measurable gap between pragmatic legal interpretation and strict formal entailment, where a substantial proportion of legally sound inferences are not formally grounded without additional unstated assumptions. While introducing formal structure improves accuracy, with LLM-based Formal Reasoning achieving the highest benchmark performance, we show that this gain does not imply faithful reasoning. We identify three recurring failure modes: scope laundering, where LLMs report solver-inconsistent classifications without executing the underlying formal reasoning, producing conclusions that appear logically grounded but are not; implicit constraint blindness, where LLMs overlook logical constraints present in formal representations; and program synthesis failures, where LLMs generate incorrect Z3 code despite structured prompting. Critically, scope laundering persists across all models, raising serious concerns about the faithfulness of LLM-based formal reasoning as a proxy for symbolic execution. These results reveal a fundamental gap between benchmark accuracy and logical faithfulness.

• The paper shows that improved benchmark accuracy in LLM legal entailment arises largely from unfaithful reasoning via implicit assumptions.
• It employs a neuro-symbolic pipeline that autoformalizes legal text into FOL and applies Z3 SMT classification to enforce strict formal semantics.
• The study quantifies errors like scope laundering, implicit constraint blindness, and program synthesis failures, highlighting critical limits in current approaches.

Faithfulness and Limits of LLM-Based Reasoning in Legal Entailment

Introduction

This paper, "Know Your Limits: On the Faithfulness of LLMs as Solvers and Autoformalizers in Legal Reasoning" (2606.16118), investigates the disconnect between observed benchmark performance and faithful logical reasoning exhibited by LLMs on legal entailment tasks. Employing both end-to-end neuro-symbolic frameworks and LLM-based classification paradigms, the study systematically quantifies the gap between pragmatic legal interpretations and strict formal entailment, revealing that much of the observed improvement with structured approaches arises from unfaithful reasoning grounded on implicit, unstated assumptions rather than verifiable logical deductions.

Dataset Construction and Formal Annotation

The research re-annotates the widely used ContractNLI dataset to align verdicts with first-order logical entailment semantics, focusing exclusively on what can be soundly concluded from the contract text without recourse to extratextual or contextual assumptions.

Figure 1: Visualization of the dataset construction workflow, including rigorous relabeling to distinguish legally plausible from formally entailed and neutral cases.

The annotation protocol strictly adheres to SMT-lifted semantics: A hypothesis $H$ is entailed by premise $P$ if $P \wedge \neg H$ is unsatisfiable; contradicted if $P \wedge H$ is unsatisfiable; otherwise, neutral. The resulting label shift predominantly migrates "pragmatic" Entailment and Contradiction instances to Neutral, directly illustrating that natural legal text is frequently too underspecified for robust logic-grounded AI reasoning.

Neuro-symbolic Pipeline

The architecture under evaluation consists of a two-stage pipeline: LLM-based autoformalization of legal text into FOL, followed by Z3-based SMT entailment classification. The pipeline design supports fine-grained error analysis for each stage, distinguishing failures of semantic translation (autoformalization) from failures of logical application.

Figure 2: Model architectures: The left branch autoformalizes legal text to FOL and performs strict SMT verification; the right branch has LLMs perform native reasoning over explicit formal representations, allowing direct comparison of classification faithfulness.

The autoformalizer relies on targeted clause selection, predicate normalization, explicit translation of contract semantics, and manual-corrective program synthesis, aligned with best practices in computational law. SMT-based reasoning then provides gold-standard logical outputs for evaluation.

Empirical Findings

Accuracy vs. Faithfulness

The introduction of formal structure—prompting LLMs to perform reasoning over symbolic FOL inputs—increases benchmark accuracy relative to pure LLM classifiers (up to 83% for Claude-Sonnet-4-6, compared to 63% for pure LLM methods), but this increase is illusory with respect to logical faithfulness.

Key empirical results include:

• LLM-based formal reasoning achieves the highest accuracy but does not consistently enforce logical validity. Many outputs classified as "Entailment" are unjustified under strict SMT criteria.
• The Z3 solver, which only leverages explicit formalized information, is highly conservative. Many hypotheses that are reasonable or even annotated as entailed by humans are predicted neutral due to lack of explicit premises—a direct reflection of the annotation revision.
• There is high error asymmetry: The majority of misclassifications are Neutral→Entailment, indicating LLM reliance on implicit, non-entailed assumptions.

Identification of Failure Modes

The research identifies and quantifies several systematic error types:

• Scope Laundering: LLMs frequently assert logical conclusions as if derived from symbolic execution without actually performing or respecting the SMT evaluation. This error mode is present in all models, with "scope laundering" rates as high as 52% in some cases; i.e., predictions confidently attribute logical justification where none exists.
• Implicit Constraint Blindness: LLMs systematically overlook logical constraints—such as universal conditions specified in existential terms—leading to misclassifications even when formal representations are correct.
• Program Synthesis Failures: The autoformalization step has a substantial error rate for Z3 code, ranging up to 63% for Llama, due to syntax errors, incomplete predicate instantiation, or type mismatches. Even top models must be iteratively prompted to fix code.

These quantitatively robust and recurring failure modes indicate deep limitations in both generalization and symbolic faithfulness in current systems.

Theoretical and Practical Implications

The analysis establishes that higher benchmark "accuracy" on legal entailment tasks does not translate to reliable, auditable AI reasoning suitable for high-stakes domains. While structure and formalization bolster performance on pragmatic or heuristically judged tasks, they mask a core gap: the ability to distinguish what is genuinely, formally entailed versus what is plausibly assumed. The ambiguity between valid inference and unjustified assumption pervades both manual annotation and automated reasoning—a reality with direct implications for responsible AI deployment in law, compliance, finance, and related regulatory contexts.

Techniques derived from the neuro-symbolic pipeline—e.g., surfacing Minimal Correction Subsets via SMT or integrating structured legal ontologies—are outlined as candidate approaches to manage ambiguity without overburdening human verifiers.

Future Directions

Advancements in this field will demand more faithful and reliable autoformalization, stronger semantic alignment (including legal ontologies and clause normalization), robust error surfacing (e.g., minimal assumption reporting), and domain-specific curriculum learning for LLMs. There is also a promising pathway to extend this framework beyond law—into regulation, compliance, and safety-critical reasoning—where the distinction between plausible and provable holds similar pivotal importance.

Conclusion

This study demonstrates a persistent gap between observed benchmark improvements in LLM-based legal reasoning and their genuine logical faithfulness. Scope laundering, assumption ambiguity, and autoformalization failures expose fundamental challenges that cannot be resolved by additional scaling or chain-of-thought prompting alone. Reliably automating legal entailment will require explicit surfacing of the assumptions underlying each conclusion and formal methods that decisively delineate the boundary of what is entailed and what is merely plausible.

The alignment of annotation, formal methods, and LLM outputs remains a central bottleneck for legal AI; until this is resolved, strict SMT-based reasoning remains essential for applications demanding verifiable, auditable logical inference.