Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability

Abstract: LLMs are increasingly used for tasks that implicitly reduce to Boolean satisfiability (SAT), yet their reasoning ability on SAT remains unclear. We present a systematic study of LLMs on 2-SAT and 3-SAT, together with two canonical reductions, Vertex Cover and discrete 3D packing, to probe representation-invariant reasoning. We first evaluate models using conventional metrics, including accuracy, precision, recall, and F1, as well as the SAT phase-transition setting. We find that these metrics can be misleading: many models obtain high scores by over-predicting satisfiable formulas, fail to reproduce the classical easy-hard-easy signature around the 3-SAT threshold, and degrade sharply as the number of variables grows. To address this problem, we introduce a paired-formula protocol based on minimally different satisfiable and unsatisfiable instances, together with Accurate Differentiation Rate (ADR), which requires both members of each pair to be classified correctly. ADR separates reasoning-oriented models from heuristic ones and correlates with witness validity. Beyond CNF, we test cross-representation consistency by converting CNF to Vertex Cover and 3-SAT to discrete 3D packing. Model decisions on CNF and on the corresponding graph or packing instances agree for most models on more than 80 percent of instances, suggesting stable decision rules across representations. Overall, our results show that SAT is a conservative probe for LLM reasoning, and that paired evaluation with ADR provides a more faithful and representation-robust assessment than conventional metrics.

• The paper introduces a novel evaluation protocol using paired SAT/UNSAT instances and the Accurate Differentiation Rate (ADR) metric to assess LLM reasoning.
• The study reveals that while LLMs perform well on small SAT instances, their scalability diminishes, with classical metrics often misrepresenting true competence.
• Empirical tests show that ADR correlates with LLM performance on broader NP reductions, positioning SAT as a foundational benchmark for logic-based reasoning.

Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability

Introduction

The paper "Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability" (2605.28602) presents a systematic and rigorous empirical analysis of the capacity of LLMs to solve Boolean satisfiability (SAT) problems, specifically focusing on 2-SAT and 3-SAT, and examining their performance under canonical NP reductions (Vertex Cover and discrete 3D-packing). The authors critique conventional evaluation metrics and introduce a novel, pair-based protocol and metric—Accurate Differentiation Rate (ADR)—to yield a more principled measure of reasoning ability.

LLMs and the Challenge of SAT: Motivation and Experimental Paradigm

SAT and its variants serve as archetypal benchmarks for evaluating combinatorial and logical reasoning, given SAT's centrality in complexity theory and its ubiquity as a substrate for numerous practical problems. The increasing deployment of LLMs in settings that implicitly or explicitly encode constraint satisfaction problems (e.g., scheduling, planning, program analysis) necessitates a robust probe of whether these models can perform solver-like reasoning or simply exploit superficial statistical cues.

The study targets several axes:

• Establish whether LLMs exhibit the classical easy–hard–easy phase transition and threshold phenomena observed in algorithmic SAT solvers.
• Assess the reliability of standard metrics (accuracy, precision, recall, F1, MCC) in reflecting reasoning competence.
• Develop and validate evaluation methodologies—particularly the paired formula protocol and the ADR metric—that are robust to dataset imbalance and distributional artifacts.

Methodology Overview

The evaluation protocol encompasses:

• Constructing comprehensive datasets of random 2-SAT and 3-SAT CNFs, with careful control of variable count ($N$) and clause-to-variable ratio ($\alpha$), spanning the phase transition region.
• Designing paired SAT/UNSAT instance sets via minimal edits to facilitate fair, structure-sensitive evaluation.
• Querying a spectrum of 12 LLMs (including mainstream and reasoning-optimized variants) via standardized prompts mimicking SAT solver interfaces to elicit decisions and, where appropriate, witness assignments.

A critical methodological contribution is the formulation of the ADR metric, which expresses the fraction of SAT/UNSAT pairs where both instances are correctly classified, circumventing the pathologies of traditional metrics under class imbalance.

Empirical Findings: Classical Metrics Are Inadequate

The investigation begins with an analysis of LLM outputs under the standard phase transition paradigm. While CDCL solvers display hallmark easy–hard–easy behavior as $\alpha$ varies (Figure 1):

Figure 1: Phase transition and easy–hard–easy signature for random 3-SAT with $N=75$.

LLMs, in contrast, fail to reproduce this solver phase diagram. Reported branch counts and SAT/UNSAT probability curves for LLMs are poorly correlated with actual algorithmic hardness or phase transition points.

Figure 2: LLM performance across classical metrics and $\alpha$; models exhibit a strong and persistent SAT bias, high traditional metric scores even when reasoning ability is absent, and are insensitive to the classical phase transition.

Trivial strategies (e.g., always predicting SAT) can lead to deceptively good accuracy, precision, or recall due to label imbalance, especially in low or high $\alpha$ regimes. Among the evaluated models, only the most reasoning-optimized (e.g., GPT-5, deepseek-reasoner, o1) manifest any partial sensitivity to $\alpha$, but still fall short of solver-like phase-sensitivity.

Paired Formulas and ADR: Discriminating True Reasoning

Confronted with the inadequacy of classical metrics, the authors introduce a paired evaluation setting: for each UNSAT instance, a corresponding SAT instance is produced via minimal clause or literal perturbations. Accurate Differentiation Rate (ADR) is defined as the fraction of pairs where both are labeled correctly.

This methodology exposes whether models can distinguish between near-identical instances that differ only in a clause or literal responsible for flipping satisfiability.

Figure 3: Performance of models on paired SAT/UNSAT instances; (i) classical metrics (a–h) can be misleading or uninformative under imbalance or bias, while (i) ADR cleanly identifies the collapse of reasoning ability as $N$ increases.

Empirical results show:

• For small $N$, models like GPT-5 achieve high ADR, indicating genuine sensitivity.
• ADR decays smoothly with increasing $N$, revealing where reasoning breaks down, even as accuracy/F1 under pair-balance stabilizes near 0.5 (random choice).
• Models with strong SAT bias or lacking reasoning specialization quickly drop to ADR values indistinguishable from chance.

2-SAT, Scaling, and Cross-Problem Transfer

The study further examines 2SAT, a tractable SAT fragment, as a baseline. All models perform better on 2-SAT at small $\alpha$0, but ADR continues to be the most diagnostic indicator under scale. Importantly, performance on SAT (as captured by ADR) reliably predicts performance under reductions to Vertex Cover and 3D packing, confirming that SAT is a conservative upper bound for LLM-based reasoning on a broad class of constraint problems.

Figure 4: Comparative performance on 2-SAT using ADR and classical metrics; high performance at small $\alpha$1 collapses under scaling in non-reasoning models, while reasoning-oriented LLMs sustain ADR at higher $\alpha$2.

Cross-Representation Robustness: Vertex Cover and 3D Packing

The study evaluates LLM predictions on CNFs and their canonical reductions to Vertex Cover and discrete 3D-packing:

• Prediction consistency across representations is above 80% for most models, but when predictions disagree, correctness favors SAT CNFs over the reductions.
• Reasoning-optimized models exhibit the same ADR decay under all representations, solidifying ADR as a representation-agnostic probe.
• These findings confirm that SAT competence upper-bounds LLM performance on NP-equivalent problem transformations.

Theoretical and Practical Implications

The findings substantiate several key points:

• Standard machine learning metrics can be systematically misleading in reasoning-laden, imbalanced settings.
• Performance on SAT (especially under paired-testing and ADR evaluation) is a robust predictor for model reliability on any task reducible to SAT, affirming the "foundational probe" thesis.
• Scale remains the central challenge: no model sustains high ADR for $\alpha$3 in 3-SAT, revealing a bottleneck for deploying LLMs in hard logic-driven settings.

On a practical front, the authors advocate for:

• Integrating LLMs with external, verified solvers (CDCL, etc.) to combine ease of use with guarantees of correctness and scalability.
• Employing paired evaluation and ADR as standard components in benchmarking suites for logical reasoning and constraint satisfaction.

Limitations and Future Directions

The main limitations are rooted in the scope of generated instances (random 2SAT/3SAT) and the models’ input length constraints, which limit the $\alpha$4 that can be evaluated. Creation of paired instances beyond SAT requires principled, deterministic editing to orchestrate label flips while preserving all but one constraint.

Future directions cited include:

• Developing training regimes that reward pairwise discrimination and verifiable witness construction.
• Extending ADR-based evaluation to richer forms of logical reasoning and to end-to-end systems employing LLM/solver hybrids.
• Creating larger and more diverse benchmarks to probe representation-invariant reasoning and generalization under reduction.

Conclusion

The work establishes that while recent LLMs manifest basic reasoning skills in SAT solving for small-scale instances, none exhibit solver-like scalability. Classical evaluation metrics are unsuitable in this domain due to imbalance and superficial predictive strategies. The introduction of paired formulas and ADR provides a principled, distribution-robust lens on logical competence, exposing the limits of current LLMs and establishing a pathway for future benchmarking and model development in logic-intensive settings.

Overall, this research underlines SAT (and similar constraint problems) as a stringent benchmark for LLM reasoning and positions ADR-based paired evaluation as the methodological standard for the field.