Accurate Differentiation Rate (ADR) in LLM Evaluation

• Accurate Differentiation Rate (ADR) is a metric defined over minimally edited SAT/UNSAT pairs that measures joint-correctness to assess LLM reasoning.
• The paired-formula protocol employs minimal edits—such as clause deletion or literal polarity flip—to isolate the effect of a single change on satisfiability.
• Empirical results show that ADR robustly orders models by logical competence and outperforms conventional metrics, even across NP reductions.

Accurate Differentiation Rate (ADR) is a matched-pair evaluation metric introduced for assessing LLMs on satisfiability problems in "Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability" (Zhang et al., 27 May 2026). It is defined over minimally edited satisfiable and unsatisfiable instance pairs and requires both members of a pair to be classified correctly. In the study that introduced it, ADR is proposed as a more faithful assessment of reasoning capability than conventional metrics such as accuracy, precision, recall, and F1, particularly in settings where models over-predict satisfiable formulas, where class balance varies sharply with clause density, and where representation-invariant reasoning is of interest (Zhang et al., 27 May 2026).

1. Definition and mathematical properties

Let $\{(F_i^{SAT}, F_i^{UNSAT})\}_{i=1}^M$ be $M$ paired formulas, where $F_i^{SAT}$ is satisfiable and $F_i^{UNSAT}$ is unsatisfiable, differing only by a minimal edit. Let $\hat y(F) \in \{SAT, UNSAT\}$ be the model’s predicted label. Define the indicator variables

$$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$

ADR is then the fraction of pairs on which both judgments are correct:

$ADR = \frac{1}{M}\sum_i A_i B_i.$

This definition makes ADR a joint-correctness criterion rather than a per-instance average (Zhang et al., 27 May 2026).

The paper also gives an equivalent characterization in terms of per-class recall. If

$$r_S = \frac{1}{M}\sum A_i, \qquad r_U = \frac{1}{M}\sum B_i,$$

then

$$\max(0, r_S + r_U - 1) \le ADR \le \min(r_S, r_U) \le \frac{r_S + r_U}{2} = Accuracy.$$

These inequalities formalize the distinction between ADR and ordinary accuracy. Accuracy can remain moderate even when pairwise differentiation fails, whereas ADR falls unless the model correctly separates satisfiable and unsatisfiable members of the same matched pair (Zhang et al., 27 May 2026).

In the formulation used in the paper, ADR is therefore not merely another aggregate classification score. It encodes a stricter requirement: the model must correctly identify the logical consequence of a minimal edit while holding the surrounding structure nearly fixed. This suggests that ADR is designed to track discriminative logical sensitivity rather than generic label preference.

2. Paired-formula protocol

The paired-formula protocol begins with an LLM-verified UNSAT CNF and generates a partner SAT CNF by a single minimal edit, checking with a classical SAT solver (MiniSat) that the edited formula is satisfiable. In practice, the edit is one of three operations: deleting one clause, flipping the polarity of one literal, or replacing one literal with another variable. The result is a satisfiable/unsatisfiable pair differing only at a minimal local point (Zhang et al., 27 May 2026).

Each pair shares the same variables $N$ and nearly identical clause density $M$0, differing by at most $M$1. The stated rationale is that any solution-space change is due solely to the edit. This isolates the effect of a single clause- or literal-level perturbation and suppresses confounding differences in size or density (Zhang et al., 27 May 2026).

The protocol is explicitly motivated as a way to force the model to classify two formulas that look almost the same. Under this construction, trivial heuristics such as “always guess SAT” fail, since one formula is SAT and the other is UNSAT. The paper presents this as the core reason ADR isolates genuine reasoning more effectively than evaluations on unpaired mixed collections of SAT and UNSAT instances (Zhang et al., 27 May 2026).

3. Relation to conventional evaluation metrics

The study evaluates models with conventional metrics including accuracy, precision, recall, and F1, as well as the SAT phase-transition setting, but argues that these metrics can be misleading. On mixed SAT/UNSAT benchmarks, low-$M$2 formulas are almost all SAT and high-$M$3 formulas almost all UNSAT, so benchmark composition itself can inflate scores. The paper also reports that most LLMs strongly prefer SAT, which further distorts standard measures (Zhang et al., 27 May 2026).

ADR addresses these issues by requiring joint correctness on both members of each pair. Under ADR, trivial strategies such as always-SAT or always-UNSAT yield $M$4. By contrast, accuracy can remain at $M$5 even when pairwise differentiation collapses. The paper gives a simple counterexample: if a model always predicts SAT, then $M$6 and $M$7, so Accuracy $M$8 but $M$9 (Zhang et al., 27 May 2026).

The paper further states that F1 and MCC become undefined or misleading under severe label skew in each $F_i^{SAT}$0-bin. This is part of the broader argument that conventional metrics on random SAT benchmarks can conflate genuine logical discrimination with class imbalance and model bias. ADR is presented as a metric that remains well-behaved under class balance because it is computed over balanced SAT/UNSAT pairs (Zhang et al., 27 May 2026).

A common misreading, directly countered by these results, is that high aggregate accuracy alone establishes reasoning capability. The study instead finds that many models obtain high scores by over-predicting satisfiable formulas, fail to reproduce the classical easy-hard-easy signature around the 3-SAT threshold, and degrade sharply as the number of variables grows (Zhang et al., 27 May 2026).

4. Empirical behavior on 3-SAT and 2-SAT

For 3-SAT paired evaluation, the setup uses $F_i^{SAT}$1 with 70 SAT/UNSAT pairs per $F_i^{SAT}$2 at low clause density $F_i^{SAT}$3. Under this protocol, GPT-5 has $F_i^{SAT}$4 for $F_i^{SAT}$5 and drops to $F_i^{SAT}$6 at $F_i^{SAT}$7. DeepSeek-reasoner has $F_i^{SAT}$8 at $F_i^{SAT}$9, about $F_i^{UNSAT}$0 at $F_i^{UNSAT}$1, and below $F_i^{UNSAT}$2 by $F_i^{UNSAT}$3. Claude-opus-4 has $F_i^{UNSAT}$4 at $F_i^{UNSAT}$5 but collapses to approximately $F_i^{UNSAT}$6 at $F_i^{UNSAT}$7, and approximately $F_i^{UNSAT}$8 for larger $F_i^{UNSAT}$9 (Zhang et al., 27 May 2026).

The paper emphasizes that ADR in this setting does not show trivial easy-hard-easy curves. Because it is computed over balanced pairs, it instead exhibits a smooth monotonic decay with $\hat y(F) \in \{SAT, UNSAT\}$0, which the study interprets as faithfully capturing scale-induced reasoning breakdown. This is an important contrast with phase-transition analyses based on unpaired data, where benchmark composition can dominate the metric (Zhang et al., 27 May 2026).

For 2-SAT paired evaluation, the setup uses $\hat y(F) \in \{SAT, UNSAT\}$1 with 70 SAT/UNSAT pairs per $\hat y(F) \in \{SAT, UNSAT\}$2. GPT-5 retains $\hat y(F) \in \{SAT, UNSAT\}$3 up to $\hat y(F) \in \{SAT, UNSAT\}$4. DeepSeek-reasoner reaches $\hat y(F) \in \{SAT, UNSAT\}$5 at $\hat y(F) \in \{SAT, UNSAT\}$6. Claude-opus-4 crashes to near $\hat y(F) \in \{SAT, UNSAT\}$7 by $\hat y(F) \in \{SAT, UNSAT\}$8 (Zhang et al., 27 May 2026).

The paper compares these 2-SAT results with 3-SAT and notes that all models achieve higher ADR on 2-SAT, a P-time fragment, but that scaling still exposes weaker models. This suggests that the paired evaluation is sensitive both to formal problem class and to model-specific robustness under increasing instance size (Zhang et al., 27 May 2026).

5. Transfer across reductions and representations

Beyond CNF, the study tests cross-representation consistency by converting CNF to Vertex Cover and 3-SAT to discrete 3D packing. For Vertex Cover, the reduction is the standard CNF-to-graph construction yielding a vertex-cover of size $\hat y(F) \in \{SAT, UNSAT\}$9 if and only if the original CNF is satisfiable. Across all models, 75.3% of instance-level predictions agree between the SAT and VC representations. At $$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$0, ADR for VC is approximately $$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$1 for GPT-5, compared with $$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$2 on SAT, and approximately $$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$3 for DeepSeek (Zhang et al., 27 May 2026).

For discrete 3D packing, each clause becomes a packing constraint in a small 3D grid, with satisfiability corresponding to feasible placement. The paper reports 80.2% concordance between SAT and packing decisions. ADR for packing is slightly lower for GPT-5 than for SAT, while non-reasoners remain near zero. The study also reports that higher ADR correlates with actual valid packings when the model predicts “YES” (Zhang et al., 27 May 2026).

These transfer results are used to probe representation-invariant reasoning. The paper states that model decisions on CNF and on the corresponding graph or packing instances agree for most models on more than 80 percent of instances, suggesting stable decision rules across representations. At the same time, the lower ADR values after reduction indicate that representation consistency and strong paired discrimination are not identical properties (Zhang et al., 27 May 2026).

6. Discriminative role and interpretive significance

The paper includes a simple example at $$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$4 for 3-SAT pairs that contrasts Accuracy and ADR:

Model	Accuracy	ADR
GPT-5	0.88	0.79
DeepSeek	0.60	0.45
Claude-4	0.53	0.12

The point of this comparison is that Accuracy suggests GPT-5 > DeepSeek > Claude, but does not reflect that DeepSeek’s joint-correct rate is far below GPT-5’s, and that Claude-4 is effectively guessing because $$A_i = 1[\hat y(F_i^{SAT}) = SAT], \qquad B_i = 1[\hat y(F_i^{UNSAT}) = UNSAT].$$5 (Zhang et al., 27 May 2026).

The study’s conclusion is that ADR, computed over minimally edited SAT/UNSAT pairs, filters out trivial biases and directly measures an LLM’s ability to “notice” the critical logical difference induced by a single clause or literal change. Empirically, it is reported to cleanly order models by genuine reasoning strength, remain well-behaved under class balance, and scale monotonically with problem size. On that basis, the paper states that ADR outperforms accuracy, precision, recall, F1, and MCC as a faithfulness-to-reasoning metric and extends naturally across NP reductions such as Vertex Cover and 3D packing, making ADR a robust, representation-agnostic yardstick for LLM logical competence (Zhang et al., 27 May 2026).

A plausible implication is that ADR is best understood not as a replacement for all task-level metrics, but as a conservative probe of reasoning in settings where SAT structure is latent and where minimally edited counterfactual instance pairs can be constructed. That reading is consistent with the study’s broader claim that SAT is a conservative probe for LLM reasoning and that paired evaluation with ADR provides a more faithful and representation-robust assessment than conventional metrics (Zhang et al., 27 May 2026).