NLI-Based Logical Consistency Score (NLICS)

• NLICS is a measure that evaluates whether a model's response is logically consistent with specified semantic constraints using entailment probabilities.
• It operationalizes personalization by checking both correctness and alignment, categorizing responses into personalization, generalization, sycophancy, or failure.
• Empirical results show that NLICS offers robust, interpretable, and computationally efficient evaluation compared to traditional surface-matching metrics.

NLI-based Logical Consistency Score (NLICS) is an NLI-grounded measure of whether a model output is logically supported by specified semantic constraints. In the NLI Constraint Verification (NLICV) framework for LLM personalization, NLICS is tied directly to two entailment probabilities—correctness and alignment—and to a categorical decision over four behavioral modes: personalization, generalization, sycophancy, and failure. The formulation is explicitly grounded in possible-world semantics, where sentence meanings are treated as truth-condition sets and constraint satisfaction is operationalized by a discriminative NLI model with thresholded entailment scores (Li et al., 15 Jun 2026).

1. Conceptual definition

Within NLICV, the central object is a personalized response $r$ to a query $q$, conditioned on a user profile $u$ and retrieved user information $i = \mathrm{RAG}(q,u)$. The framework introduces a latent answer space $a = \mathrm{Answer}(q)$, representing all valid answers to $q$, and imposes two output-stage constraints: $\mathrm{Correct}(r,a)$ and $\mathrm{Aligned}(r,i)$ (Li et al., 15 Jun 2026).

The score itself is defined from the NLI entailment probabilities attached to those two constraints. Let

• $S_c = S_{\mathrm{ent}}(r,a)$ be the correctness score,
• $S_a = S_{\mathrm{ent}}(r,i)$ be the alignment score,

where $q$0 is the model-predicted entailment probability. NLICS is then the ordered pair

$q$1

together with a category decision determined by thresholded indicators $q$2 and $q$3, with $q$4 in experiments (Li et al., 15 Jun 2026).

Logical consistency with personalization constraints corresponds to the joint entailment requirement $q$5, which is the category labeled Personalization at threshold $q$6 (Li et al., 15 Jun 2026).

2. Truth-conditional semantics and constraint structure

The formal semantics of NLICS are stated in terms of possible worlds. Let $q$7 be the set of possible worlds, and let a proposition have type $q$8, i.e., a function from worlds to truth values. For a sentence $q$9, semantic interpretation is written as $u$0, where $u$1 iff $u$2 is true in world $u$3. The associated truth-condition set is

$u$4

Entailment is defined as set inclusion:

$u$5

This same inclusion structure is used to define the personalization constraints. Correctness is

$u$6

and alignment is

$u$7

Jointly, the response must satisfy

$u$8

The broader personalization process is written as

$u$9

A personalized response is specified by maximizing alignment subject to a strict correctness threshold:

$i = \mathrm{RAG}(q,u)$0

This semantics gives NLICS a precise interpretation: it is not a lexical similarity score, but a truth-conditional proxy for whether a response stays within the admissible worlds determined by the answer space and the user-specific information (Li et al., 15 Jun 2026).

3. NLI operationalization and behavioral regimes

NLICV maps the semantic constraints into premise–hypothesis NLI checks. Entailment remains the operational surrogate for set inclusion:

$i = \mathrm{RAG}(q,u)$1

For output evaluation, the framework performs two discriminative NLI passes per item. The correctness check uses $i = \mathrm{RAG}(q,u)$2 and computes $i = \mathrm{RAG}(q,u)$3. The alignment check uses $i = \mathrm{RAG}(q,u)$4 and computes $i = \mathrm{RAG}(q,u)$5. A constraint is satisfied iff the corresponding entailment probability is at least $i = \mathrm{RAG}(q,u)$6, which is fixed to $i = \mathrm{RAG}(q,u)$7 in the reported experiments (Li et al., 15 Jun 2026).

The resulting binary decisions define a four-way Personalization Confusion Matrix.

Category	Condition
Personalization	$i = \mathrm{RAG}(q,u)$8
Generalization	$i = \mathrm{RAG}(q,u)$9
Sycophancy	$a = \mathrm{Answer}(q)$0
Failure	$a = \mathrm{Answer}(q)$1

This partition is a distinguishing feature of NLICS as used in NLICV. Rather than collapsing evaluation into a single binary judgment, it separates factual validity relative to the answer space from user-specific alignment, thereby exposing cases where a response is factually correct but unpersonalized, or aligned to user preferences but factually wrong (Li et al., 15 Jun 2026).

Across datasets or instances, the framework aggregates $a = \mathrm{Answer}(q)$2 and $a = \mathrm{Answer}(q)$3 as averages per behavioral regime. These aggregated constraint scores serve as corpus-level NLICS summaries for each regime (Li et al., 15 Jun 2026).

4. Evidence attribution and semantic interpretability

NLICV supplements scoring with an ablation-based interpretability procedure. Premise and hypothesis are decomposed into atomic propositions,

$a = \mathrm{Answer}(q)$4

with

$a = \mathrm{Answer}(q)$5

The paper states a monotonicity principle: adding propositions shrinks admissible worlds, and removing propositions expands them (Li et al., 15 Jun 2026).

First-order ablation tests premise necessity and hypothesis criticality. A premise atom $a = \mathrm{Answer}(q)$6 is critical evidence if removing it breaks entailment to $a = \mathrm{Answer}(q)$7, formally when $a = \mathrm{Answer}(q)$8, or operationally when $a = \mathrm{Answer}(q)$9 while $q$0. A hypothesis atom $q$1 is a critical constraint unit if removing it breaks the entailment of $q$2 by $q$3 itself, formally $q$4, or operationally when $q$5 (Li et al., 15 Jun 2026).

Second-order ablation identifies evidence–constraint pairs. The pair $q$6 is an evidence-constraint pair if

$q$7

or operationally if

$q$8

The intended interpretation is explicit in the paper: this shows that $q$9 is indispensable specifically for satisfying $\mathrm{Correct}(r,a)$0, and thus yields faithful, understandable rationales identifying which sentences drive verification (Li et al., 15 Jun 2026).

The worked examples illustrate both semantic equivalence and personalized reasoning. In one example, the gold label “The answer is [2].” and the response “[2] is correct.” are true in the same worlds, so $\mathrm{Correct}(r,a)$1 and correctness holds. In another, a response about “A low-cost wear-leveling algorithm for block-mapping solid-state disks” entails the preference “Hardware Characterization,” so alignment holds. A history-domain case study further shows how ablation isolates evidence for both correctness and alignment when a response connects Liangzhu burial artifacts to economics through resource allocation and inequality (Li et al., 15 Jun 2026).

5. Empirical behavior, robustness, and efficiency

The reported NLICV experiments use discriminative NLI backbones including DeBERTa-v3 Large and Small, MiniLMv2, DistilRoBERTa, and RoBERTa, implemented in Python on NVIDIA H100 NVL GPUs. Evaluation covers two main settings: semantic invariance on LaMP benchmark tasks—Citation Identification, Movie Tagging, and Product Rating—with synonym-based lexical perturbations from rephrase rates $\mathrm{Correct}(r,a)$2 to $\mathrm{Correct}(r,a)$3; and reference-free personalization verification on MMLU with 10 user topic profiles and four response regimes per query (Li et al., 15 Jun 2026).

On semantic invariance and human alignment, DeBERTa-Large achieves peak mean accuracy $\mathrm{Correct}(r,a)$4 with standard deviation $\mathrm{Correct}(r,a)$5 across rephrase rates, maintaining robustness to semantic-preserving lexical changes. By contrast, BLEU, ROUGE, and embedding similarity hover near random performance, around $\mathrm{Correct}(r,a)$6 accuracy, and degrade under perturbation. The paper additionally notes that negative instances can receive spuriously high scores at zero rephrase, with BLEU Neg approximately $\mathrm{Correct}(r,a)$7 (Li et al., 15 Jun 2026).

Constraint scores by regime make the decomposition of NLICS concrete. For DeBERTa-Large, illustrative entries are:

• Generalized: $\mathrm{Correct}(r,a)$8, $\mathrm{Correct}(r,a)$9,
• Personalized: $\mathrm{Aligned}(r,i)$0, $\mathrm{Aligned}(r,i)$1,
• Sycophancy: $\mathrm{Aligned}(r,i)$2, $\mathrm{Aligned}(r,i)$3,
• Failure: $\mathrm{Aligned}(r,i)$4, $\mathrm{Aligned}(r,i)$5.

The framework also documents miscalibration in LLM judges. Qwen3-8B, for example, yields $\mathrm{Aligned}(r,i)$6 on Generalized instances that are factually correct but unaligned, thereby conflating correctness and alignment (Li et al., 15 Jun 2026).

In the MMLU personalization-verification setting with a one-hour/user budget, DeBERTa-Large reaches mean accuracy $\mathrm{Aligned}(r,i)$7, with per-item average latency $\mathrm{Aligned}(r,i)$8 seconds and zero token generation cost. LLM judges remain in the approximately $\mathrm{Aligned}(r,i)$9–$S_c = S_{\mathrm{ent}}(r,a)$0 mean-accuracy range, with per-item average latency from $S_c = S_{\mathrm{ent}}(r,a)$1 to $S_c = S_{\mathrm{ent}}(r,a)$2 seconds and total tokens in the millions; Qwen3-8B is reported at $S_c = S_{\mathrm{ent}}(r,a)$3 total tokens. The resulting speed and cost profile is summarized by the paper as up to $S_c = S_{\mathrm{ent}}(r,a)$4 inference speedup over LLM-as-a-judge baselines (Li et al., 15 Jun 2026).

The comparative conclusion stated in the paper is twofold. First, surface-matching metrics do not preserve semantic invariance. Second, LLM judges can miss the separation between correctness and personalization, producing near-zero accuracy in the Generalized category and weak detection of Sycophancy. NLICV’s NLICS formulation is presented as semantically invariant, interpretable, and computationally efficient under these conditions (Li et al., 15 Jun 2026).

6. Related formulations and limitations

NLICS is not a single cross-task formula in the broader literature. Related work instantiates NLI-based logical consistency scores differently depending on the evaluation target. In multi-turn dialogue consistency, D-SMART defines a turn-level Consistency Score

$S_c = S_{\mathrm{ent}}(r,a)$5

with dialogue-level averaging and a complementary Dialogue Entailment Rate; the paper explicitly states that the principal metric corresponding to NLICS is $S_c = S_{\mathrm{ent}}(r,a)$6 (Lei et al., 15 Oct 2025). For set-consistency verification, SC-Energy maps a set-level energy $S_c = S_{\mathrm{ent}}(r,a)$7 to a calibrated consistency score such as $S_c = S_{\mathrm{ent}}(r,a)$8, again presented as an NLICS-style formulation for global logical coherence over statement sets (Song et al., 12 Mar 2025). In summarization, SummaC defines a document-level score by segmenting source and summary into sentences, taking the maximum entailment per summary sentence, and averaging; SLiC-NLI instead uses the summary-level entailment probability $S_c = S_{\mathrm{ent}}(r,a)$9 as the consistency signal for calibration of sequence likelihoods (Laban et al., 2021, Zablotskaia et al., 2023). ConCoRD, by contrast, defines NLICS as a weighted fraction of satisfied NLI-derived constraints over a factor graph of answer candidates (Mitchell et al., 2022). This suggests that the term functions as a family of NLI-grounded consistency measures rather than a universally fixed scalar.

The limitations of NLICV itself are explicitly tied to the NLI backbone. The framework inherits limitations of current DeBERTa and RoBERTa NLI models: implicit entailment, multi-hop reasoning, and long-context dependencies can reduce verification accuracy. Domain generalization depends on NLI model robustness, and ambiguous or implicit constraints can challenge entailment decisions. The reported output-stage evaluation focuses on $S_a = S_{\mathrm{ent}}(r,i)$0 and $S_a = S_{\mathrm{ent}}(r,i)$1; input-stage constraints $S_a = S_{\mathrm{ent}}(r,i)$2 and $S_a = S_{\mathrm{ent}}(r,i)$3 are assumed satisfied in output-only evaluation (Li et al., 15 Jun 2026).

A common misconception is that any high semantic similarity score constitutes logical consistency. The NLICV results reject that equation: semantically equivalent paraphrases can be mis-scored by BLEU, ROUGE, or embedding similarity, while generalized responses can be incorrectly treated as aligned by LLM judges. In the NLICV formulation, logical consistency is narrower and more technical: it is the joint entailment of validity with respect to the answer space and alignment with retrieved user information, expressed through $S_a = S_{\mathrm{ent}}(r,i)$4 and resolved categorically at the entailment threshold (Li et al., 15 Jun 2026).