---
title: NLI-Based Logical Consistency Score (NLICS)
url: https://www.emergentmind.com/topics/nli-based-logical-consistency-score-nlics
type: topic
---

# NLI-Based Logical Consistency Score (NLICS)

NLI-based Logical Consistency Score (NLICS) is an NLI-grounded measure of whether a model output is logically supported by specified semantic constraints. In the NLI Constraint Verification (NLICV) framework for LLM personalization, NLICS is tied directly to two entailment probabilities—correctness and alignment—and to a categorical decision over four behavioral modes: personalization, generalization, sycophancy, and failure. The formulation is explicitly grounded in possible-world semantics, where sentence meanings are treated as truth-condition sets and constraint satisfaction is operationalized by a discriminative NLI model with thresholded entailment scores [2606.16368].

## 1. Conceptual definition

Within NLICV, the central object is a personalized response $r$ to a query $q$, conditioned on a user profile $u$ and retrieved user information $i = \mathrm{RAG}(q,u)$. The framework introduces a latent answer space $a = \mathrm{Answer}(q)$, representing all valid answers to $q$, and imposes two output-stage constraints: $\mathrm{Correct}(r,a)$ and $\mathrm{Aligned}(r,i)$ [2606.16368].

The score itself is defined from the NLI entailment probabilities attached to those two constraints. Let
- $S_c = S_{\mathrm{ent}}(r,a)$ be the correctness score,
- $S_a = S_{\mathrm{ent}}(r,i)$ be the alignment score,

where $S_{\mathrm{ent}}(P,H)=\Pr(P \vDash H)$ is the model-predicted entailment probability. NLICS is then the ordered pair

$$
\mathrm{NLICS}(r;a,i) = (S_c, S_a),
$$

together with a category decision determined by thresholded indicators $I_c = \mathbf{1}[S_c \ge \tau_{\mathrm{sem}}]$ and $I_a = \mathbf{1}[S_a \ge \tau_{\mathrm{sem}}]$, with $\tau_{\mathrm{sem}} = 0.5$ in experiments [2606.16368].

Logical consistency with personalization constraints corresponds to the joint entailment requirement $\mathrm{Correct} \land \mathrm{Aligned}$, which is the category labeled *Personalization* at threshold $\tau_{\mathrm{sem}}$ [2606.16368].

## 2. Truth-conditional semantics and constraint structure

The formal semantics of NLICS are stated in terms of possible worlds. Let $W$ be the set of possible worlds, and let a proposition have type $\langle s,t\rangle$, i.e., a function from worlds to truth values. For a sentence $S$, semantic interpretation is written as $\llbracket S \rrbracket : W \to \{1,0\}$, where $\llbracket S \rrbracket(w)=1$ iff $S$ is true in world $w$. The associated truth-condition set is

$$
\mathcal{T}(S)=\{w \in W \mid \llbracket S \rrbracket(w)=1\}.
$$

Entailment is defined as set inclusion:

$$
S_1 \vDash S_2 \iff \mathcal{T}(S_1)\subseteq \mathcal{T}(S_2).
$$

This same inclusion structure is used to define the personalization constraints. Correctness is

$$
\mathrm{Correct}(r,a) \iff \llbracket r \rrbracket \subseteq \llbracket a \rrbracket,
$$

and alignment is

$$
\mathrm{Aligned}(r,i) \iff \llbracket r \rrbracket \subseteq \llbracket i \rrbracket.
$$

Jointly, the response must satisfy

$$
\llbracket r \rrbracket \subseteq (\llbracket a \rrbracket \cap \llbracket i \rrbracket).
$$

The broader personalization process is written as

$$
(\mathrm{Interested}(u,i)\wedge \mathrm{Related}(q,i)) \to (\mathrm{Correct}(r,a)\wedge \mathrm{Aligned}(r,i)).
$$

A personalized response is specified by maximizing alignment subject to a strict correctness threshold:

$$
r^* = \arg\max_{r \in \mathcal{R}} \mathrm{Aligned}(r,i)
\quad \text{s.t.} \quad \mathrm{Correct}(r,a)\ge \tau_{\mathrm{correct}}.
$$

This semantics gives NLICS a precise interpretation: it is not a lexical similarity score, but a truth-conditional proxy for whether a response stays within the admissible worlds determined by the answer space and the user-specific information [2606.16368].

## 3. NLI operationalization and behavioral regimes

NLICV maps the semantic constraints into premise–hypothesis NLI checks. Entailment remains the operational surrogate for set inclusion:

$$
P \vDash H \iff \mathcal{T}(P)\subseteq \mathcal{T}(H).
$$

For output evaluation, the framework performs two discriminative NLI passes per item. The correctness check uses $(P,H)=(r,a)$ and computes $S_c=S_{\mathrm{ent}}(r,a)$. The alignment check uses $(P,H)=(r,i)$ and computes $S_a=S_{\mathrm{ent}}(r,i)$. A constraint is satisfied iff the corresponding entailment probability is at least $\tau_{\mathrm{sem}}$, which is fixed to $0.5$ in the reported experiments [2606.16368].

The resulting binary decisions define a four-way Personalization Confusion Matrix.

| Category | Condition |
|---|---|
| Personalization | $\mathrm{Correct} \land \mathrm{Aligned}$ |
| Generalization | $\mathrm{Correct} \land \neg \mathrm{Aligned}$ |
| Sycophancy | $\neg \mathrm{Correct} \land \mathrm{Aligned}$ |
| Failure | $\neg \mathrm{Correct} \land \neg \mathrm{Aligned}$ |

This partition is a distinguishing feature of NLICS as used in NLICV. Rather than collapsing evaluation into a single binary judgment, it separates factual validity relative to the answer space from user-specific alignment, thereby exposing cases where a response is factually correct but unpersonalized, or aligned to user preferences but factually wrong [2606.16368].

Across datasets or instances, the framework aggregates $S_c$ and $S_a$ as averages per behavioral regime. These aggregated constraint scores serve as corpus-level NLICS summaries for each regime [2606.16368].

## 4. Evidence attribution and semantic interpretability

NLICV supplements scoring with an ablation-based interpretability procedure. Premise and hypothesis are decomposed into atomic propositions,

$$
P=\{p_\theta\}_{\theta=1}^m,\qquad H=\{h_\phi\}_{\phi=1}^n,
$$

with

$$
\llbracket P \rrbracket = \bigcap_{\theta=1}^m \llbracket p_\theta \rrbracket,\qquad
\llbracket H \rrbracket = \bigcap_{\phi=1}^n \llbracket h_\phi \rrbracket.
$$

The paper states a monotonicity principle: adding propositions shrinks admissible worlds, and removing propositions expands them [2606.16368].

First-order ablation tests premise necessity and hypothesis criticality. A premise atom $p_\theta$ is critical evidence if removing it breaks entailment to $H$, formally when $\llbracket P_{-\theta}\rrbracket \nsubseteq \llbracket H\rrbracket$, or operationally when $S_{\mathrm{ent}}(P_{-\theta},H)\le \tau_{\mathrm{sem}}$ while $S_{\mathrm{ent}}(P,H)>\tau_{\mathrm{sem}}$. A hypothesis atom $h_\phi$ is a critical constraint unit if removing it breaks the entailment of $H$ by $H$ itself, formally $\llbracket H_{-\phi}\rrbracket \nsubseteq \llbracket H\rrbracket$, or operationally when $S_{\mathrm{ent}}(H_{-\phi},H)\le \tau_{\mathrm{sem}}$ [2606.16368].

Second-order ablation identifies evidence–constraint pairs. The pair $(p_\theta,h_\phi)$ is an evidence-constraint pair if

$$
\llbracket P_{-\theta}\rrbracket \nsubseteq \llbracket H\rrbracket
\quad \text{and} \quad
\llbracket P_{-\theta}\rrbracket \subseteq \llbracket H_{-\phi}\rrbracket,
$$

or operationally if

$$
S_{\mathrm{ent}}(P_{-\theta},H)\le \tau_{\mathrm{sem}}
\quad \text{and} \quad
S_{\mathrm{ent}}(P_{-\theta},H_{-\phi})> \tau_{\mathrm{sem}}.
$$

The intended interpretation is explicit in the paper: this shows that $p_\theta$ is indispensable specifically for satisfying $h_\phi$, and thus yields faithful, understandable rationales identifying which sentences drive verification [2606.16368].

The worked examples illustrate both semantic equivalence and personalized reasoning. In one example, the gold label “The answer is [2].” and the response “[2] is correct.” are true in the same worlds, so $\mathcal{T}(S_g)=\mathcal{T}(S_r)$ and correctness holds. In another, a response about “A low-cost wear-leveling algorithm for block-mapping solid-state disks” entails the preference “Hardware Characterization,” so alignment holds. A history-domain case study further shows how ablation isolates evidence for both correctness and alignment when a response connects Liangzhu burial artifacts to economics through resource allocation and inequality [2606.16368].

## 5. Empirical behavior, robustness, and efficiency

The reported NLICV experiments use discriminative NLI backbones including DeBERTa-v3 Large and Small, MiniLMv2, DistilRoBERTa, and RoBERTa, implemented in Python on NVIDIA H100 NVL GPUs. Evaluation covers two main settings: semantic invariance on LaMP benchmark tasks—Citation Identification, Movie Tagging, and Product Rating—with synonym-based lexical perturbations from rephrase rates $0.0$ to $1.0$; and reference-free personalization verification on MMLU with 10 user topic profiles and four response regimes per query [2606.16368].

On semantic invariance and human alignment, DeBERTa-Large achieves peak mean accuracy $98.00\%$ with standard deviation $1.08\%$ across rephrase rates, maintaining robustness to semantic-preserving lexical changes. By contrast, BLEU, ROUGE, and embedding similarity hover near random performance, around $50\%$ accuracy, and degrade under perturbation. The paper additionally notes that negative instances can receive spuriously high scores at zero rephrase, with BLEU Neg approximately $0.83$ [2606.16368].

Constraint scores by regime make the decomposition of NLICS concrete. For DeBERTa-Large, illustrative entries are:
- Generalized: $S_c=0.96$, $S_a=0.03$,
- Personalized: $S_c=0.58$, $S_a=0.68$,
- Sycophancy: $S_c=0.18$, $S_a=0.68$,
- Failure: $S_c=0.18$, $S_a=0.03$.

The framework also documents miscalibration in LLM judges. Qwen3-8B, for example, yields $S_a=0.62$ on Generalized instances that are factually correct but unaligned, thereby conflating correctness and alignment [2606.16368].

In the MMLU personalization-verification setting with a one-hour/user budget, DeBERTa-Large reaches mean accuracy $68.44\%$, with per-item average latency $0.07$ seconds and zero token generation cost. LLM judges remain in the approximately $27$–$33\%$ mean-accuracy range, with per-item average latency from $20.97$ to $228.72$ seconds and total tokens in the millions; Qwen3-8B is reported at $8{,}206{,}940$ total tokens. The resulting speed and cost profile is summarized by the paper as up to $2100\times$ inference speedup over LLM-as-a-judge baselines [2606.16368].

The comparative conclusion stated in the paper is twofold. First, surface-matching metrics do not preserve semantic invariance. Second, LLM judges can miss the separation between correctness and personalization, producing near-zero accuracy in the Generalized category and weak detection of Sycophancy. NLICV’s NLICS formulation is presented as semantically invariant, interpretable, and computationally efficient under these conditions [2606.16368].

## 6. Related formulations and limitations

NLICS is not a single cross-task formula in the broader literature. Related work instantiates NLI-based logical consistency scores differently depending on the evaluation target. In multi-turn dialogue consistency, D-SMART defines a turn-level Consistency Score

$$
CS_i = \frac{(P_{E_i}-P_{C_i})+1}{2},
$$

with dialogue-level averaging and a complementary Dialogue Entailment Rate; the paper explicitly states that the principal metric corresponding to NLICS is $CS$ [2510.13363]. For set-consistency verification, SC-Energy maps a set-level energy $E_\theta(S)$ to a calibrated consistency score such as $\sigma(-(E_\theta(S)-\mu)/\tau)$, again presented as an NLICS-style formulation for global logical coherence over statement sets [2503.10695]. In summarization, SummaC defines a document-level score by segmenting source and summary into sentences, taking the maximum entailment per summary sentence, and averaging; SLiC-NLI instead uses the summary-level entailment probability $p_e(y,x)$ as the consistency signal for calibration of sequence likelihoods [2111.09525] [2310.08764]. ConCoRD, by contrast, defines NLICS as a weighted fraction of satisfied NLI-derived constraints over a factor graph of answer candidates [2211.11875]. This suggests that the term functions as a family of NLI-grounded consistency measures rather than a universally fixed scalar.

The limitations of NLICV itself are explicitly tied to the NLI backbone. The framework inherits limitations of current DeBERTa and RoBERTa NLI models: implicit entailment, multi-hop reasoning, and long-context dependencies can reduce verification accuracy. Domain generalization depends on NLI model robustness, and ambiguous or implicit constraints can challenge entailment decisions. The reported output-stage evaluation focuses on $\mathrm{Correct}$ and $\mathrm{Aligned}$; input-stage constraints $\mathrm{Interested}$ and $\mathrm{Related}$ are assumed satisfied in output-only evaluation [2606.16368].

A common misconception is that any high semantic similarity score constitutes logical consistency. The NLICV results reject that equation: semantically equivalent paraphrases can be mis-scored by BLEU, ROUGE, or embedding similarity, while generalized responses can be incorrectly treated as aligned by LLM judges. In the NLICV formulation, logical consistency is narrower and more technical: it is the joint entailment of validity with respect to the answer space and alignment with retrieved user information, expressed through $(S_c,S_a)$ and resolved categorically at the entailment threshold [2606.16368].

Source: https://www.emergentmind.com/topics/nli-based-logical-consistency-score-nlics