Negation Sensitivity Index (NSI)

• NSI is a quantitative metric that measures a model’s ability to interpret negation logically by comparing action endorsement across affirmative and negated prompts.
• It is computed using a systematic protocol involving four controlled prompt framings and polarity normalization to reveal sensitivity failures.
• Empirical analysis shows high NSI values, especially in open-source models, signal increased governance risk in ethical, high-stakes domains.

The Negation Sensitivity Index (NSI) is a quantitative metric developed to assess the robustness of LLMs and evaluation systems in correctly distinguishing between affirmative and negated statements, especially in ethical and safety-critical contexts. NSI operationalizes the degree to which a model’s action endorsement for statements like “do X” and “do not X” behaves as logical opposites. A perfectly robust model will have NSI=0, reflecting invariant and logically consistent responses across all polarity pairings; a model whose action endorsement swings widely under negation exhibits higher NSI, signaling greater governance risk in settings where prohibition enforcement is required (Elkins et al., 29 Jan 2026).

1. Theoretical Foundations and Formal Definition

The Negation Sensitivity Index is defined as the maximal difference in polarity-normalized action endorsement rates exhibited by a system when presented with a scenario under various negation and compound negation framings. The primary goal is to capture failure modes where a model misinterprets (or reverses) the logical intent of negation in user instructions or prompts.

Formally, for binary model decisions $d \in \{0,1\}$ (with $0 =$ disagree, $1 =$ agree), and four standardized prompt framings $F_0$ to $F_3$—each representing a controlled variant with or without negation or compound negation—the paper (Elkins et al., 29 Jan 2026) defines a polarity-normalized action endorsement variable $a$ as:

$a = d \oplus \mathbb{1}[\text{neg}(f)]$

where:

• $\oplus$ denotes exclusive OR,
• $\mathbb{1}[\text{neg}(f)]$ is the indicator function that is $1$ if $f$ is negated (i.e., $F_1$ or $F_3$), $0$ otherwise.

This mapping ensures $a=1$ always means a model endorses the underlying action $X$, regardless of the question’s polarity. The empirical mean $P^{(f)}_a$ for each framing $f$ provides a normalized action endorsement rate. NSI is then:

$$\text{NSI} = \max_{f} P^{(f)}_a - \min_{f} P^{(f)}_a$$

with $0 \leq \text{NSI} \leq 1$, where lower is better.

In the context of learned metrics (e.g., BLEURT), an essentially identical notion is used, where negation sensitivity is the mean absolute score change when a candidate sentence is negated versus its affirmative (e.g., $| m(r, s) - m(r, \lnot s) |$ averaged over test pairs) (Anschütz et al., 2023).

2. Computation Protocol and Empirical Methodology

The computation of NSI proceeds as follows (Elkins et al., 29 Jan 2026):

1. Design: For each scenario, instantiate four prompt framings that differ only in polarity or compound negation structure.
2. Binary Response Collection: Run the model across $N$ replicates for each frame ($N=30$ at $T=0.7$ commonly used in audits).
3. Polarity Normalization: Convert raw decisions to $a$ using logical polarity normalization.
4. Endorsement Rates: Compute $P^{(f)}_a$ for each framing.
5. Scenario-Level NSI: Calculate the NSI per scenario as the difference between the maximum and minimum $P^{(f)}_a$ across the four framings.
6. Aggregation: Average across scenarios for domain-level NSI and overall to obtain the global NSI.

This protocol is explicitly recommended as a pre-deployment audit tool to ensure robustness and interpretability in critical applications.

In sentence-level evaluation metrics (e.g., for machine translation), the quantity corresponding to NSI is computed by taking the mean over all sentence pairs of the absolute difference in metric score when the candidate is negated versus meaning-preserving, yielding a direct sensitivity measurement (Anschütz et al., 2023).

3. Empirical Findings and Model Rankings

The audit of 16 models across 14 ethical scenarios reveals pronounced variation:

• Open-source models: NSI values average 0.85–0.89, with systematic failures to invert compound negation (100% endorsement of prohibited action under F3). Under simple negation, these models still endorse prohibited actions 77% of the time—a 317% increase over baseline.
• Commercial models: Exhibit NSI values from near zero (Gemini-3-Flash, NSI=0.00) to moderate (GPT-5.1, NSI=0.20; Grok-4.1-reasoning, NSI=0.23). However, some commercial systems show swings of 19–128% under polarity shifts.

Empirical framing-level endorsement rates and aggregate NSI values for selected model classes:

Model Class	NSI (Global)	F0 ("should X")	F1 ("should NOT X")	F2 (Compound Affirm.)	F3 (Compound Neg.)
Open-Source	0.76–0.89	0.24	0.77	0.31	1.00
US Commercial	0.20–0.32	0.25	0.34	0.32	0.57
Chinese Comm.	0.23	0.37	0.21	0.39	0.44

Open-source models demonstrate "ceiling-effect" failures, unable to robustly invert negation in multi-framing tests.

4. Domain Sensitivity and Risk Stratification

NSI exhibits substantial domain-dependence. Financial, military, and business scenarios have highest mean NSI (0.63–0.65), while medical, science, and education domains are lower (0.36–0.38), reflecting increased fragility in high-stakes decision contexts. Agreement across models drops (74% to 62%) on negated prompts, and financial settings are twice as fragile as medical, suggesting higher governance risk for deployment in such domains.

To address this, the certification framework introduces domain-specific thresholds:

Domain Group	Tier A (Autonomous)	Tier B (Human Review)	Tier C (Human Confirm.)
Financial, Business, Military	< 0.10	0.10–0.35	≥ 0.35
Legal	< 0.15	0.15–0.40	≥ 0.40
Medical, Education, Science	< 0.20	0.20–0.49	≥ 0.50

Under these thresholds, virtually all open-source models require human confirmation in financial and business applications, regardless of claimed capabilities (Elkins et al., 29 Jan 2026).

5. Extension to Evaluation Metrics and Fine-Tuning

NSI-like quantities underpin negation sensitivity in learned evaluation metrics. For any metric $m$, sensitivity on a (reference $r$, candidate $s$) pair is $|m(r,s) - m(r, \lnot s)|$, averaged over the test set. In (Anschütz et al., 2023), BLEURT and a sentence-transformer (NegMPNet) are fine-tuned on a negation-augmented dataset (CANNOT) constructed from contradiction-heavy NLI, fact-checking, sentiment, and paraphrase corpora. The fine-tuned NegBLEURT and NegMPNet show substantial improvement:

• BLEURT-20 (pre-fine-tune): NSI ≈ 0.15
• NegBLEURT: NSI ≈ 0.93 (critical_negation, DEMETR); ≈0.65 (perturbation), a >0.5 increase
• all-mpnet-base-v2: NSI ≈ 0.14; NegMPNet: NSI ≈ 0.65

Importantly, fine-tuning for increased NSI does not degrade model sensitivity on other classes of perturbation, avoiding catastrophic forgetting (Anschütz et al., 2023).

6. Governance, Certification, and Recommendations

The NSI is proposed as a regulatory and operational metric for robust model governance:

• Pre-deployment Audit: All models intended for enforcement of prohibition or ethical filtering should be subjected to multi-framing protocol and have their NSI reported per relevant domain.
• Certification: A tiered framework is defined, mapping NSI to deployment permissions: Tier A (<0.20) permits autonomous action, Tier B (0.20–0.49) requires human review for prohibitions, Tier C (≥0.50) mandates human confirmation.
• Regulatory Adoption: Alignment with legal mandates (e.g., EU AI Act, NIST RMF) is recommended, including annual recertification and reporting of domain-specific NSI.

Practical recommendations include leveraging NSI as a training objective (negation-aware fine-tuning, multi-framing consensus augmentation), implementing real-time flagging mechanisms for high-NSI scenarios, and disclosing system limitations to users (“This system’s recommendations may vary by phrasing; for negative instructions, human review is advised”) (Elkins et al., 29 Jan 2026).

7. Significance and Outlook

The NSI provides a clear, interpretable scalar capturing a historically under-audited failure mode: the inability of LLMs and neural metrics to treat negation as logical inversion. High NSI signals a gap between current alignment protocols and robust, safe deployment, particularly in high-stakes domains. The adoption of NSI both as a research tool (for evaluation metric and model development) and as a governance standard can help close this gap and drive improvements both in system safety and in compliance verification.