Negation Effect Score (NES) in NLP

• Negation Effect Score is a contrastive metric that quantifies changes in next-token prediction between paired affirmative and negated inputs.
• It aggregates log-probability differences over aligned targets, isolating polarity cues without being a full measure of language understanding.
• The metric is used as a causal diagnostic tool through interventions like activation patching and ablation to uncover mid-layer attention-head contributions in GPT-2 Small.

Searching arXiv for the cited paper and closely related negation-evaluation work.

Negation Effect Score (NES) is a contrastive behavioral metric for quantifying how a LLM’s next-token distribution changes when an input prefix is transformed from an affirmative form to a matched negated form. In “Interpreting Negation in GPT-2: Layer- and Head-Level Causal Analysis,” NES is the central quantity used to test whether GPT-2 Small distinguishes affirmative contexts from their negated counterparts at the point where it predicts a target continuation. The metric is defined on minimally different affirmative/negated sentence pairs, so that the polarity cue is the principal controlled difference rather than a byproduct of broader lexical or syntactic variation (Mofael et al., 12 Mar 2026).

1. Formal definition and conceptual scope

For a paired example, GPT-2 Small is evaluated on the same target token $t$ under an affirmative prefix $a$ and a matched negated prefix $n$. NES is defined as

$$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$

where $P_{\theta}$ is the next-token probability distribution produced by GPT-2 with parameters $\theta$, $t$ is the target token whose likelihood is scored, $a$ is the affirmative prefix, and $n$ is the matched negated prefix. In implementation, the paper extends this scoring to multi-token targets by summing token-level log-probabilities across the target span, so the operational quantity is still a log-probability contrast, but over an aligned span rather than necessarily a single token (Mofael et al., 12 Mar 2026).

The design is deliberately narrow. NES is not introduced as a generic task-accuracy metric, and it is not intended to summarize full-sentence understanding or free-form generation quality. Its purpose is to isolate sensitivity to logical polarity at a single next-token prediction site. Because the affirmative and negated inputs are paired and differ only in the negation cue, NES measures whether the model reweights probability mass in the appropriate direction when polarity changes. This suggests that NES is best understood as a local polarity-sensitive contrast score rather than a global semantic competence measure.

A central implication of this formulation is that NES is inherently relational. It has meaning only with respect to a matched affirmative/negated pair and a shared prediction target. The metric therefore belongs to the family of contrastive probes rather than standalone correctness measures.

2. Interpretation, sign convention, and aggregation

The sign convention is essential. Because NES is $\log P(t\mid a)-\log P(t\mid n)$, a negative value means that the target is more likely under the negated prefix than under the affirmative prefix. The paper explicitly interprets this as good negation sensitivity: the model has responded to the logical reversal. A positive value means the model still favors the affirmative continuation, which the authors treat as weaker negation sensitivity or failure. A value near zero indicates little distinction between affirmative and negated contexts (Mofael et al., 12 Mar 2026).

The paper therefore uses three interpretive regions. Negative NES means the model “responds correctly to the logical reversal.” Positive NES means the model “favors the affirmative continuation.” $a$0 means the model shows little polarity distinction. Since NES is a log-probability difference, it is not bounded to a fixed numerical range; what matters is direction and magnitude. More negative values indicate stronger preference for the target under negation, while more positive values indicate stronger affirmative bias.

Aggregation is performed at multiple levels. NES is computed for each example and then aggregated within every semantic template. For each template, the paper reports mean NES, median NES, failure rate $a$1, and 95% confidence intervals estimated using $a$2. The failure criterion is intentionally strict: any example with $a$3 is counted as a failure, even if the model’s top-1 continuation might still be linguistically acceptable. NES is therefore a strict polarity-robustness measure rather than a synonym for task accuracy (Mofael et al., 12 Mar 2026).

The paper also defines intervention-relative shifts. For a negation form $a$4, the form-specific ablation shift is

$a$5

For layerwise activation patching, the shift is

$a$6

During such interventions, increases in $a$7NES indicate weaker negation sensitivity, that is, movement toward affirmative drift; decreases indicate stronger negation sensitivity. This relative interpretation is necessary because baseline negative NES is desirable, whereas an intervention-induced increase in NES is undesirable even if the resulting score remains negative.

3. Data, alignment, and computation protocol

The empirical basis for NES in the GPT-2 analysis is a self-curated dataset of 12,000 matched sentence pairs spanning eight semantic templates: capital_of, can_ability, likes, is_a_job, color_is, has_object, in_container, and drives_vehicle. The dataset also spans seven negation forms: not, never, no, does not, doesn’t, can’t, and cannot. For the causal experiments, the paper focuses mainly on the can_ability subset (Mofael et al., 12 Mar 2026).

That subset contains 1,523 examples over five forms and is equalized to 268 items per form, yielding 1,340 pairs in total. It is split into 938 development examples for head analysis and 402 test examples for ablation and verification. Additional capital_of and is_a_job subsets are reserved for template generalization. For external validation, the paper also uses xNot360, converting each natural affirmative/negated pair into the same prefix-target format using tokenizer-based alignment so that both contexts share the same scored target (Mofael et al., 12 Mar 2026).

The relevant model output is always the next-token distribution at the final prefix token. This detail constrains the scope of NES: the metric is computed at one aligned prediction site rather than over an entire generated sequence. For activation patching, the cached/intervened representation is the post-attention projection vector corresponding to the last token of the prefix. For multi-token targets, log-probabilities are summed across the span. The model is frozen GPT-2 Small in evaluation mode, the native tokenizer is used, fixed random seeds are applied across Python, NumPy, and PyTorch, and outputs are stored as versioned CSVs (Mofael et al., 12 Mar 2026).

This computation protocol makes prefix-target alignment a structural prerequisite of NES. A plausible implication is that NES is most reliable when target spans can be aligned cleanly across affirmative and negated contexts; if alignment is noisy, the score may confound polarity effects with tokenization or segmentation artifacts.

4. NES as a causal readout for internal negation processing

NES functions not only as a behavioral score but as the principal readout for mechanistic intervention. The first intervention is layerwise activation patching. The affirmative run’s post-attention projection vector at the last prefix token for layer $a$8 is cached and inserted into the corresponding position of the negated run. If this patch increases NES, affirmative internal evidence from that layer is pushing the negated example toward an affirmative interpretation. Large positive $a$9 values are therefore interpreted as evidence that the patched component carries polarity-relevant information (Mofael et al., 12 Mar 2026).

The layer influence profile peaks in the middle of the network, especially around layers 4–5, with weaker or mixed effects in deeper layers. At head level, the same logic is applied to individual attention-head output slices. A compact set of heads emerges as especially influential, with L5H11 and L4H4 singled out as major carriers of polarity information. More broadly, the influential heads are concentrated primarily within layers 4 to 6, supporting the paper’s claim that negation processing in GPT-2 Small is not diffusely distributed but localized in a mid-layer attention-head circuit (Mofael et al., 12 Mar 2026).

The second intervention family is ablation and rescue. For ablation, the selected top-$n$0 heads $n$1 are zeroed at the final prefix token during the negated run:

$n$2

For rescue, those same heads are set to cached affirmative activations:

$n$3

Ablation tests necessity. Rescue tests sufficiency, but in this paper rescue is not restoration of correct negation processing; it is deliberate reintroduction of affirmative evidence. That distinction is critical for interpreting NES shifts. On in-domain data, rescue increasing NES further than ablation is treated not as recovery failure but as confirmation that the targeted heads carry affirmative signal.

5. Empirical profile of NES in GPT-2 Small

The baseline template-level results show that GPT-2 Small often exhibits strong affirmative bias. Mean NES values are positive for most templates: capital_of $n$4, can_ability $n$5, likes $n$6, is_a_job $n$7, color_is $n$8, has_object $n$9, and in_container $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$0. Only drives_vehicle has a clearly negative mean NES, at $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$1. The associated failure rates are also high in most templates: capital_of $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$2, likes $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$3, is_a_job $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$4, color_is $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$5, can_ability $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$6, has_object $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$7, in_container $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$8, and drives_vehicle $$\mathrm{NES} = \log P_{\theta}(t \mid a) - \log P_{\theta}(t \mid n),$$9. Median NES values reinforce the same pattern, with drives_vehicle again the notable exception, having median $P_{\theta}$0 (Mofael et al., 12 Mar 2026).

These results show that NES is graded rather than binary. Capital_of and likes are especially poor. Drives_vehicle is the only template with clearly negative mean and median NES, indicating stronger baseline negation sensitivity there. The paper’s interpretation is that positive mean NES values reveal a failure to reverse preference under negation, not merely an occasional ranking mistake.

Cross-form analysis supports a degree of robustness of the identified mechanism. On held-out can_ability test data, ablating the key head set consistently increases NES across can’t, cannot, does not, doesn’t, and never. The paper summarizes this as positive $P_{\theta}$1NES across all morphological variants, indicating affirmative drift and weaker negation sensitivity across multiple negation forms rather than dependence on a single lexical cue. The authors also report pairwise Jaccard similarities between top-$P_{\theta}$2 head sets across forms, with closely related forms such as does not and doesn’t showing greater overlap, though no numeric Jaccard values are provided (Mofael et al., 12 Mar 2026).

External validation on xNot360 reveals smaller and partly distribution-dependent effects. The reported mean NES values are $P_{\theta}$3 at baseline, $P_{\theta}$4 after ablation, and $P_{\theta}$5 after rescue. Thus, unlike the in-domain setting, ablation slightly decreases NES and rescue restores performance above baseline. The authors interpret this as evidence that the same mid-layer circuit remains causally involved in negation on external data, but that its net behavioral role depends on distributional context. Their stated hypothesis is that in synthetic templates the heads mainly encode a strong prior for affirmative factual statements, whereas in xNot360’s richer natural contexts the same heads may instead amplify local lexical negation cues such as not or never (Mofael et al., 12 Mar 2026).

6. Limitations, interpretive caveats, and relation to adjacent metrics

The paper explicitly presents several limitations of NES. First, the metric is local: it measures next-token preference at a single aligned prefix position, not full-sentence meaning or generation quality. Second, it depends on careful prefix-target alignment; xNot360 required tokenizer-based alignment so that the same target could be scored in both contexts. Third, NES can reflect interactions with factual priors rather than pure logical competence. A positive score may arise because the model strongly favors affirmative factual continuations, and interventions may alter NES by weakening or strengthening that prior rather than by implementing a fully abstract negation operator. Fourth, rescue does not necessarily mean recovery of correct negation behavior, because rescue here explicitly reintroduces affirmative activations. Fifth, NES is stricter than top-1 accuracy, so it may classify some examples as failures even when the model would generate an acceptable continuation. Sixth, the synthetic-template versus xNot360 reversal shows that NES effects are distribution-sensitive (Mofael et al., 12 Mar 2026).

A broader literature-level caveat is terminological. NES is not a standardized acronym across negation research. “Negation-Instance Based Evaluation of End-to-End Negation Resolution” does not define a metric called Negation Effect Score; its closest concept is Negation-Instance Based Scoring, especially NIS$P_{\theta}$6 and NIS$P_{\theta}$7 (Sineva et al., 2021). Likewise, contrastive negation benchmarks such as NaN-NLI and ScoNe, and sentence-level evaluations such as Thunder-NUBench, are designed to isolate negation understanding, but they do so through benchmark structure and task accuracy rather than through a metric named NES (Truong et al., 2022, She et al., 2023, So et al., 17 Jun 2025). This suggests that NES, as used in the GPT-2 causal-analysis paper, is best treated as a specific mechanistic evaluation measure for polarity-sensitive next-token reweighting, not as a universal negation metric.

Within that narrower scope, NES plays four distinct roles simultaneously: baseline behavioral measure, intervention readout, failure criterion, and bridge from behavior to mechanistic localization. Its importance in the GPT-2 study lies precisely in this multifunctionality. By tracking whether interventions push the log-probability contrast upward or downward, the metric provides the causal signal used to argue that negation processing in GPT-2 Small is concentrated in a compact mid-layer attention-head subcircuit rather than distributed uniformly across the model (Mofael et al., 12 Mar 2026).