NevIR Dataset for Negation in Neural IR

• NevIR dataset is a controlled benchmark of negation handling, using contrastive document pairs with strict query constraints to test neural IR models.
• The evaluation methodology employs paired accuracy to measure semantic sensitivity to negation, highlighting limitations in sparse, bi-encoder, and cross-encoder architectures.
• Empirical results reveal that even fine-tuned, large-scale models struggle with negation cues, underscoring a trade-off between improved specificity and general retrieval performance.

The NevIR dataset is a specialized resource for evaluating the ability of neural information retrieval (IR) models to handle negation, a linguistic feature that presents significant challenges for both classical and neural IR architectures. Developed within the context of modern IR’s heavy reliance on LLMs, NevIR provides a controlled benchmark for investigating how negation affects document relevance ranking. The dataset, its construction methodology, benchmark protocols, and empirical impact on state-of-the-art IR models have shaped subsequent research on negation-aware retrieval (Weller et al., 2023, Elsen et al., 19 Feb 2025).

1. Construction and Structure of the NevIR Dataset

The NevIR dataset is constructed from contrastive document pairs curated from the CondaQA dataset, ensuring that each pair differs exclusively in a key negation while remaining lexically and contextually aligned. Paraphrastic edits are excluded to maintain semantic contrast focused solely on negation phenomena. On average, documents are about 113 words in length, with an average difference of only 4 words per pair, as shown by the dataset’s detailed word difference distribution.

Crowdsourced queries complement each document pair, generated under strict constraints:

• Both queries target equivalent answers (span-based).
• Sufficient context is provided to uniquely associate the query with its correct document.
• Annotators are instructed not to use any word that is exclusive to one document of the pair, eliminating trivial lexical cues and enforcing reliance on semantic processing of negation.

This process yields a dataset of 2,556 document pairs, each with two carefully crafted queries, where the only substantive difference is the presence or absence (or scope) of negation.

2. Benchmarking Protocol and Evaluation Metrics

NevIR uses a paired (contrastive) ranking setup to accurately assess a model’s negation-sensitivity. Models must rank two nearly identical documents in opposite order when presented with queries that are themselves minimally different, differing only by negation.

The principal metric is pairwise accuracy: a model is credited only if, for both targeted queries, it ranks the correct document above its contrastive pair—the necessary criterion being that it must invert the ranking under negation. This strict evaluation addresses the common issue where models that ignore negation may keep the same preferred document regardless of query polarity, yielding artificially inflated accuracy on standard ranking metrics.

Traditional IR metrics for general tasks are thus supplanted by this pairwise accuracy, which directly measures semantic sensitivity to negation.

3. Model Performance on NevIR

Performance on NevIR reveals stark limitations across the spectrum of neural IR architectures. Key experimental outcomes include:

Model Type	Example Models	Paired Accuracy (%)
Sparse Models	TF-IDF, SPLADEv2++	2–5
Bi-Encoders	DPR, coCondenser	5–12
Late Interaction Models	ColBERTv1, ColBERTv2	13–20
Cross-Encoders	MonoT5-base, MonoT5-3B	35–51
Random Ranking Baseline	—	25

Despite strong results on non-negation benchmarks (e.g., MS MARCO), state-of-the-art neural IR models—including cross-encoders—generally perform at, or often below, the random ranking baseline on NevIR (Weller et al., 2023). Only the largest cross-encoders, such as MonoT5-3B, approach 50% accuracy, far from the 100% achieved by human annotators.

4. Negation Sensitivity and Architectural Analysis

Detailed analytic results show that most neural IR models largely ignore negation cues:

• In ColBERT, for example, the MaxSim operator disregards tokens contributing to negation (“not”), instead attending to other contextually frequent terms, leading to failures on contrastive negation pairs.
• Bi-encoders and sparse models, which encode documents and queries independently, display particularly poor performance, indicating their representations lack the nuance required for semantic polarity.
• Cross-encoders, which jointly process queries and documents, perform comparatively better, but even the best models reveal a substantial gap relative to human-level negation comprehension.

This architectural disparity illustrates a fundamental weakness in how current IR models process logical and semantic operators like negation.

5. Model Size, Fine-Tuning, and Impact on Generalization

Extensive experiments with fine-tuning reveal that exposure to contrastive negation training data boosts pairwise accuracy across all model families, with larger models (e.g., MonoT5-3B) gaining the most. Nevertheless, the improvement comes with important trade-offs:

• Even with fine-tuning, the top models do not close the gap with human performance.
• Fine-tuning on negation-centric data may diminish effectiveness on generic IR tasks such as those evaluated by MS MARCO, emphasizing a precision–generalization trade-off that practitioners must weigh.
• These results confirm that size and architecture are not alone sufficient for robust negation understanding—targeted data and training are essential.

6. Reproducibility and Extension to Broader Negation Benchmarks

Reproducibility studies confirm the core findings on the NevIR dataset: notwithstanding architectural enhancements, the majority of retrieval systems remain insensitive to negation distinctions (Elsen et al., 19 Feb 2025). Beyond NevIR, the ExcluIR dataset introduces exclusionary queries to probe negation in broader formulations. However, experiments demonstrate that fine-tuning on NevIR does not reliably transfer to ExcluIR and vice versa, signaling a lack of robust cross-dataset generalization.

Evaluation with listwise LLM re-rankers (e.g., RankGPT, GPT-4o) further underscores the point that only the largest LLMs, especially when deployed in listwise re-ranking protocols, realize substantial improvements on NevIR, achieving up to 77.3% pairwise accuracy. Nonetheless, these strategies impose considerable computational cost and remain below human-level accuracy.

7. Implications for Neural IR Research

The NevIR dataset highlights persistent limitations in neural IR regarding logical semantics. Despite improvements from specialized fine-tuning and increased model capacity, negation remains a significant challenge. The research indicates:

• Cross-encoder and large listwise LLM re-rankers are necessary but not sufficient for robust negation handling.
• Generalization across distinct negation benchmarks is limited, suggesting the need for comprehensive and diverse training corpora.
• The design of evaluation metrics such as pairwise accuracy is critical to accurately quantify semantic understanding.
• Pragmatic deployment must consider the trade-offs between model size, computational feasibility, negation sensitivity, and overall retrieval quality.

The NevIR resource, coupled with continued advances in contrastive benchmarking and negation-specific training strategies, is central to ongoing efforts in developing semantically robust IR systems. This line of inquiry bears particular significance for high-stakes domains—such as medical, legal, and scientific retrieval—where failures to account for negation can have material consequences.