ConTest-NLI: Robustness in Natural Language Inference

• The paper reveals that NLI models rely heavily on superficial patterns, with a 17.4% drop from standard SNLI to contrast-set performance.
• The study employs two ConTest-NLI formulations—contrast set via synonym substitution and construction-based templates—to probe shallow heuristics versus deep form-meaning mappings.
• Fine-tuning on contrast-enhanced data yields measurable gains, yet models continue to struggle with constructional generalization, highlighting persistent abstraction gaps.

ConTest-NLI designates robustness-oriented Natural Language Inference resources that use controlled variation to test whether high model performance reflects semantic competence rather than shallow heuristics. In the supplied literature, the term appears in two closely related formulations. One is a contrast-set framework built on SNLI, where hypotheses are perturbed through meaning-preserving synonym substitution to diagnose whether models truly understand entailment relations (Sanwal, 2024). The other is a later construction-informed benchmark, "Constructional Test Natural Language Inference," designed to probe LLMs’ generalization over English constructions and their form-meaning mappings (Mackintosh et al., 19 Sep 2025). In both formulations, the central question is whether models rely on genuine language comprehension or on surface pattern recognition.

1. Terminological scope and core variants

A recurring source of confusion is that ConTest-NLI does not refer to a single monolithic dataset in the supplied literature. Rather, it names two resources that share a contrastive evaluation philosophy but differ in linguistic target, generation pipeline, and evaluation regime.

Variant	Core design	Representative result
Contrast-set ConTest-NLI	SNLI-based contrast set generated by synonym substitution of verbs, adjectives, and adverbs in hypotheses, followed by manual review	ELECTRA-small: 89.9% on standard SNLI, 72.5% on the contrast set, 85.5% after contrast-enhanced fine-tuning
Construction-based ConTest-NLI	Benchmark of approximately 80,000 sentences across eight English constructions, generated via templating and a model-in-the-loop filter	Zero-shot accuracy drops from 88% on naturalistic data to 64% on adversarial data; fine-tuning yields up to 9% improvement

The SNLI-based formulation targets local lexical robustness under label-preserving perturbation. The construction-based formulation targets deeper abstraction: whether models learn constructional semantics across the substantive-schematic continuum. This suggests that the same label, ConTest-NLI, has been used to investigate two different failure modes in NLI systems: fragility to minimal paraphrastic change and failure of constructional generalization (Sanwal, 2024, Mackintosh et al., 19 Sep 2025).

2. Contrast-set ConTest-NLI over SNLI

In the contrast-set formulation, the purpose is to assess whether NLI models, specifically ELECTRA-small in the study, truly understand language entailment or merely latch onto surface patterns. The resource is built from the SNLI validation set by generating sentence pairs whose label should remain unchanged despite subtle lexical variation (Sanwal, 2024).

The pipeline operates on premise-hypothesis pairs, but the perturbation is applied to the hypothesis. Each hypothesis sentence is parsed using NLTK’s POS tagger. Only verbs, adjectives, and adverbs are considered for substitution; nouns are avoided to preserve core sentence meaning. For each identified word, NLTK’s WordNet is queried for synonyms, and a synonym is randomly selected to replace the original word. Although generation is automated, the resulting hypotheses are manually reviewed and slightly edited if necessary to ensure grammaticality and preservation of the main meaning and entailment relation. An illustrative example replaces “talks” with “babbles” in the hypothesis “A man talks on a cellphone,” while leaving the label unchanged (Sanwal, 2024).

The framework is formalized as a contrast-set function

$(p, h') = A(p, h, f),$

where $A$ takes a premise $p$ and hypothesis $h$ and generates a new pair $(p, h')$, possibly conditioned on a model $f$. Predictions on the contrast pair are evaluated using the model probabilities $f(p, h')$. The study defines a contrast cross-entropy error over $N$ instances as

$$\text{Contrast Error} = \frac{1}{N} \sum_{n=1}^N \text{CrossEntropy}(y_{nc}, P_{nc}),$$

with standard cross-entropy

$$\text{CrossEntropy} = - \sum_{c=1}^{C} y_c \log(p_c).$$

The paper’s explicit motivation is that Cross-Entropy Loss, while widely employed as a standard error metric in NLI, falls short in effectively evaluating a model’s capacity to understand language entailments (Sanwal, 2024).

3. Diagnostic findings and contrast-enhanced fine-tuning

The initial empirical result is a marked discrepancy between standard validation accuracy and contrast-set accuracy. ELECTRA-small achieves 89.9% on the conventional SNLI validation set but 72.5% on the SNLI-based contrast set, a performance gap of

$A$0

The paper interprets this as evidence that models are over-reliant on surface form and fragile to minor, meaning-preserving changes (Sanwal, 2024).

The study then constructs a contrast-enhanced training set by applying the same synonym-substitution procedure to a random subset, for example 1,000 samples, from the SNLI training data. ELECTRA-small is fine-tuned on original and contrast-augmented data and then re-evaluated on both original and contrast validation sets. After fine-tuning on the contrast-enhanced dataset, contrast-set accuracy improves to 85.5%, while original validation set accuracy remains stable. The reported learning curve shows that as the number of contrast training examples increases, contrast-set accuracy steadily increases, whereas standard validation accuracy plateaus, indicating improved robustness without catastrophic forgetting (Sanwal, 2024).

These findings support a narrower interpretation of what standard SNLI accuracy measures. High performance on the original benchmark does not guarantee true understanding when superficial patterns are broken. The paper therefore advocates integrating diverse linguistic expressions into NLI datasets and recommends incorporating contrast sets into the construction of NLI resources more generally. A plausible implication is that ConTest-NLI, in this formulation, is best understood not only as a test set but also as a framework for diagnosing and ameliorating robustness deficits under controlled label-preserving perturbation (Sanwal, 2024).

4. Constructional Test Natural Language Inference

A later use of the term introduces ConTest-NLI as a construction-informed benchmark for probing whether LLMs learn deep form-meaning mappings as defined by Construction Grammar. The benchmark contains approximately 80,000 sentences covering eight English constructions, ranging from highly lexicalized to highly schematic (Mackintosh et al., 19 Sep 2025).

The eight constructions are: Let Alone, Comparative Correlative, Caused Motion with Theme, Conative, Way Construction, Intransitive Motion, Caused Motion, and Resultative. They were selected for coverage across the substantive-schematic continuum and because several share similar syntactic forms while having distinct semantic entailments. Each construction includes at least 10,000 examples, systematically varied through at least eight templates and a large set of lexical fillers (Mackintosh et al., 19 Sep 2025).

Generation proceeds through hand-crafted templates, lexicalized slot filling, and filtering. For each construction, 8–12 templates encode canonical syntactic configurations, including variable word order, clause types, voice, adjunct positions, and optional modifiers. Template slots are populated with mid-frequency lemmas from the 20th–60th percentile in BookCorpus, with WordNet used for synonyms, hyponyms, and antonyms; lemminflect is used for automated morphological inflection; and balanced adverbial pools increase variation. For each premise, three hypotheses are generated, one each for entailment, neutral, and contradiction, using construction-specific semantic rules. A model-in-the-loop filter is then applied to cull obvious false positives and trivial cases (Mackintosh et al., 19 Sep 2025).

Quality control combines automatic filtering with manual validation. In a stratified sample of 100 triples, 99/100 examples correctly reflected the target construction and 94/100 NLI labels were deemed correct. The same analysis also identifies limitations, including some repetition of hypotheses across triples and occasional trivial entailments due to lexical overlap. Data splitting uses a lexeme-held-out 70/15/15 train/dev/test regime for each construction, ensuring that no verb lemma appears in both train/dev and test. Each construction yields approximately 4,000 triples, resulting in approximately 32,000 fully balanced NLI instances (Mackintosh et al., 19 Sep 2025).

5. Empirical profile of constructional generalization

The benchmark is used to evaluate LLaMA 3.1 8B-Instruct, Mistral 8B-Instruct, and GPT-4o, with both zero-shot and fine-tuned settings. The headline zero-shot result is a 24% drop in accuracy between naturalistic data, where GPT-4o reaches 88%, and adversarial data, where it reaches 64%; schematic patterns are reported as hardest. Human performance is 90% on CxN-NLI and 83% on CxN-NLI-Distinction, so the human drop is about 7 percentage points, substantially smaller than the model drop (Mackintosh et al., 19 Sep 2025).

Fine-tuning on a subset of ConTest-NLI yields measurable in-domain gains but weak transfer to adversarial or constructionally ambiguous conditions. On CxN-NLI, LLaMA 3.1 8B improves from 57% to 66% and Mistral 8B from 49% to 63%; on CxN-NLI-Distinction, the same models move from 33% to 39% and from 38% to 39%, respectively. GPT-4o improves from 88% to 91% on CxN-NLI and from 64% to 65% on CxN-NLI-Distinction. The abstract summarizes the fine-tuning effect as yielding up to 9% improvement, while emphasizing persistent abstraction gaps (Mackintosh et al., 19 Sep 2025).

The reported error profile is structurally informative. Models overfit to surface lexical or syntactic cues, fail to generalize entailments across constructions with shared forms, and neglect scalar, negation, or path entailments encoded by the construction. The paper argues that these failures persist across all constructions, indicating a general inability to achieve abstraction rather than isolated local errors. This suggests that ConTest-NLI, in the construction-based sense, is not merely another adversarial benchmark; it is a theory-anchored test of whether models can internalize constructional semantics rather than memorize prototypical patterns (Mackintosh et al., 19 Sep 2025).

6. Position within the NLI evaluation landscape

ConTest-NLI belongs to a broader movement in NLI research that treats benchmark accuracy as insufficient unless paired with tests of reasoning robustness, contextual adequacy, or evidential fidelity. In logic-oriented probing, automated reasoning over SNLI using Attempto Controlled English and theorem provers increased parseable coverage from 7.06% to 16.61% after syntactic rewrites, but overall NLI accuracy remained 28.7% without semantic rules and 33.3% with them; notably, entailment predictions had perfect precision when produced (Marji et al., 2020). This provides a complementary perspective to ConTest-NLI: robustness can be examined either by controlled perturbation or by formal deductive analysis.

Other benchmarks widen the scope of inference rather than perturbing short sentence pairs. ConTRoL introduces 8,325 expert-designed context-hypothesis pairs over 1,970 passages, with a human ceiling of 94.4% and BART-NLI-FT at 61.0%, highlighting the difficulty of passage-level contextual reasoning over long texts (Liu et al., 2020). ContractNLI moves to document-level inference over 607 annotated contracts and requires evidence identification in addition to entailment classification; Span NLI BERT-large reaches 0.875 NLI accuracy and 0.922 evidence mAP, yet contradiction remains difficult and exception-based negation is a major challenge (Koreeda et al., 2021). In clinical NLI, NLI4CT evaluates entailment and contradiction over Clinical Trial Reports using F1, faithfulness, and consistency; GPT-4 averages 0.8328, while merged PEFT adapters improve open models’ F1 and consistency but do not surpass GPT-4 in faithfulness or consistency (Gema et al., 2024).

Robustness-oriented training methods also align with ConTest-NLI’s goals. A neuro-symbolic contrastive learning framework combines PLMs and ILP to generate hard positive and negative examples, improving inference accuracy in in-domain, cross-domain, and cross-form settings; for example, BERT-base improves from 0.54 to 0.70 in-domain and from 0.49 to 0.63 in cross-domain transfer (Liu et al., 13 Feb 2025). In applied verification, NLI models constructed from QA instances can improve selective QA confidence estimation; at 20% coverage, F1 rises from 81.6 for the QA model alone to 87.1 for an NLI+QA ensemble (Chen et al., 2021). Taken together, these results situate ConTest-NLI within a larger research program: evaluating whether NLI systems remain correct when lexical form varies, when context length increases, when evidence must be localized, or when inference must align with explicit logical structure.

Across its two main formulations, ConTest-NLI crystallizes a common methodological claim. Standard benchmark success is compatible with substantial semantic brittleness. Whether instantiated as SNLI contrast sets or as a construction-based benchmark of English form-meaning pairings, ConTest-NLI is designed to measure the gap between surface-level success and robust inference under controlled variation (Sanwal, 2024, Mackintosh et al., 19 Sep 2025).