Zero-shot Natural Language Inference

• Zero-shot Natural Language Inference is a paradigm that predicts entailment, contradiction, or neutrality between premise–hypothesis pairs without domain-specific fine-tuning.
• It leverages pretrained language models with techniques like prompt-based hypothesis construction and universal classification to assess generalization and transfer across tasks.
• Its applications span text classification, hate speech detection, and requirements engineering, while challenges include prompt sensitivity and limited logical compositionality.

Zero-shot Natural Language Inference (NLI) describes the paradigm in which a model is tasked with predicting entailment, contradiction, or neutrality between premise–hypothesis pairs drawn from task or domain distributions never seen during supervised fine-tuning. Unlike standard supervised NLI, where models are tuned on large, task-aligned labeled data, in zero-shot NLI prediction is made exclusively based on prior knowledge contained in the pretrained model and the test prompt itself. This paradigm is central to assessing model generalization, transfer, and robustness, especially for new domains, tasks with evolving label sets, or truly low-resource and cross-lingual benchmarks.

1. Formal Problem and Zero-Shot Methodology

Zero-shot NLI is operationalized as a three-way classification. Given an input $x = (\text{premise}, \text{hypothesis})$, a model computes label posteriors $p(c \mid x)$ over $$c \in \{ \text{entailment}, \text{contradiction}, \text{neutral} \}$$ and predicts

$\hat{y} = \arg\max_{c \in C} p(c \mid x)$

as detailed in (Faria et al., 2024). In universal classification frameworks, the hypothesis can be constructed from an arbitrary label or class description, enabling flexible application beyond canonical NLI (Laurer et al., 2023).

Zero-shot NLI models either:

• Leverage off-the-shelf pretrained LLMs (PLMs) fine-tuned only on generic NLI datasets (e.g., MNLI, SNLI, ANLI, FEVER) but never on the target domain/task;
• Or are prompted in natural language to produce NLI-style judgements using no labeled in-domain data nor any further parameter updates (Madaan et al., 2024, Ebrahimi et al., 2021, Bareiß et al., 2024, Yang et al., 2022, Akoju et al., 2023).

Input formatting and score extraction vary:

• [CLS] premise [SEP] hypothesis [SEP] for encoder-only transformer NLI models, returning softmaxed logits per label (Laurer et al., 2023, Faria et al., 2024, Ebrahimi et al., 2021).
• Multiple-choice reformulation: presents several hypothesis options jointly, scored by masked-language modeling heads (Yang et al., 2022).
• Prompt-based LLM querying: textual prompt string presents premise, hypothesis, and label options (or asks for a generative label), e.g., "Premise: … Hypothesis: … Choose one label from {entailment, contradiction, neutral}. Answer:" (Faria et al., 2024, Mikami et al., 17 Sep 2025).
• Universal classification: Input $x$ paired with class-verbalizations $h_j = T(c_j)$, evaluated for entailment probability $P(e \mid x, h_j)$ (Laurer et al., 2023, Plaza-del-Arco et al., 2022).

2. Model Architectures, Training Regimens, and Efficiency

Encoder-only transformers (e.g., DeBERTa, RoBERTa, BART, XLM-R) dominate zero-shot NLI, most commonly trained on MultiNLI, SNLI, ANLI, XNLI, FEVER, and other transfer benchmarks (Laurer et al., 2023, Ebrahimi et al., 2021).

• Universal NLI classifiers (e.g., DeBERTa-v3-zeroshot-v1.1) train on a union of NLI and non-NLI classification datasets, recasting each as premise–hypothesis–label triples. Loss is per-example cross-entropy over three labels: $$\ell_i = - \sum_{k \in \{e, n, c\}} \mathbf{1}_{y_i=k} \log P(k \mid x_i, h_i)$$ with minibatch objective

$\mathcal{L} = \frac{1}{|B|} \sum_{i \in B} \ell_i$

(Laurer et al., 2023). For binary configuration, "neutral" and "contradiction" are merged into "not-entailment".

• Multilingual models (XLM-R, XLM-V, mDeBERTa-v3, MiniLMv2, Ernie-m) are essential for cross-lingual zero-shot NLI, often trained on XNLI and TaskSource (Bareiß et al., 2024, Ebrahimi et al., 2021).
• Generative, decoder-only LLMs (GPT-3.5, Llama3, Gemini) are evaluated in prompt-based settings by eliciting NLI judgements via natural language output (Faria et al., 2024, Madaan et al., 2024, Mikami et al., 17 Sep 2025).
• Model parameterization and inference efficiency are a core design consideration: encoder-only DeBERTa-v3 models with ∼200–430M parameters achieve universal zero-shot capacity at ~5× lower latency versus 7B decoder-only LLMs (Laurer et al., 2023).

3. Prompt Engineering and Hypothesis Construction

Zero-shot NLI requires careful prompt or hypothesis template design. Critical principles:

• Natural Language Hypotheses: Each class label $c_j$ is mapped to a short, explicit hypothesis sentence $h_j = T(c_j)$. Templates may be generic ("This text is about {}"), emotion-oriented ("The author feels {}"), domain-specific ("This app review is about {}") (Laurer et al., 2023, Plaza-del-Arco et al., 2022, Bareiß et al., 2024).
• Prompt Sensitivity: Performance depends acutely on template context and verbalization. For instance, “This text expresses anger” outperforms bare label names for certain emotion classification tasks; WordNet-style definitions underperform (Plaza-del-Arco et al., 2022, Bareiß et al., 2024).
• Prompt Language: For cross-lingual applications, English-language prompts consistently outperform translated prompts even on non-English data, reflecting a bias inherited from predominantly English NLI pretraining corpora (Bareiß et al., 2024).
• Hypothesis Ensembles: Averaging entailment scores over an ensemble of prompt templates generally provides robust performance close to the best prompt per corpus, minimizing manual tuning (Plaza-del-Arco et al., 2022).
• Hypothesis Engineering for Control: For fine-grained or robust zero-shot classification, several auxiliary hypotheses may be composed and combined via simple logic:
  • Filtering by target group, counterspeech, reclaimed slurs (for hate speech) (Goldzycher et al., 2022)
  • Tree-structured decomposition for complex ontologies (political relation extraction) (Hu et al., 2023)

4. Empirical Results and Failure Modes

General Zero-Shot NLI

• On general NLI, zero-shot accuracy from best universal models (e.g., DeBERTa-v3-all-33) on held-out benchmarks averages ~0.75 in balanced accuracy, +9.4% over NLI-only baseline. Gains are especially marked for multi-class tasks over binary ones (Laurer et al., 2023).
• In classical three-way NLI (SNLI, MNLI, ANLI, HANS), true zero-shot LLM accuracy hovers near chance (<35%; 50% for binary tasks). Few-shot in-context examples yield large gains (Madaan et al., 2024).
• Classical NLI benchmarks continue to discriminate model quality and training stage, but are not fully saturated even for 405B parameter LLMs. Alignment of model softmax with human uncertainty distribution improves with scale but remains distant from inter-human agreement (Madaan et al., 2024).

Cross-Lingual and Low-Resource

• In truly low-resource languages (e.g., 10 AmericasNLI indigenous tongues), XLM-R's zero-shot accuracy is poor (mean 38.62%), only ~5 points above chance; continued pretraining offers ~5–6 point improvement; translation-based fine-tuning brings accuracy up to ~49.1% (Ebrahimi et al., 2021).
• For Bangla/XNLI, state-of-the-art LLMs (GPT-3.5, Gemini 1.5 Pro) under zero-shot prompt lag well behind best fine-tuned Bengali PLMs (BanglaBERT: 82.0%, GPT-3.5: 74.0%) (Faria et al., 2024).
• Prompt sensitivity, domain-mismatch, and world knowledge gaps underlie most zero-shot NLI errors in low-resource settings.

Structured Reasoning and Compositionality

• In biomedical syllogistic NLI (SylloBio-NLI), zero-shot 7–8B LLMs underperform even on basic inference schemes: accuracy for generalized modus ponens ~70%, for disjunctive syllogism ~23%. In-context few-shot prompting can boost accuracy by up to 43% for strong models but reliability remains low (Wysocka et al., 2024).
• For compositional NLI involving quantifiers and negation (SICCK), zero-shot transformer accuracy is moderate on adjectives/adverbs (F1~0.58) and quantifiers (F1~0.5–0.6), but weak on negation (F1<0.3); fine-tuning on small synthetic sets yields minimal further improvement (Akoju et al., 2023).
• For Japanese comparatives, prompt template variance is high (±0.04–0.06 accuracy), logic-form scaffolding can patch failures in deep compositional inference (Mikami et al., 17 Sep 2025).

5. Applications and Domain Generalization

Zero-shot NLI enables rapid extension to new tasks and domains, for which labeled data is scarce or labels change frequently:

• Text Classification: Formulated as NLI by mapping each class to a hypothesis via a universal template. Efficiency and interpretability are key advantages (Laurer et al., 2023, Plaza-del-Arco et al., 2022).
• Requirements Engineering: NLI models, by reframing requirements classification or conflict detection as premise-hypothesis evaluation, outperform prompt-based models and fine-tuned PLMs—especially in cross-project, zero-shot settings (Fazelnia et al., 2024).
• Hate Speech and Emotion Detection: Hypothesis engineering, with targeted supporting hypotheses, delivers robust modular classifiers and boosts accuracy by >7–10 pp over vanilla zero-shot (Goldzycher et al., 2022, Plaza-del-Arco et al., 2022).
• Political Relation Extraction: NLI with codebook-derived hypothesis templates (e.g., ZSP) offers interpretable, maintainable, and high-performing zero-shot alternatives to dictionary or prompt-based approaches (Hu et al., 2023).
• Multimodal and Visual NLI: Visual grounding via text-to-image generation followed by VQA or multimodal embedding alignment achieves competitive zero-shot SNLI accuracy (VQA: 77%; CSS: 69% for 3-way) and resists textual heuristics (Ignatev et al., 21 Nov 2025).

6. Limitations, Challenges, and Prospects

• Instability and Prompt Sensitivity: Zero-shot NLI is sensitive to template design, label verbalization, and prompt language. Small phrasing or contextual shifts can alter classification outcomes substantially, especially for LLM-based approaches (Plaza-del-Arco et al., 2022, Bareiß et al., 2024, Mikami et al., 17 Sep 2025).
• Logical and Compositional Gaps: SOTA models consistently underperform in logical/quantified inference, negation, and complex compositionality—both in structured (SICCK, SylloBio-NLI) and logically rich languages (Japanese comparatives) (Akoju et al., 2023, Wysocka et al., 2024, Mikami et al., 17 Sep 2025).
• True Zero-Shot Generalization Remains Elusive: Even strong universal NLI models and LLMs have limited ability to transfer to unseen class semantics, rare languages, or complex inference schemes in a purely zero-shot regime without further adaptation or engineered prompts (Ebrahimi et al., 2021Madaan et al., 2024).
• Improvement Prospects: Recommended research directions include combining parallel data with monolingual continued pretraining (Ebrahimi et al., 2021), integrating explicit reasoning modules or logic-aware curricula (Wysocka et al., 2024), and leveraging dynamic hypothesis ensembles (Goldzycher et al., 2022, Plaza-del-Arco et al., 2022).
• Evaluation Practice: Robust benchmarking should include reporting mean and variance across templates and languages; hypothesis-only and artifact tests to delineate surface-heuristic learning (Ebrahimi et al., 2021Mikami et al., 17 Sep 2025).

• "AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages" (Ebrahimi et al., 2021)
• "Natural Language Inference Prompts for Zero-shot Emotion Classification in Text across Corpora" (Plaza-del-Arco et al., 2022)
• "Building Efficient Universal Classifiers with Natural Language Inference" (Laurer et al., 2023)
• "Lost in Inference: Rediscovering the Role of Natural Language Inference for LLMs" (Madaan et al., 2024)
• "English Prompts are Better for NLI-based Zero-Shot Emotion Classification than Target-Language Prompts" (Bareiß et al., 2024)
• "Zero-Shot Learners for Natural Language Understanding via a Unified Multiple Choice Perspective" (Yang et al., 2022)
• "SylloBio-NLI: Evaluating LLMs on Biomedical Syllogistic Reasoning" (Wysocka et al., 2024)
• "Leveraging Codebook Knowledge with NLI and ChatGPT for Zero-Shot Political Relation Classification" (Hu et al., 2023)
• "Unraveling the Dominance of LLMs Over Transformer Models for Bangla Natural Language Inference: A Comprehensive Study" (Faria et al., 2024)
• "Can LLMs Robustly Perform Natural Language Inference for Japanese Comparatives?" (Mikami et al., 17 Sep 2025)
• "Don't Learn, Ground: A Case for Natural Language Inference with Visual Grounding" (Ignatev et al., 21 Nov 2025)
• "Lessons from the Use of Natural Language Inference (NLI) in Requirements Engineering Tasks" (Fazelnia et al., 2024)
• "Hypothesis Engineering for Zero-Shot Hate Speech Detection" (Goldzycher et al., 2022)
• "Synthetic Dataset for Evaluating Complex Compositional Knowledge for Natural Language Inference" (Akoju et al., 2023)