---
title: Entailment-Based Zero-Shot Classifier
url: https://www.emergentmind.com/topics/entailment-based-zero-shot-classifier-zsc
type: topic
---

# Entailment-Based Zero-Shot Classifier

An entailment-based zero-shot classifier (ZSC) is a class of models that reformulate the zero-shot classification problem as a natural language inference (NLI) or textual entailment task. These systems make use of pretrained or fine-tuned NLI models, leveraging natural-language label descriptions (verbalizations) to match textual inputs with candidate labels—without requiring any labeled examples for the downstream classification task or label space during training. Recent research has established NLI cross-encoders as a foundational approach for ZSC and has enabled principled comparisons with embedding-based, reranking, and large language model (LLM) alternatives [2603.11991, 1909.00161, 2210.07587, 2109.03659].

## 1. Entailment Reformulation and Label Verbalization

The core idea is to cast classification as a premise–hypothesis entailment problem. For each unlabeled text $x$ and each candidate label $\ell$ from a label set $\mathcal{L}$, a natural-language template or "verbalizer" $T[x, \ell]$ is instantiated, which forms the hypothesis $h_\ell$. For example, for sentiment classification (Amazon Polarity), a verbalizer might be:

- **Premise:** $x$ = review text,
- **Hypothesis:** $h_\ell$ = "The overall sentiment within the Amazon product review is {label}." (with {label} replaced by "positive" or "negative")

The text $x$ and hypothesis $h_\ell$ are concatenated and fed into the NLI model as $[\text{CLS}]\, x\, [\text{SEP}]\, h_\ell\, [\text{SEP}]$ [2603.11991, 1909.00161]. Label verbalization is critical: templates should be concise, context-rich, and, where necessary, augmented by label definitions for improved performance in the fully unsupervised regime [1909.00161].

## 2. Scoring Functions and Decision Rules

Entailment-based ZSCs use the NLI model to compute a score or probability that the premise $x$ entails each candidate hypothesis $h_\ell$. There are two scoring head variants:

- **Binary entailment head:** Outputs a single logit $l_\text{ent}$ per $(x, h_\ell)$; computes $\sigma(l_\text{ent})$ with standard binary cross-entropy loss.
- **Three-way (triplet) NLI head:** Produces logits $(l_\text{ent}, l_\text{neut}, l_\text{contra})$; uses cross-entropy over three classes (entailment, neutral, contradiction). Inference collapses neutral and contradiction to compute 
  $$
  S(x, h) = l_\text{ent} - \log[\exp(l_\text{neut}) + \exp(l_\text{contra})]
  $$
  Optionally, probabilities are obtained by $\sigma(S)$ [2603.11991].

For each $x$, the ZSC evaluates all hypotheses $h_1, ..., h_L$ and predicts the label maximizing the relevant score (no thresholding in the zero-shot protocol). In multi-label or partially-seen evaluation protocols, additional calibration between seen and unseen labels can be applied using a margin parameter $\alpha$ [1909.00161].

## 3. Architectures, Training Protocols, and Optimization

Strong entailment-based ZSCs typically exploit transformer-based cross-encoders (e.g., DeBERTa-v3-large, RoBERTa-large, BART-large, ALBERT, BERT-base). These are pretrained and often further fine-tuned on the union of large-scale NLI resources (MNLI, ANLI, WANLI, FEVERNLI, LingNLI) using the corresponding classification head (binary or triplet). Key training configurations include:

- Batch size 32, 3 epochs, early stopping by dev loss plateau.
- AdamW optimizer with backbone learning rate $8 \times 10^{-6}$, head learning rate $4 \times 10^{-5}$.
- Cosine decay learning-rate schedule, mixed-precision training, dropout/GELU/LayerNorm classifier head atop the [CLS] token [2603.11991].

Label verbalizer templates should be fixed across models and datasets where possible for consistency; per-model tuning is discouraged [2603.11991].

## 4. Empirical Results and Domains of Strength

The empirical macro-F1 performance of entailment-based ZSCs on comprehensive benchmarks, such as BTZSC (22 datasets), is summarized below [2603.11991]:

| Model (Cross-Encoder)         | Macro-F1 | Accuracy |
|-------------------------------|----------|----------|
| bart-large-mnli               | ~0.51    | ~0.53    |
| nli-roberta-base              | ~0.49    | ~0.50    |
| bert-large-nli-triplet        | ~0.52    | ~0.55    |
| DeBERTa-v3-large-nli-triplet  | ~0.60    | ~0.62    |

Domain breakdown for DeBERTa-v3-large-nli-triplet:
- Sentiment: F1 ≈ 0.90 (near saturation)
- Topic: F1 ≈ 0.50
- Intent: F1 ≈ 0.45
- Emotion: F1 ≈ 0.42

Entailment-based ZSCs excel on standard sentiment tasks, significantly outperforming embedding-only models on seen labels and achieving high accuracy on unseen labels in label-partially-unseen setups. For example, fine-tuned NLI models show +8–15 accuracy improvement on unseen topics over supervised baselines [1909.00161]. In relation extraction, entailment-based ZSCs (with hand-crafted templates) deliver 63% F1 zero-shot—substantially above supervised classifiers in the few-shot regime [2109.03659].

## 5. Comparison with Embedding Models, Rerankers, and LLMs

Systematic benchmarking shows that entailment-based ZSCs, while robust and data-efficient, are now outperformed in absolute terms by larger rerankers and LLMs on challenging multi-class tasks (topic, intent, emotion) [2603.11991]:

| Model Family                 | Macro F1 | Inference Speed (qualitative)                          | Strengths                               |
|------------------------------|----------|--------------------------------------------------------|-----------------------------------------|
| Rerankers (Qwen3-8B)         | ~0.72    | 5–20× slower than embedders/cross-encoders             | State-of-the-art; excels at all tasks   |
| Embedding models (GTE-large) | ~0.62    | Fastest (few hundred ms/batch)                         | Best speed–accuracy trade-off           |
| Cross-encoders (DeBERTa-L)   | ~0.60    | ~2× the time of embedding models                       | Sentiment, few-shot, efficiency         |
| Instruction-tuned LLMs       | ~0.66–0.67| Slowest; scale brings gains (>3B params)                | Strong on topic; under rerankers on F1  |

Plateauing accuracy above 300–400M parameters is observed for NLI cross-encoders. Scaling benefits rerankers/LLMs more than embedding or cross-encoder models. Embedding models deliver the best accuracy/latency Pareto front; entailment-based ZSCs remain competitive for efficient mid-sized deployment and remain significantly faster than rerankers and LLMs [2603.11991].

## 6. Best Practices, Ablation Results, and Limitations

Empirical ablations reveal that:
- Triplet (three-way) NLI loss marginally improves intent classification but harms topic classification; binary and triplet losses are otherwise similar.
- Large transformer backbones ($\geq$300M) outperform base-size consistently by ~+3 macro-F1, but gains diminish beyond large models.
- Clean, context-rich label verbalizers generalize well across tasks and models.
- Early stopping on mixed-domain NLI dev sets mitigates overfitting [2603.11991].

Limitations:
- Entailment-based ZSCs are relatively weaker on emotion detection and on high-cardinality intent tasks (F1 < 0.45).
- Returns in accuracy rapidly diminish as backbone model size increases past several hundred million parameters [2603.11991].
- Main bottleneck in relation extraction ZSCs is recall for NO-RELATION, though small calibration sets (2 dev examples per label) recover most of the gap [2109.03659].
- Scaling to large label spaces ($>$10 classes) is not fully established [2210.07587].

## 7. Generalization Across Tasks and Future Directions

Entailment-based ZSCs have proven effective across classification domains (sentiment, topic, emotion, intent, relation extraction), label-verbalization schemes, and evaluation protocols (restrictive and fully-unseen zero-shot). Flexible hypothesis generation (template plus definition) and bootstrappable few-shot adaptation remain central to strong cross-domain performance [1909.00161, 2109.03659].

Recent advances include:
- Nested entailment meta-task reformulations enabling absorption of knowledge across dozens of datasets via supervised contrastive pretraining and improving zero-shot accuracy by 9.4 points over prior discriminative entailment models [2210.07587].
- Integration of reusable label verbalizers, supervised contrastive loss, and cross-task transfer protocols.
- Prospects for multilingual expansion, improved generic premises/calibration, and scalability to more complex structures and label-rich tasks [2210.07587].

Entailment-based zero-shot classifiers remain a baseline for data-efficient semantic classification, with ongoing relevance for practitioners balancing annotation cost, inference speed, and domain generalization [2603.11991, 1909.00161, 2210.07587, 2109.03659].

Source: https://www.emergentmind.com/topics/entailment-based-zero-shot-classifier-zsc