e-SNLI-VE: Visual NLI with Explanations

• e-SNLI-VE is a vision-language natural language inference corpus that pairs images with textual hypotheses and human explanations.
• It extends SNLI-VE by correcting label noise and integrating explanation supervision, thus enhancing benchmark reliability in multimodal tasks.
• The dataset supports joint learning for label prediction and explanation generation, highlighting trade-offs between model accuracy and explanation quality.

e-SNLI-VE is a vision-language natural language inference corpus in which the premise is an image, the hypothesis is a sentence, the label is one of entailment, contradiction, or neutral, and each instance is paired with a human natural language explanation. It extends the visual entailment formulation introduced for SNLI-VE, where a model predicts $p_\theta(y \mid I,h)$ for an image $I$, hypothesis $h$, and label $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$, by adding explanation supervision and explanation-based evaluation (Xie et al., 2018). The dataset was first introduced through the correction of SNLI-VE label noise and the addition of explanations, and was later integrated into the e-ViL benchmark as the largest dataset in that benchmark and, at the time, the largest vision-language dataset with natural language explanations, with over 430k instances (Do et al., 2020, Kayser et al., 2021).

1. Origins in textual and visual entailment

e-SNLI-VE inherits its semantics from two earlier resources. The first is SNLI-VE, which turns textual entailment into visual entailment by replacing the SNLI textual premise caption with the corresponding Flickr30k image, while keeping the SNLI hypothesis and label unchanged (Xie et al., 2018). The second is e-SNLI, which augments SNLI with human-written explanations for why a premise–hypothesis pair is entailment, contradiction, or neutral (Camburu et al., 2018).

Under the visual entailment formulation, entailment means that the image provides enough visual evidence to conclude that the hypothesis is true; contradiction means that the image visually falsifies some crucial aspect of the hypothesis; neutral means that the image neither entails nor contradicts the hypothesis because evidence is insufficient (Xie et al., 2018). e-SNLI-VE preserves this three-way semantics and adds an explanation field, so that each example contains an image premise, a textual hypothesis, a label, and a free-form natural language explanation (Kayser et al., 2021).

The task design places e-SNLI-VE at the intersection of textual NLI, visual grounding, and natural language explanation generation. A direct antecedent of this design is the observation from e-SNLI that label prediction can be sustained by dataset artifacts much more easily than explanation generation: a hypothesis-only model can reach approximately $67\%$ label accuracy on SNLI but only about $7\%$ correct explanations, which motivates explanation supervision as a stronger constraint on reasoning behavior (Camburu et al., 2018). In the visual setting, this motivation becomes sharper because the explanation is expected to connect textual claims to visual evidence.

2. Construction, relabeling, and curation

The initial construction route is inherited from SNLI-VE. Because many SNLI premises are Flickr30k captions, each SNLI triple $(P,h,y)$ can be converted into a visual entailment triple $(I,h,y)$ by replacing the caption premise $P$ with its corresponding image $I$ (Xie et al., 2018). This transfer is natural for entailment, but it introduces instability for neutral and contradiction because the image may contain information absent from the caption.

That instability became a central issue for e-SNLI-VE. In the 2020 introduction of the dataset, an independent annotation of 150 random neutral SNLI-VE test examples found a $I$0 error rate for neutral labels, while prior work had already suggested that neutral was much noisier than entailment and contradiction (Do et al., 2020). In the later e-ViL account, a manually labeled subset of 535 examples yielded a $I$1 error rate among neutral labels, and the authors consequently focused on correcting and filtering neutral instances (Kayser et al., 2021).

The 2020 correction procedure targeted neutral examples in validation and test. Amazon Mechanical Turk workers saw an image and hypothesis, selected one of the three labels, highlighted words in the hypothesis that supported the decision, and wrote an explanation that used at least half of the highlighted words (Do et al., 2020). Each pair received three annotations. Quality control included trusted pairs inserted into annotation batches and in-browser checks that blocked submission unless the trusted item was correctly labeled; examples receiving three different labels were treated as too ambiguous and removed, accounting for $I$2 of validation neutral examples and $I$3 of test neutral examples (Do et al., 2020). On a manual sample of 150 corrected validation neutrals, the neutral error rate dropped from about $I$4 to $I$5 (Do et al., 2020).

The e-ViL release extended curation beyond dev/test relabeling. For the training split, it automatically removed likely false neutrals by scoring each image–hypothesis pair against the five Flickr30k captions of the image with a Roberta-large NLI model trained on MNLI; if the sum of entailment or contradiction confidence over captions exceeded $I$6, the example was filtered out (Kayser et al., 2021). On the 535-example subset, this reduced neutral label error from $I$7 to $I$8, and on the relabeled validation set from $I$9 to $h$0 (Kayser et al., 2021).

Explanation quality also required explicit cleanup. A naive merge of e-SNLI explanations into visual entailment produced an estimated $h$1 low-quality, $h$2 medium-quality, and $h$3 high-quality explanation mix, largely because many e-SNLI explanations were written for textual premises rather than images (Kayser et al., 2021). The e-ViL release therefore combined re-annotation of neutral explanations in validation and test with three filters: a keyword filter that removed premise-oriented explanations containing terms such as “synonym,” “sentence,” or “rephrasing”; a similarity filter that removed entailment cases where textual premise and hypothesis had ROUGE-1 above $h$4; and an uncertainty filter for contradiction cases based on dispersion of Roberta-NLI scores across the five captions (Kayser et al., 2021). In a manual sample of 100 newly collected neutral explanations, $h$5 were rated high quality (Kayser et al., 2021).

3. Dataset variants, splits, and annotation properties

The name e-SNLI-VE refers both to the initial 2020 release and to the later, more heavily filtered e-ViL benchmark release. The latter is smaller because it removes likely false neutrals and low-quality explanations.

Release	Split sizes	Distinguishing curation
Initial 2020 release	Train 529,527; Val 17,547; Test 17,574	Corrected neutral dev/test labels; inherited e-SNLI explanations plus newly crowdsourced neutral explanations (Do et al., 2020)
e-ViL benchmark release	Train 401,717; Val 14,339; Test 14,740	False-neutral filtering and explanation filtering; final benchmark version (Kayser et al., 2021)

In the e-ViL benchmark version, the split by images is 29,783 train, 1,000 validation, and 1,000 test images, with label distributions $h$6 of $h$7 for train, $h$8 for validation, and $h$9 for test (Kayser et al., 2021). Mean hypothesis length is $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$0, $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$1, and $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$2 tokens for train, validation, and test respectively; mean explanation length is $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$3, $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$4, and $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$5 (Kayser et al., 2021). The total number of image–hypothesis–explanation triples is 430,796 (Kayser et al., 2021).

The initial 2020 release preserved all original training pairs, so its training set remained close to the nearly balanced SNLI-VE class structure, while validation and test became more imbalanced after neutral relabeling. In that release, only $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$6 of originally neutral pairs remained neutral, while $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$7 became contradiction and $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$8 became entailment; the corrected validation and test distribution became approximately $$y \in \{\text{entailment}, \text{contradiction}, \text{neutral}\}$$9 entailment, $67\%$0 neutral, and $67\%$1 contradiction (Do et al., 2020). Because of this imbalance, the 2020 paper reported balanced accuracy, defined as the unweighted average of per-class accuracies (Do et al., 2020).

Ground-truth explanation quality in the e-ViL release was assessed by external annotators using a four-point scale. For e-SNLI-VE, $67\%$2 of explanations were judged “No,” $67\%$3 “Weak No,” $67\%$4 “Weak Yes,” and $67\%$5 “Yes,” so $67\%$6 were rated Weak Yes or Yes (Kayser et al., 2021). This indicates substantial but not perfect alignment between reference explanations and the visual entailment decision.

4. Modeling paradigms and benchmark performance

The first explanation-aware models on e-SNLI-VE were built around the Bottom-Up and Top-Down Attention architecture. In the 2020 work, the base VTE model used Faster R-CNN region proposals for image features, GloVe embeddings and a GRU for the hypothesis, top-down attention over regions conditioned on the hypothesis, and a three-way label classifier (Do et al., 2020).

Two explanation architectures were then introduced. “Predict and Explain” (PaE-BUTD-VE) used a shared multimodal feature vector for both label prediction and explanation generation, with a 512-dimensional LSTM decoder conditioned on the label token and trained with a joint loss

$67\%$7

with the best validation balanced accuracy obtained at $67\%$8 (Do et al., 2020). “Explain Then Predict” (EtP-BUTD-VE) separated the stages: one network generated an explanation from image and hypothesis, and a second network, ExplToLabel-VE, predicted the label from the explanation alone (Do et al., 2020).

These early models exposed a trade-off between label performance and explanation relevance. On e-SNLI-VE, PaE-BUTD-VE reached $67\%$9 balanced accuracy with $7\%$0 explanation relevance under manual evaluation, while EtP-BUTD-VE reached $7\%$1 balanced accuracy with $7\%$2 explanation relevance; the non-explanatory BUTD baseline obtained $7\%$3 balanced accuracy (Do et al., 2020). The second-stage explanation-to-label classifier achieved $7\%$4 balanced accuracy when trained on gold explanations, showing that a good explanation is highly predictive of the entailment label (Do et al., 2020).

The 2021 e-ViL benchmark replaced these recurrent architectures with a transformer-based predict-and-explain model, e-UG, combining UNITER as the task module and GPT-2 as the explanation module (Kayser et al., 2021). In this setup, the task module predicts the label from Faster R-CNN region features and the hypothesis, while the explanation module conditions on UNITER’s contextualized embeddings together with the hypothesis and the predicted answer. Training jointly optimizes

$7\%$5

and benchmark evaluation distinguishes task score $7\%$6, explanation score $7\%$7, and overall score $7\%$8 (Kayser et al., 2021).

On e-SNLI-VE, e-UG substantially outperformed prior baselines in the e-ViL benchmark. Human evaluation reported $7\%$9, $(P,h,y)$0, and $(P,h,y)$1, compared with $(P,h,y)$2 for PJ-X, $(P,h,y)$3 for FME, and $(P,h,y)$4 for RVT (Kayser et al., 2021). Class-wise automatic evaluation showed entailment accuracy $(P,h,y)$5, neutral accuracy $(P,h,y)$6, and contradiction accuracy $(P,h,y)$7 for e-UG, confirming that neutral remained the hardest class (Kayser et al., 2021). Joint training with explanation loss produced small gains for FME and e-UG relative to training the task module alone, which the benchmark interprets as evidence that explanations can act as learning signals (Kayser et al., 2021).

5. Faithfulness, attribution, and explanation diagnostics

A later line of work treated e-SNLI-VE not merely as a generation benchmark but as a benchmark for explanation faithfulness. In the 2023 study on faithfulness metrics, e-SNLI-VE was one of the datasets of the e-ViL benchmark and the largest of them, with train/dev/test sizes of 401,717/14,339/14,740 instances (Ambsdorf, 2023). The study formalized faithfulness through Integrated Gradients attributions for both the answer model and the explanation generator.

Its main metric, Attribution-Similarity, compares the relevance distributions assigned to the shared inputs by the task model and the explanation model. At the dataset level, after splitting by modality, the score is

$(P,h,y)$8

where $(P,h,y)$9 and $(I,h,y)$0 are text relevances for answer and explanation, and $(I,h,y)$1 and $(I,h,y)$2 are the corresponding image relevances (Ambsdorf, 2023). The same study adapted ERASER-style sufficiency and comprehensiveness to explanation attributions, yielding NLE-Sufficiency and NLE-Comprehensiveness scores.

On e-SNLI-VE, the central finding was that removing redundant inputs from the GPT-2 explanation module increased linguistic Attribution-Similarity without materially harming task accuracy or explanation quality (Ambsdorf, 2023). Task accuracy stayed within about $(I,h,y)$3 across e-UG and its ablations: $(I,h,y)$4 for the reproduced e-UG, $(I,h,y)$5 for NQ, $(I,h,y)$6 for NA, $(I,h,y)$7 for OU, and $(I,h,y)$8 for NU (Ambsdorf, 2023). METEOR and BERTScore also remained close: for example, e-UG obtained METEOR $(I,h,y)$9 and BERTScore $P$0, while OU obtained METEOR $P$1 and BERTScore $P$2 (Ambsdorf, 2023).

Faithfulness results were sharper on the linguistic modality. Language Attribution-Similarity rose from $P$3 for default e-UG to $P$4 for NQ, $P$5 for NA, and $P$6 for OU, while combined Attribution-Similarity rose from $P$7 to $P$8 (Ambsdorf, 2023). Vision Attribution-Similarity stayed near $P$9 and showed no equally strong monotonic pattern (Ambsdorf, 2023). NLE-Sufficiency and NLE-Comprehensiveness behaved differently: their correlations with Attribution-Similarity on e-SNLI-VE were all small, with Pearson $I$0, which led the study to treat them as complementary diagnostics rather than interchangeable faithfulness measures (Ambsdorf, 2023).

These results established e-SNLI-VE as a particularly informative testbed for studying whether explanations actually track model reasoning, especially on the textual side of the image–hypothesis relation. A plausible implication is that the combination of a large dataset and a simple three-label answer space makes mismatches between answer and explanation processing easier to expose.

6. Zero-shot transfer, large multimodal models, and benchmark limitations

Although e-SNLI-VE is usually used in supervised predict-and-explain settings, later work also treated it as a target for zero-shot or lightly supervised transfer. UniFine, a zero-shot framework built for SNLI-VE rather than e-SNLI-VE, combines frozen CLIP, RoBERTa, OFA, Faster-RCNN, and clustering heuristics, and explicitly notes that the same label-prediction pipeline transfers directly to e-SNLI-VE, with explanations available as supervision or evaluation targets for region and token importance rather than as inputs to the classifier (Wang et al., 2023). In that framework, region selection and caption–hypothesis similarity are presented as fine-grained internal signals that align naturally with the kind of evidence human rationales in e-SNLI-VE describe (Wang et al., 2023).

A 2025 study used e-SNLI-VE as its sole benchmark for probing whether high visual entailment performance really indicates grounded multimodal understanding in large multimodal LLMs (Pitta et al., 23 Jul 2025). With LLaMA 3.2 11B Vision, zero-shot performance was low and prompt-sensitive: overall zero-shot accuracy across six label-order permutations was $I$1, while two main prompts achieved $I$2 and $I$3 accuracy on the test set (Pitta et al., 23 Jul 2025). When the visual input was replaced by black images, accuracy remained $I$4 and $I$5, and the model overwhelmingly predicted entailment, showing that performance only slightly degraded relative to full-image zero-shot (Pitta et al., 23 Jul 2025).

The same study also evaluated generated explanations against e-SNLI-VE references with BERTScore. Fine-tuning yielded $I$6 accuracy and overall F1 $I$7, outperforming OFA-X’s $I$8 on e-SNLI-VE, while explanation BERTScore F1 reached $I$9 (Pitta et al., 23 Jul 2025). Yet zero-shot explanations on full images scored around $I$00 to $I$01, and black-image explanations still scored about $I$02, even though the model had no visual evidence and often hallucinated scene details (Pitta et al., 23 Jul 2025). This finding does not negate the utility of e-SNLI-VE as a training and comparison benchmark, but it does show that high explanation similarity and strong label accuracy are not sufficient by themselves to establish visual grounding.

The resulting picture is two-sided. e-SNLI-VE has become a standard resource for visual entailment with explanations, for joint label-and-rationale modeling, and for faithfulness research. At the same time, later benchmark analyses show that it also inherits the classic difficulties of SNLI-derived datasets: linguistic priors, residual ambiguity in neutrality, and the possibility that apparently strong explanations are semantically plausible without being causally grounded in image evidence (Pitta et al., 23 Jul 2025). In that sense, e-SNLI-VE functions both as a benchmark and as a diagnostic instrument for the limitations of current vision-language explanation systems.