Visual Entailment in Multimodal Inference

• Visual Entailment is a multimodal inference task that evaluates image–text pairs to classify entailment, contradiction, or neutrality based on visual evidence.
• Variants range from phrase-level grounding and binary true/false formulations to update-based reasoning, each refining how visual evidence supports textual claims.
• Recent advances leverage neural attention, alignment methods, and synthetic data, yet challenges persist in prompt sensitivity, noisy labels, and complex crowded scenes.

Searching arXiv for recent and foundational papers on Visual Entailment. Visual Entailment (VE) is a multimodal inference task in which an image serves as the premise and a natural-language sentence serves as the hypothesis, and the system predicts whether the image entails, contradicts, or is neutral with respect to the hypothesis. In the contemporary literature, VE is usually formalized as a three-way classification problem over image–text pairs, but the broader research area also includes earlier phrase-level formulations grounded in image segments, binary true/false variants, explanation-augmented settings, and more recent extensions that model how additional updates strengthen or weaken an image–hypothesis relation (Xie et al., 2018, Izadinia et al., 2015, Zhang et al., 2024).

1. Formal scope and task variants

The standard formulation treats VE as the visual analogue of textual Natural Language Inference. Given an image premise $I$ and a textual hypothesis $H$, a model predicts $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$, or equivalently learns $p(y \mid I, H)$ and outputs $\hat y = \arg\max_y p(y \mid I, H)$. Under this formulation, entailment means that the image provides sufficient visual evidence that the hypothesis is true, contradiction means that the image visually rules out the hypothesis, and neutral means that the image does not provide enough information to decide (Xie et al., 2018).

An important precursor used the term in a narrower, phrase-centric sense. In that setting, the inputs are two visual phrases $X$ and $Y$, each grounded by a set of segmentation masks from a Segment-Phrase Table, and entailment is defined through directional asymmetry of region-to-image similarity. The paper formalizes this as

$\mbox{entail}(X \vDash Y) := Sim_{R2I}^{\rightarrow}(X,Y) - Sim_{R2I}^{\rightarrow}(Y,X),$

so that positive asymmetry indicates that $X$ is visually more specific and therefore entails $Y$ (Izadinia et al., 2015). This older formulation is not identical to the later image-premise, sentence-hypothesis benchmark style, but it established the idea that entailment can be operationalized through grounded visual evidence rather than text alone.

Several later variants narrow or alter the label space. COREVQA formulates a binary visual entailment-style task on crowded scenes in which each image is paired with a single plausible true/false statement and the model outputs whether the statement is true or false given the image (Chintapatla et al., 17 Jul 2025). The select-and-rerank VQA framework recasts candidate answers as synthetic statements and treats answer verification as a binary entailment-versus-contradiction problem, explicitly omitting a neutral label (Si et al., 2021). Defeasible Visual Entailment extends VE in a different direction: given an image $H$0, a hypothesis $H$1, and an update $H$2, the task is to determine whether the update is a weakener or a strengthener, or to generate such an update, thereby modeling changes in entailment strength rather than a single static label (Zhang et al., 2024).

These formulations suggest that VE is best viewed as a family of grounded inference problems rather than a single benchmark protocol. The common core is not a particular dataset format, but the requirement that semantic relations be adjudicated against visual evidence.

2. Historical development and benchmark lineage

A useful lineage begins with phrase-level grounding. The Segment-Phrase Table work introduced a large collection of bijective associations between textual phrases and their corresponding segmentations, then used those segment sets to define visual entailment, visual paraphrasing, and an ILP-based entailment graph with transitivity constraints (Izadinia et al., 2015). In this formulation, the “worlds” of possible-world semantics were approximated by the set of images or segments in which a phrase appears.

The next major step was the grounding of textual entailment with actual images. “Grounded Textual Entailment” constructed V-SNLI by taking SNLI premise captions and recovering the corresponding Flickr30k images, thereby turning a sentence–sentence NLI task into a visually grounded one. The paper explicitly interprets images as concrete “worlds” or “situations” in which premise and hypothesis are evaluated, and reports that the resulting V-SNLI contains 565,286 pairs, with a V-SNLI$H$3 subset derived from SNLI$H$4 (Vu et al., 2018).

The task was then crystallized under the name “Visual Entailment,” with SNLI-VE replacing the textual premise by the original image and keeping the three SNLI labels. The original SNLI-VE construction yields 529,527 train examples, 17,858 validation examples, and 17,901 test examples, with class-balanced train, validation, and test splits and 29,783 train images (Xie et al., 2018). This benchmark anchored most subsequent work.

Later work showed that this automatic construction introduced substantial noise, especially in the neutral class. SNLI-VE-2.0 re-annotated the neutral portions of validation and test, removed examples where all three annotators disagreed, and changed the validation and test label distribution to $H$5 for entailment, neutral, and contradiction. The corresponding e-SNLI-VE resource further appended natural language explanations, combining inherited e-SNLI explanations with newly collected visually grounded explanations for corrected neutral cases (Do et al., 2020).

A later re-examination of e-SNLI-VE argued that about $H$6 of neutral labels in dev/test were incorrect in the original SNLI-VE, and reported an e-SNLI-VE version with over 430k instances, including train/dev/test counts of 401,717, 14,339, and 14,740 labels respectively (Pitta et al., 23 Jul 2025). This corrected setting has become central in studies of prompt sensitivity, explanation quality, and visual grounding.

Recent benchmarks broaden the empirical scope. COREVQA focuses on dense CrowdHuman scenes and pairs 5,608 images with synthetically generated but fully human-verified true/false statements, with 1,566 true and 4,042 false statements, average statement length 30.20 words, and strong use of multi-clause constructions such as “while” and “despite” (Chintapatla et al., 17 Jul 2025). Defeasible Visual Entailment creates a benchmark of image–hypothesis–update quadruples by mapping $H$7-NLI updates back to Flickr30k images, producing 93,082 train, 1,888 validation, and 1,972 test examples with balanced weakener and strengthener labels (Zhang et al., 2024). Synthetic-NLI-VE, in turn, addresses data sparsity by generating one Stable Diffusion image per SNLI premise, yielding 157,567 premises, 565,286 hypotheses, and roughly 157k synthetic images (Reijtenbach et al., 15 Aug 2025).

Benchmark or resource	Core format	Distinguishing property
SNLI-VE	image + hypothesis $H$8 E/N/C	Flickr30k image replaces SNLI textual premise
e-SNLI-VE / SNLI-VE-2.0	image + hypothesis + explanation	corrected neutral labels and natural-language rationales
COREVQA	image + statement $H$9 True/False	crowded scenes and adversarial multi-clause statements
DVE	image + hypothesis + update	weakener/strengthener reasoning
Synthetic-NLI-VE	synthetic image + hypothesis $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$0 E/N/C	Stable Diffusion premises for low-cost VE training

3. Modeling paradigms

VE methods have ranged from symbolic reasoning and segment-based asymmetry to region-attention models, alignment matrices, and large pretrained vision–language encoders.

The early phrase-level approach grounded phrases in segmentation masks and measured entailment through asymmetric region-to-image similarity, then enforced transitivity over an entailment graph using an ILP: $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$1 This “SPT + ILP” combination outperformed both a language-only baseline and a detection-box baseline for phrase-level entailment, paraphrasing, and relative similarity, indicating that segment-level reasoning mattered for subtle pose and action relations (Izadinia et al., 2015).

At the image-premise level, early neural VE models were adapted from VQA. The Explainable Visual Entailment model used a text branch with GloVe embeddings, self-attention, and a GRU, plus either ResNet101 feature maps or Mask R-CNN region proposals in the image branch, followed by self-attention, text–image attention, and a classifier. On SNLI-VE, EVE-Image achieved 71.36% test accuracy, outperforming Hypothesis Only at 67.01%, Image Captioning at 67.47%, Relational Network at 68.39%, Attention Top-Down at 70.30%, and Attention Bottom-Up at 69.34% (Xie et al., 2018).

A distinct line emphasized explicit relation modeling rather than generic multimodal content fusion. AlignVE represents the image–hypothesis interaction as an alignment matrix

$$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$2

where $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$3 are contextualized image-region features and $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$4 are contextualized token features. Adaptive average and max pooling reduce this variable-sized matrix to a fixed vector for classification. AlignVE-Grid reached 72.45% test accuracy on SNLI-VE and outperformed previous content-based models under the same settings; replacing the alignment matrix with standard parallel co-attention caused an approximately 6% absolute drop (Cao et al., 2022).

Another line treated VE as a probe of cross-modal representation alignment. A CLIP-based method froze the image encoder $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$5 and text encoder $$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$6, fused premise and hypothesis embeddings as

$$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$7

and trained only a small MLP classifier. Training on text–text premise/hypothesis pairs and evaluating on image–text SNLI-VE yielded 64.66% test accuracy with CLIP$$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$8, 64.86% with CLIP$$y \in \{\text{entailment}, \text{neutral}, \text{contradiction}\}$$9, and 66.63% with CLIP$p(y \mid I, H)$0, while masking the image reduced accuracy to near the majority baseline (Song et al., 2022). This showed that pretrained contrastive encoders alone could support nontrivial zero-shot cross-modality transfer for VE.

The symbolic alternative remains important conceptually. A multimodal logical inference system maps scene graphs or FOL structures extracted from images into first-order logic formulas, maps hypotheses to logic with CCG-based semantic parsing, and uses theorem proving to test whether the image formula $p(y \mid I, H)$1 entails the text formula $p(y \mid I, H)$2, i.e. $p(y \mid I, H)$3. The paper showed that closed-world predicate circumscription improved handling of quantifiers, relations, and negation on complex visual-textual inference queries (Suzuki et al., 2019). This suggests that, alongside neural architectures, VE has also functioned as a testbed for explicit semantic representations.

More recent studies fine-tune large multimodal models directly. On e-SNLI-VE, a fine-tuned LLaMA 3.2 11B Vision model reached 83.3% accuracy and outperformed the reported OFA-X baseline at 80.9% (Pitta et al., 23 Jul 2025). This result is substantially above the early SNLI-VE numbers, but later sections of the same line of work argue that strong headline accuracy does not by itself establish robust visual grounding.

4. Explanation, fine-grained reasoning, and semantic structure

VE has increasingly been treated as an explainable reasoning task rather than only a label-prediction problem. The e-SNLI-VE resource adds natural language explanations to visual-textual entailment examples, enabling models to predict both a label and a textual rationale (Do et al., 2020). In that setting, a Predict-and-Explain BUTD model achieved 73.0% balanced accuracy on SNLI-VE-2.0 and produced 20% relevant explanations under manual evaluation, while an Explain-Then-Predict variant produced 35% relevant explanations but lower balanced accuracy at 69.40%. These results established a recurring VE trade-off: tighter coupling between explanations and labels can improve rationale quality yet hurt classification performance.

Later work pushed explanation faithfulness beyond post-hoc rationales. CALeC introduced a Chunk-aware Semantic Interactor, a relation inferrer, and a Lexical Constraint-aware Generator for VE with natural language explanations. The method explicitly models chunk-level rather than only token-level alignment between sentence structure and image regions, then injects lexical constraints derived from the relation inferrer into explanation decoding so that the generated rationale more directly reflects the decision-making evidence (Yang et al., 2022). The paper reports that CALeC significantly outperformed competitor models on both inference accuracy and explanation quality across three datasets.

An even more granular extension is Fine-Grained Visual Entailment. This formulation decomposes the hypothesis into AMR-derived knowledge elements, including both node-level concepts and tuple-level relations, and predicts entailment, neutral, or contradiction for each knowledge element rather than only for the whole sentence. Because only sample-level labels are available during training, the model uses a multi-instance learning objective together with semantic structural constraints over the AMR graph. The full system reached 68.18% overall KE-level accuracy and 96.98% structural consistency, while remaining competitive at the sample level (Thomas et al., 2022). This makes explicit a fact often implicit in VE: sentence-level predictions usually depend on multiple latent sub-judgments about entities, events, and relations.

These developments indicate that explanation in VE has at least three distinct meanings. It can denote natural-language justification, interpretable cross-modal alignment, or fine-grained semantic decomposition. The literature increasingly treats all three as desirable, especially when the goal is diagnostic understanding rather than benchmark accuracy alone.

5. Evaluation, diagnostics, and failure modes

A central theme in VE research is that nominal benchmark performance can mask weak grounding, prompt brittleness, or dataset artifacts. The correction of SNLI-VE into SNLI-VE-2.0 already showed that evaluation can change materially once visually inappropriate neutral labels are fixed, and that balanced accuracy is more appropriate than raw accuracy when the corrected split becomes class-imbalanced at $p(y \mid I, H)$4 for entailment, neutral, and contradiction (Do et al., 2020).

Prompt sensitivity has emerged as a distinct concern for multimodal LLMs. In an extensive study using e-SNLI-VE, zero-shot performance of LLaMA 3.2 11B Vision depended strongly on label order in the prompt: one prompt ordering achieved 0.445 accuracy while another achieved 0.413, and majority vote across six label-order permutations fell to 0.337. Three-shot inference improved performance to 0.487 in the best configuration, but six-shot inference degraded to 0.350–0.365 and introduced a strong neutral bias. Black-image ablations still yielded 0.360 and 0.369 accuracy, with the model predicting entailment more than 84–90% of the time and generating explanations that hallucinated visual content (Pitta et al., 23 Jul 2025). A plausible implication is that some reported VE performance reflects linguistic priors and answer-format effects as much as grounded inference.

Crowded-scene evaluation exposes a different failure regime. COREVQA reports that even GPT-4.1, the best tested model, achieved only 77.57% accuracy, while other models ranged from 39.98% to 76.60%. On the hard subset, the dominant failure modes were action recognition failures in 81.3% of difficult cases, detail oversight in 78.1%, counting inaccuracies in 60.8%, spatial reasoning failures in 41.7%, and negation handling problems in 31.3% (Chintapatla et al., 17 Jul 2025). These numbers suggest that VE becomes substantially harder when hypotheses require multi-entity attribution under occlusion and crowd density.

Metamorphic testing has provided a complementary evaluation lens. VEglue aligns object mentions in the hypothesis with object regions in the image, then applies three object-aligned joint-erasing metamorphic relations. Across OFA-VE, ALBEF-VE, and LLaVA, VEglue detected 11,609 issues on average, which was 194%–2,846% more than the baselines, and achieved an average Issue Finding Rate of 52.5%, outperforming baselines by 17.1%–38.2%. Retraining on VEglue-generated tests improved accuracy on the new tests by 50.8% on average without sacrificing accuracy on the original test set (Chang et al., 2024). This result underscores that VE robustness is not only a matter of label prediction but also of invariance and sensitivity under controlled semantic perturbations.

These diagnostic results have corrected a common misconception: high VE accuracy does not necessarily imply strong visual grounding. Across corrected labels, crowd benchmarks, prompt ablations, and metamorphic tests, the literature repeatedly finds that models may solve portions of VE through linguistic bias, shallow plausibility, or brittle correlations unless evaluation is explicitly designed to expose those shortcuts.

6. Extensions, applications, and recurrent limitations

VE has increasingly been used as a reusable reasoning module in neighboring multimodal tasks. In VQA, the select-and-rerank framework converts each question–answer pair into a synthetic statement and uses a visual entailment scorer to verify whether the image entails that statement. This VE-based reranking raised VQA-CP v2 performance to 66.73%, a 7.55% absolute improvement over the previous state of the art, and reduced the gap between VQA v2 and VQA-CP v2 to 2.49% (Si et al., 2021). The operational lesson is that entailment-style verification can serve as a second-pass authenticity check over candidate answers.

A different reformulation appears in grounded multimodal named entity recognition. RiVEG decomposes GMNER into MNER–VE–VG, using VE as a binary groundability detector: given an image and a named-entity referring expression, the VE module predicts entailment if the entity is groundable in the image and contradiction if it is not. This VE gate allows the system to pass only entailed entities to the grounding module, and the full framework reports absolute gains of 10.65%, 6.21%, and 8.83% across the three GMNER subtasks (Li et al., 2024). Here VE functions less as an end benchmark than as a bridge between entity recognition and visual grounding.

Defeasible Visual Entailment extends the application space from static verification to belief revision. It introduces an inference-aware evaluator trained by pairwise contrastive learning and categorical information learning, with a multitask loss

$p(y \mid I, H)$5

The resulting evaluator correlates strongly with human judgments of whether an update strengthens or weakens a hypothesis given an image, reaching Pearson $p(y \mid I, H)$6 and Spearman $p(y \mid I, H)$7 for GPT-4o outputs, and can serve as a reward in an iterative update-refinement loop (Zhang et al., 2024). This suggests a route from VE toward dynamic, non-monotonic multimodal inference.

Data creation has likewise become a research topic in its own right. Synthetic-NLI-VE generates one Stable Diffusion image per SNLI premise and shows that a CLIP-based VE classifier trained on synthetic images incurs only a slight drop on real SNLI-VE, with F-score 0.686 compared to 0.703 when trained on real data; on SICK-VTE, the drop is from 0.400 to 0.384 (Reijtenbach et al., 15 Aug 2025). The result does not remove domain-generalization problems, but it indicates that synthetic images can be a practical response to data sparsity in VE.

Across these extensions, several limitations recur. Phrase-level visual entailment in SPT is limited to short visual phrases such as “horse grazing” and does not address sentence-level negation, quantifiers, or causal structure (Izadinia et al., 2015). SNLI-derived datasets inherit language artifacts and, unless corrected, significant label noise in neutral cases (Do et al., 2020). Large vision–LLMs remain sensitive to prompt order and can hallucinate explanations when visual evidence is absent (Pitta et al., 23 Jul 2025). Crowded scenes expose persistent weaknesses in action, counting, spatial, and negation reasoning (Chintapatla et al., 17 Jul 2025). Synthetic data helps with scale but does not solve cross-dataset transfer (Reijtenbach et al., 15 Aug 2025). Taken together, these findings suggest that VE is most informative when treated not as a solved benchmark, but as an active probe of cross-modal alignment, compositional reasoning, and evidential faithfulness.