InspectVLM: Auditing Vision–Language Models

• InspectVLM is an integrative paradigm that systematically audits and diagnoses vision–language models using controlled linguistic perturbations, adversarial analysis, and visualization literacy.
• It employs specific protocols like Language-Guided Invariance Probing and VIB-Probe to measure quantitative metrics such as invariance error, semantic sensitivity, and hallucination risk.
• The framework unifies inspection benchmarks across research and industrial settings, enabling targeted improvements in visual grounding, object detection, and anomaly flagging.

InspectVLM is an integrative paradigm and suite of methodologies for systematically auditing, evaluating, and improving the robustness, reliability, and diagnosis of Vision–LLMs (VLMs) in both generic and domain-specialized visual reasoning tasks. It encompasses a spectrum of approaches—ranging from controlled linguistic perturbation, multi-task industrial inspection, adversarial and reflective auditing, standardized visualization literacy, and hallucination analysis—each providing complementary axes for quantitative and qualitative VLM inspection. This article consolidates core frameworks and diagnostics from foundational studies, with a particular emphasis on the Language-Guided Invariance Probing (LGIP) protocol (Lee, 17 Nov 2025), industrial unified inspection benchmarks (Wallace et al., 3 Aug 2025), adversarial re-examination analysis (Shi et al., 15 May 2026), diagrammatic object detection auditing (Ghosh, 3 Oct 2025), and information-theoretic hallucination probes (Zhang et al., 9 Jan 2026).

1. Language-Guided Invariance Probing: Foundations and Methodology

The cornerstone of InspectVLM is the LGIP framework, which assesses VLM robustness via controlled linguistic perturbations and semantic counterfactuals (Lee, 17 Nov 2025). Given an image–caption pair $(I, c)$, LGIP constructs two disjoint perturbation sets:

• Paraphrases $\mathcal{P}(I, c)$: Meaning-preserving rewrites using natural-language templates (e.g., "a photo of $c$", "this image shows $c$"), while content words remain fixed. Up to $K_\text{same}=6$ deduplicated paraphrases are generated per caption.
• Semantic flips $\mathcal{F}(I, c)$: Meaning-changing edits substituting key tokens for object, color, or number via curated lexicons. E.g., object flip ("a brown dog..." $\to$ "a brown person..."), color flip, and count flip, each capped at $K_\text{diff}=6$ per caption.

For each VLM, cosine similarity $$s(I, c) = \operatorname{sim}(f_\text{img}(I), f_\text{text}(c))$$ is computed on $\ell_2$-normalized visual/textual embeddings. Three central metrics are reported:

• Invariance Error $\mathcal{P}(I, c)$0:

$\mathcal{P}(I, c)$1

Lower values indicate greater paraphrase robustness.

• Semantic Sensitivity Gap $\mathcal{P}(I, c)$2:

$\mathcal{P}(I, c)$3

Higher is better, indicating greater penalization of contradictory text.

• Positive Rate (PR):

$\mathcal{P}(I, c)$4

Reflects fraction of flips where the original caption is preferred; PR$\mathcal{P}(I, c)$50.5 denotes random, PR$\mathcal{P}(I, c)$60.5 reliability.

Experiments on 40k MS COCO images with five captions each demonstrate substantial disparities in robustness across model families. For example, EVA02-CLIP achieves $\mathcal{P}(I, c)$7, $\mathcal{P}(I, c)$8, PR=0.896, while SigLIP base models exhibit a 10$\mathcal{P}(I, c)$9 higher invariance error and negative semantic sensitivity, often preferring flipped captions over true descriptions.

2. Unified Vision–Language Inspection in Industrial Contexts

InspectVLM extends its auditing framework to precision-critical industrial inspection via unified VLMs trained on the InspectMM dataset (Wallace et al., 3 Aug 2025). Here, classification (anomaly flagging), object detection, and keypoint localization are reframed as prompt-based language tasks. The Florence-2 backbone integrates a DaViT vision encoder with a multimodal transformer, using spatial task tokens to encode pixel locations.

Comprehensive evaluation indicates that, while InspectVLM achieves strong image-level classification (accuracy 75.9%) relative to ResNet-50 (66.7%) and parity with keypoint baselines in structured layouts (solar panel grids: PCK 94.5% vs. 97.7%), it underperforms in fine-grained defect detection (object detection precision 16.5% vs. Faster R-CNN's 46.1%), reveals catastrophic label collapse (predicts "yes" for all anomaly prompts under low variability), and exhibits weak visual grounding (outputs remain unchanged when the input image is shuffled under fixed prompts).

Object detection failure analysis attributes errors to quantization artifacts and the lack of explicit multiscale processing. Recommendations include increasing prompt and answer diversity, enforcing visual justification, and penalizing degenerate linguistic answers.

3. Reflective and Adversarial Probing: VisualSwap and VS-BENCH

InspectVLM serves as the organizing protocol for adversarial re-examination benchmarks, notably VisualSwap (Shi et al., 15 May 2026). Using the VS-BENCH dataset (800 paired images from MathVista, MathVerse, MathVision, MMMU-Pro), models are probed by a two-stage protocol: (1) generate a chain-of-thought $c$0 on image $c$1 and question $c$2, then (2) inject a visually similar but semantically contradictory image $c$3 with the same $c$4 and a self-reflective prompt.

Probe accuracy ($c$5) collapses under reflection (Qwen3-235B-Think: $c$6acc = 54.7), with "thinking" models %%%%27$c$28%%%% more vulnerable than instructed variants; multi-turn user-supplied instructions can restore visual grounding ($c$9 +51.3). Attention analysis reveals that autoregressive chain-of-thought generation suppresses attention to new visual tokens, and only explicit user interrupts elevate attention to the image content. This exposes a persistent decoupling between claimed re-examination and actual visual evidence utilization in current VLMs.

4. Visualization Literacy and Chart Reasoning Benchmarks

InspectVLM facilities are extended to standardized visualization reasoning via integration with VLAT (Visualization Literacy Assessment Test) and CALVI (Critical Thinking Assessment for Literacy in Visualizations) (Pandey et al., 20 Mar 2025). Models are evaluated over a randomized, multi-run protocol spanning conventional and deceptive charts, anomaly spotting, and multi-encoding inference.

Empirical results show that state-of-the-art VLMs (e.g., GPT-4, Claude, Gemini, Llama3) exhibit strong performance on bar/line/hierarchical chart questions (up to 96% accuracy) but considerable difficulty on dense scatter/bubble (18.6–61.4%) and misleading visualizations (CALVI mislead: max 30%). Uncertainty management varies, with Gemini omitting questions more frequently (4.5%) and Llama3 showing the largest calibration errors (ECE=0.18). The framework incorporates per-category scores, calibration curves, and dynamic probing into InspectVLM’s diagnostic toolkit.

5. Hallucination Detection via Information Bottleneck Probing

InspectVLM encompasses internal-hallucination analysis using information-theoretic methods such as VIB-Probe (Zhang et al., 9 Jan 2026). This approach collects attention-head outputs from all layers and compresses them via a variational bottleneck, isolating signals predictive of hallucinated text (text not substantiated by the visual evidence).

The VIB-Probe framework achieves state-of-the-art detection (A-PR: POPE 95.6%, AMBER 84.3%) and supports efficient, inference-time mitigation. It enables direct attribution of hallucination risk to a small fraction of heads in deep layers ($c$0), allowing targeted suppression to correct unfaithful output. Comparative assessment shows that logit heuristics and last-layer-only probing are inferior