VLM-Lens: Toolkit for VLM Analysis

• VLM-Lens is an interpretability toolkit that extracts hidden representations from VLMs to reveal internal competence beyond visible task performance.
• It uses a YAML-driven configuration, PyTorch forward hooks, and SQLite storage to standardize layer-wise analysis across diverse vision-language models.
• The toolkit also serves as an evaluation lens for auditing benchmark quality, selecting vision encoders, and diagnosing primitive visual competencies.

VLM-Lens denotes, in its primary and most concrete sense, a systems-oriented interpretability toolkit for open-source vision-LLMs (VLMs) that standardizes the extraction of intermediate outputs from arbitrary layers during the forward pass, with the stated goal of shifting analysis from end-task behavior to “internal competence” (Sheta et al., 2 Oct 2025). In recent arXiv usage, the expression also functions as a broader analytic lens for VLM research: it is used to audit benchmark quality, inspect primitive visual competency, and select vision encoders by cross-modal structural compatibility rather than by size or unimodal accuracy alone (Joshi et al., 5 Jan 2026, Hyeon-Woo et al., 2024, Li et al., 2 May 2026).

1. Terminological scope and conceptual role

The toolkit paper defines VLM-Lens against a distinction between behavioral performance and internal competence. Behavioral performance concerns observable input-output behavior on a task; internal competence concerns the internal, potentially generalizable knowledge encoded inside the model. The stated motivation is that exact-match or output-only evaluation can miss information present in hidden representations and can also be misleading when benchmarks admit shortcuts (Sheta et al., 2 Oct 2025).

The term has subsequently been used in several related but non-identical ways across VLM research. In that broader literature, “VLM-Lens” does not denote one unified benchmark or architecture. Rather, it names a family of diagnostic perspectives that ask what a VLM is actually measuring, encoding, or aligning.

Usage	Core object	Representative source
Toolkit	Layer-wise extraction and analysis of open-source VLMs	(Sheta et al., 2 Oct 2025)
Evaluation lens	Faithful, discriminative, and efficient benchmark curation	(Joshi et al., 5 Jan 2026)
Model-selection lens	Vision-encoder selection via Gromov-Wasserstein distance	(Li et al., 2 May 2026)
Eye-examination lens	Instruction, readiness check, and visual competency inspection via LENS	(Hyeon-Woo et al., 2024)

A recurrent misconception is that VLM-Lens is itself a new VLM architecture. In the toolkit paper, it is not: it is infrastructure for extracting and benchmarking hidden states across many existing VLMs. In the evaluation and model-selection papers, it is even less architectural; there it is a methodological stance toward benchmark design or encoder selection rather than a trainable multimodal model (Sheta et al., 2 Oct 2025, Joshi et al., 5 Jan 2026, Li et al., 2 May 2026).

2. Toolkit architecture and extraction mechanism

As a software system, VLM-Lens is organized around four components: a central execution interface, model-specific wrappers, forward-hook-based instrumentation, and standardized storage. The entry point is src/main.py; the common abstraction layer is src/models/base.py; individual model implementations live under src/models. The extraction mechanism uses PyTorch forward hooks: the toolkit loads a model and preprocessor, registers hooks on user-specified modules, runs inference, captures target tensors, writes them to a database, and unregisters the hooks afterward (Sheta et al., 2 Oct 2025).

The toolkit’s central technical claim is that it supports extraction of intermediate outputs from any layer during the forward pass of supported open-source VLMs. Because the hook callable exposes the input and output tensors of a specified layer, the system is not limited to final hidden states. The paper presents it primarily as a framework for layer-wise output extraction, while also noting relevance to signals such as attention-related tensors and intermediate features before layer normalization (Sheta et al., 2 Oct 2025).

Configuration is YAML-driven. A config specifies the architecture wrapper, checkpoint path, loading options such as dtype, input directory, prompt, target modules, and output database path. The paper emphasizes that no hard-coded parameters are used throughout the codebase and that new models can be added by implementing a model-specific class without modifying core logic. This design abstracts away architectural heterogeneity in model loading, preprocessing, prompt templating, and module naming conventions (Sheta et al., 2 Oct 2025).

Standardized storage is implemented with SQLite3. The unified schema stores name, architecture, image_path, prompt, label, layer, tensor_dim, and tensor. The stated purpose is to make extracted activations queryable and reusable for downstream interpretability workflows rather than tying them to a single bespoke script (Sheta et al., 2 Oct 2025).

3. Supported model ecology, workflow, and systems profile

VLM-Lens currently supports 16 state-of-the-art base VLMs and their over 30 variants. The explicitly listed base models are Aya-Vision, BLIP-2, CLIP, CogVLM, GLaMM, InternLM-XComposer, InternVL, Janus, LLaVA, MiniCPM-o, MiniCPM-V-2, Molmo, PaliGemma, Pixtral, PerceptionLM, and Qwen2-VL. The paper further states that it supports all variants of these models across sizes, with the only requirement being sufficient hardware resources to load the model (Sheta et al., 2 Oct 2025).

Operationally, the workflow is inspection-oriented. A user prepares a YAML config, selects target modules by name, and launches extraction through python src/main.py --config .... If the relevant module names are unknown, the toolkit provides --log-named-modules to print the internal module graph of a model. This is important because hook points differ substantially across codebases; examples in the paper include language_model.lm_head and vision_model.post_layernorm (Sheta et al., 2 Oct 2025).

The system evaluation was run on a single NVIDIA A40 GPU, batch size 1, over a 2,690-example MSCOCO subset, with timings excluding model and processor loading. The reported profile shows substantial variance across models even at similar nominal scale.

Model	Peak Mem (MB)	Per-instance (s)
CLIP	616.69	0.025
BLIP-2	15,261.28	0.055
LLaVA-1.5	29,031.55	0.291
Qwen2-VL	33,840.66	0.279

These numbers illustrate a practical point made by the paper: representation extraction is standardized by the toolkit, but its computational cost remains architecture dependent. Memory usage differs substantially across implementations, and lower precision helps but does not eliminate large-system overheads (Sheta et al., 2 Oct 2025).

The toolkit is released as open source under Apache-2.0, with the repository given as https://github.com/compling-wat/vlm-lens (Sheta et al., 2 Oct 2025).

4. Demonstration analyses and empirical findings on hidden representations

The paper demonstrates VLM-Lens with two analyses rather than presenting it as a purely infrastructural release. The first is probing on CLEVR-derived splits; the second is a Stroop-style concept-similarity experiment. Both are intended to show that hidden-state analysis can expose information not visible from benchmark scores alone (Sheta et al., 2 Oct 2025).

In the probing experiment, the authors construct six CLEVR-based splits—boolean, color, material, number, shape, and size—and extract layer-wise representations of image-text examples. For each example and layer, the token-level representation is mean pooled into a fixed-dimensional vector. A two-layer perceptron with ReLU and 512 hidden units is trained under 5-fold cross-validation, with learning rate, number of epochs, and batch size tuned over $3^3 = 27$ combinations. Each main probe is paired with a control probe trained on randomly shuffled labels, following control-probing logic rather than treating probe accuracy as self-interpreting (Sheta et al., 2 Oct 2025).

The reported findings are explicitly comparative. Qwen-7b and MiniCPM-o yielded nearly perfect probe accuracy in many splits, significantly above control. Last-layer representations often outperformed middle-layer representations, with exceptions such as BLIP-2 and PaliGemma. LLaVA-1.5-7B showed only modest gaps between main and control probes, suggesting weaker internal encoding of the tested primitive concepts. Across all models, color emerged as the most salient concept, and instruction-following multimodal models such as Qwen2-VL, MiniCPM-o, and InternVL performed well on harder splits like material, number, and shape, especially at the last layer (Sheta et al., 2 Oct 2025).

The second demonstration uses ambiguous Stroop-style images in which lexical word content, font color, and background color provide conflicting cues. With the prompt “Describe the color in this image in one word,” the analysis does not focus on the generated answer alone. Instead, it projects hidden representations into a PCA subspace built from color-reference images and computes average cosine similarity to matched and mismatched color concepts. In the reported LLaVA-1.5-7B case, all three cues were encoded, but background color produced the strongest matched-versus-mismatched separation, and lexical color content was more prominent than font color. The paper also notes that many PCA components were needed for clear separation, suggesting that color information is not concentrated in a single linear direction (Sheta et al., 2 Oct 2025).

These demonstrations are used to support the broader claim that internal competence is neither uniform across architectures nor reducible to end-task accuracy. VLM-Lens, in this formulation, is valuable precisely because it can reveal model-specific representational organization that output benchmarks flatten away (Sheta et al., 2 Oct 2025).

5. Broader uses of “VLM-Lens” in evaluation, model selection, and perceptual diagnosis

A second strand of work uses “VLM-Lens” as a benchmark-auditing perspective rather than as software. In DatBench, the term denotes a practical lens for asking whether a VLM benchmark is actually measuring the capability it claims to measure. The paper defines three desiderata—faithfulness to the modality and application, discriminability between models, and computational efficiency—and treats evaluation as a data-curation problem. It reports that converting AI2D from multiple-choice to generative evaluation drops average accuracy across 27 models from 77.56\% to 40.53\%; that in the General capability, 72.07\% of examples are blindly solvable; and that DatBench, a discriminative subset of a cleaned benchmark family, achieves 13\times average speedup, up to 50\times, while closely matching the discriminative power of the original datasets (Joshi et al., 5 Jan 2026).

A third usage concerns model selection. The Gromov-Wasserstein paper presents a VLM-Lens for choosing vision encoders not by size or zero-shot image accuracy, but by structural similarity between vision and language representations. It models visual and textual representations as metric-measure spaces and uses Gromov-Wasserstein distance as a compatibility score: $\text{GW} := \inf_{\pi\in\Pi}\mathcal{E}(\pi).$ The empirical study spans 60+ full VLM training runs and a curated pool whose appendix lists 19 vision encoders while the main text repeatedly says 18. In the reported correlation analysis, GW achieves $|r| = 0.6568$, $|\rho| = 0.5341$, and $R^2 = 0.4314$, outperforming model size, zero-shot accuracy, RSA, CCA, and MutualNN as a predictor of final VLM performance. The selector is explicitly inference-only: under the default setup with 1,000 image-text pairs, GW takes roughly ~1 minute on a single GH200 GPU/CPU, versus about 8.5 hours on 8 GH200 GPUs for a full VLM training run (Li et al., 2 May 2026).

A fourth usage is psychophysics-like inspection of visual competence. The LENS paper frames “VLM-Lens” as an eye-examination process with three stages—instruction, readiness check, and examination—implemented through a synthetic dataset called Learning ElemeNt for visual Sensory. The examination probes color, shape, and semantics. Its most distinctive finding is that tested VLMs are consistently insensitive to green: in the reported Sensitivity Area of Color values, green has the largest area across both LLaVA and InstructBLIP variants. The same paper argues that color sensitivity differences are largely driven by the visual encoder, whereas shape sensitivity and semantic recognition vary with LLM capacity even when the visual encoder is fixed (Hyeon-Woo et al., 2024).

Taken together, these strands use the term “VLM-Lens” to make different parts of the VLM pipeline legible: hidden-state organization in the toolkit, benchmark validity in DatBench, cross-modal alignability in GW-based model selection, and primitive visual competency in the LENS eye examination. A plausible implication is that the expression has become a general label for methodological instruments that inspect VLMs beneath surface task accuracy, even when the concrete object of analysis differs substantially.

6. Limitations, interpretation boundaries, and open directions

The toolkit paper is explicit that VLM-Lens is extraction infrastructure rather than a complete interpretability suite. Its current built-in downstream support is limited; it does not directly package tasks such as attention interpretation and neural circuit discovery, and its inference/database design prevents gradient-based saliency analyses such as Grad-CAM. Model coverage is broad but not universal, and the systems profile shows that large VLMs can still require substantial memory and runtime even when extraction is standardized (Sheta et al., 2 Oct 2025).

The broader “lens” literature also highlights unsettled points. DatBench notes that discriminative subsets are dynamic: the subset that is optimal for current models may shift as models scale, residual benchmark noise may remain after filtering, and the current objective does not explicitly enforce diversity. It also identifies open directions beyond static-image VLM evaluation, including video, GUI grounding, robotics perception, and multimodal agents, and proposes the idea of DatBench-Live as a continuously refreshed benchmark (Joshi et al., 5 Jan 2026).

The GW-based model-selection formulation carries its own caveats. The theory is stated in terms of $\text{GW}_{\infty}$, whereas the empirical selector uses the practical finite-sample GW objective with $\ell_1$ discrepancy. The empirical study is also restricted to image-text and to fixed feature-extraction choices such as the last-layer CLS token for vision and the second-to-last hidden layer for text. The paper explicitly notes that broader model coverage would strengthen validation (Li et al., 2 May 2026).

The LENS eye-examination work likewise remains selective in scope. It studies two VLM families in detail, relies on LoRA fine-tuning to instill the examination protocol, and reports that some patch tasks remain difficult even after readiness training. Its contribution is diagnostic rather than exhaustive: it establishes that primitive and high-level competency can be disentangled, not that the reported sensitivity patterns are the final word on VLM perception (Hyeon-Woo et al., 2024).

In encyclopedia terms, VLM-Lens is best understood as both a specific toolkit and a wider methodological orientation. Its unifying theme is not a single architecture but a shared research claim: VLMs cannot be adequately understood from output scores alone. Whether the object of study is hidden states, benchmark artifacts, encoder compatibility, or primitive perception, the various VLM-Lens formulations all redirect attention from headline performance to the structure of what a vision-LLM is actually encoding, using, and measuring (Sheta et al., 2 Oct 2025, Joshi et al., 5 Jan 2026, Li et al., 2 May 2026, Hyeon-Woo et al., 2024).