---
title: 'VLM-Lens: Toolkit for VLM Analysis'
url: https://www.emergentmind.com/topics/vlm-lens
type: topic
---

# VLM-Lens: Toolkit for VLM Analysis

VLM-Lens denotes, in its primary and most concrete sense, a systems-oriented interpretability toolkit for open-source vision-language models (VLMs) that standardizes the extraction of intermediate outputs from arbitrary layers during the forward pass, with the stated goal of shifting analysis from end-task behavior to “internal competence” [2510.02292]. In recent arXiv usage, the expression also functions as a broader analytic lens for VLM research: it is used to audit benchmark quality, inspect primitive visual competency, and select vision encoders by cross-modal structural compatibility rather than by size or unimodal accuracy alone [2601.02316][2409.14759][2605.01325].

## 1. Terminological scope and conceptual role

The toolkit paper defines VLM-Lens against a distinction between **behavioral performance** and **internal competence**. Behavioral performance concerns observable input-output behavior on a task; internal competence concerns the internal, potentially generalizable knowledge encoded inside the model. The stated motivation is that exact-match or output-only evaluation can miss information present in hidden representations and can also be misleading when benchmarks admit shortcuts [2510.02292].

The term has subsequently been used in several related but non-identical ways across VLM research. In that broader literature, “VLM-Lens” does not denote one unified benchmark or architecture. Rather, it names a family of diagnostic perspectives that ask what a VLM is actually measuring, encoding, or aligning.

| Usage | Core object | Representative source |
|---|---|---|
| Toolkit | Layer-wise extraction and analysis of open-source VLMs | [2510.02292] |
| Evaluation lens | Faithful, discriminative, and efficient benchmark curation | [2601.02316] |
| Model-selection lens | Vision-encoder selection via Gromov-Wasserstein distance | [2605.01325] |
| Eye-examination lens | Instruction, readiness check, and visual competency inspection via LENS | [2409.14759] |

A recurrent misconception is that VLM-Lens is itself a new VLM architecture. In the toolkit paper, it is not: it is infrastructure for extracting and benchmarking hidden states across many existing VLMs. In the evaluation and model-selection papers, it is even less architectural; there it is a methodological stance toward benchmark design or encoder selection rather than a trainable multimodal model [2510.02292][2601.02316][2605.01325].

## 2. Toolkit architecture and extraction mechanism

As a software system, VLM-Lens is organized around four components: a central execution interface, model-specific wrappers, forward-hook-based instrumentation, and standardized storage. The entry point is `src/main.py`; the common abstraction layer is `src/models/base.py`; individual model implementations live under `src/models`. The extraction mechanism uses PyTorch forward hooks: the toolkit loads a model and preprocessor, registers hooks on user-specified modules, runs inference, captures target tensors, writes them to a database, and unregisters the hooks afterward [2510.02292].

The toolkit’s central technical claim is that it supports extraction of **intermediate outputs from any layer during the forward pass** of supported open-source VLMs. Because the hook callable exposes the input and output tensors of a specified layer, the system is not limited to final hidden states. The paper presents it primarily as a framework for layer-wise output extraction, while also noting relevance to signals such as attention-related tensors and intermediate features before layer normalization [2510.02292].

Configuration is YAML-driven. A config specifies the architecture wrapper, checkpoint path, loading options such as dtype, input directory, prompt, target modules, and output database path. The paper emphasizes that no hard-coded parameters are used throughout the codebase and that new models can be added by implementing a model-specific class without modifying core logic. This design abstracts away architectural heterogeneity in model loading, preprocessing, prompt templating, and module naming conventions [2510.02292].

Standardized storage is implemented with SQLite3. The unified schema stores `name`, `architecture`, `image_path`, `prompt`, `label`, `layer`, `tensor_dim`, and `tensor`. The stated purpose is to make extracted activations queryable and reusable for downstream interpretability workflows rather than tying them to a single bespoke script [2510.02292].

## 3. Supported model ecology, workflow, and systems profile

VLM-Lens currently supports **16 state-of-the-art base VLMs and their over 30 variants**. The explicitly listed base models are Aya-Vision, BLIP-2, CLIP, CogVLM, GLaMM, InternLM-XComposer, InternVL, Janus, LLaVA, MiniCPM-o, MiniCPM-V-2, Molmo, PaliGemma, Pixtral, PerceptionLM, and Qwen2-VL. The paper further states that it supports all variants of these models across sizes, with the only requirement being sufficient hardware resources to load the model [2510.02292].

Operationally, the workflow is inspection-oriented. A user prepares a YAML config, selects target modules by name, and launches extraction through `python src/main.py --config ...`. If the relevant module names are unknown, the toolkit provides `--log-named-modules` to print the internal module graph of a model. This is important because hook points differ substantially across codebases; examples in the paper include `language_model.lm_head` and `vision_model.post_layernorm` [2510.02292].

The system evaluation was run on a single NVIDIA A40 GPU, batch size 1, over a 2,690-example MSCOCO subset, with timings excluding model and processor loading. The reported profile shows substantial variance across models even at similar nominal scale.

| Model | Peak Mem (MB) | Per-instance (s) |
|---|---:|---:|
| CLIP | 616.69 | 0.025 |
| BLIP-2 | 15,261.28 | 0.055 |
| LLaVA-1.5 | 29,031.55 | 0.291 |
| Qwen2-VL | 33,840.66 | 0.279 |

These numbers illustrate a practical point made by the paper: representation extraction is standardized by the toolkit, but its computational cost remains architecture dependent. Memory usage differs substantially across implementations, and lower precision helps but does not eliminate large-system overheads [2510.02292].

The toolkit is released as open source under Apache-2.0, with the repository given as `https://github.com/compling-wat/vlm-lens` [2510.02292].

## 4. Demonstration analyses and empirical findings on hidden representations

The paper demonstrates VLM-Lens with two analyses rather than presenting it as a purely infrastructural release. The first is probing on CLEVR-derived splits; the second is a Stroop-style concept-similarity experiment. Both are intended to show that hidden-state analysis can expose information not visible from benchmark scores alone [2510.02292].

In the probing experiment, the authors construct six CLEVR-based splits—boolean, color, material, number, shape, and size—and extract layer-wise representations of image-text examples. For each example and layer, the token-level representation is mean pooled into a fixed-dimensional vector. A two-layer perceptron with ReLU and 512 hidden units is trained under 5-fold cross-validation, with learning rate, number of epochs, and batch size tuned over \(3^3 = 27\) combinations. Each main probe is paired with a control probe trained on randomly shuffled labels, following control-probing logic rather than treating probe accuracy as self-interpreting [2510.02292].

The reported findings are explicitly comparative. **Qwen-7b** and **MiniCPM-o** yielded nearly perfect probe accuracy in many splits, significantly above control. Last-layer representations often outperformed middle-layer representations, with exceptions such as **BLIP-2** and **PaliGemma**. **LLaVA-1.5-7B** showed only modest gaps between main and control probes, suggesting weaker internal encoding of the tested primitive concepts. Across all models, **color** emerged as the most salient concept, and instruction-following multimodal models such as **Qwen2-VL**, **MiniCPM-o**, and **InternVL** performed well on harder splits like **material**, **number**, and **shape**, especially at the last layer [2510.02292].

The second demonstration uses ambiguous Stroop-style images in which lexical word content, font color, and background color provide conflicting cues. With the prompt “Describe the color in this image in one word,” the analysis does not focus on the generated answer alone. Instead, it projects hidden representations into a PCA subspace built from color-reference images and computes average cosine similarity to matched and mismatched color concepts. In the reported LLaVA-1.5-7B case, all three cues were encoded, but **background color** produced the strongest matched-versus-mismatched separation, and **lexical color content** was more prominent than **font color**. The paper also notes that many PCA components were needed for clear separation, suggesting that color information is not concentrated in a single linear direction [2510.02292].

These demonstrations are used to support the broader claim that internal competence is neither uniform across architectures nor reducible to end-task accuracy. VLM-Lens, in this formulation, is valuable precisely because it can reveal model-specific representational organization that output benchmarks flatten away [2510.02292].

## 5. Broader uses of “VLM-Lens” in evaluation, model selection, and perceptual diagnosis

A second strand of work uses “VLM-Lens” as a benchmark-auditing perspective rather than as software. In **DatBench**, the term denotes a practical lens for asking whether a VLM benchmark is actually measuring the capability it claims to measure. The paper defines three desiderata—**faithfulness to the modality and application**, **discriminability between models**, and **computational efficiency**—and treats evaluation as a data-curation problem. It reports that converting AI2D from multiple-choice to generative evaluation drops average accuracy across 27 models from **77.56\%** to **40.53\%**; that in the **General** capability, **72.07\%** of examples are blindly solvable; and that DatBench, a discriminative subset of a cleaned benchmark family, achieves **13\times average speedup**, up to **50\times**, while closely matching the discriminative power of the original datasets [2601.02316].

A third usage concerns model selection. The Gromov-Wasserstein paper presents a VLM-Lens for choosing vision encoders not by size or zero-shot image accuracy, but by **structural similarity between vision and language representations**. It models visual and textual representations as metric-measure spaces and uses Gromov-Wasserstein distance as a compatibility score:
\[
\text{GW} := \inf_{\pi\in\Pi}\mathcal{E}(\pi).
\]
The empirical study spans **60+ full VLM training runs** and a curated pool whose appendix lists **19** vision encoders while the main text repeatedly says **18**. In the reported correlation analysis, GW achieves \(|r| = 0.6568\), \(|\rho| = 0.5341\), and \(R^2 = 0.4314\), outperforming model size, zero-shot accuracy, RSA, CCA, and MutualNN as a predictor of final VLM performance. The selector is explicitly inference-only: under the default setup with 1,000 image-text pairs, GW takes roughly **~1 minute** on a single GH200 GPU/CPU, versus about **8.5 hours on 8 GH200 GPUs** for a full VLM training run [2605.01325].

A fourth usage is psychophysics-like inspection of visual competence. The **LENS** paper frames “VLM-Lens” as an eye-examination process with three stages—**instruction**, **readiness check**, and **examination**—implemented through a synthetic dataset called **Learning ElemeNt for visual Sensory**. The examination probes color, shape, and semantics. Its most distinctive finding is that tested VLMs are consistently **insensitive to green**: in the reported Sensitivity Area of Color values, green has the largest area across both LLaVA and InstructBLIP variants. The same paper argues that color sensitivity differences are largely driven by the visual encoder, whereas shape sensitivity and semantic recognition vary with LLM capacity even when the visual encoder is fixed [2409.14759].

Taken together, these strands use the term “VLM-Lens” to make different parts of the VLM pipeline legible: hidden-state organization in the toolkit, benchmark validity in DatBench, cross-modal alignability in GW-based model selection, and primitive visual competency in the LENS eye examination. A plausible implication is that the expression has become a general label for methodological instruments that inspect VLMs beneath surface task accuracy, even when the concrete object of analysis differs substantially.

## 6. Limitations, interpretation boundaries, and open directions

The toolkit paper is explicit that VLM-Lens is extraction infrastructure rather than a complete interpretability suite. Its current built-in downstream support is limited; it does not directly package tasks such as attention interpretation and neural circuit discovery, and its inference/database design prevents gradient-based saliency analyses such as Grad-CAM. Model coverage is broad but not universal, and the systems profile shows that large VLMs can still require substantial memory and runtime even when extraction is standardized [2510.02292].

The broader “lens” literature also highlights unsettled points. DatBench notes that discriminative subsets are dynamic: the subset that is optimal for current models may shift as models scale, residual benchmark noise may remain after filtering, and the current objective does **not explicitly enforce diversity**. It also identifies open directions beyond static-image VLM evaluation, including video, GUI grounding, robotics perception, and multimodal agents, and proposes the idea of **DatBench-Live** as a continuously refreshed benchmark [2601.02316].

The GW-based model-selection formulation carries its own caveats. The theory is stated in terms of \(\text{GW}_{\infty}\), whereas the empirical selector uses the practical finite-sample GW objective with \(\ell_1\) discrepancy. The empirical study is also restricted to image-text and to fixed feature-extraction choices such as the last-layer CLS token for vision and the second-to-last hidden layer for text. The paper explicitly notes that broader model coverage would strengthen validation [2605.01325].

The LENS eye-examination work likewise remains selective in scope. It studies two VLM families in detail, relies on LoRA fine-tuning to instill the examination protocol, and reports that some patch tasks remain difficult even after readiness training. Its contribution is diagnostic rather than exhaustive: it establishes that primitive and high-level competency can be disentangled, not that the reported sensitivity patterns are the final word on VLM perception [2409.14759].

In encyclopedia terms, VLM-Lens is best understood as both a specific toolkit and a wider methodological orientation. Its unifying theme is not a single architecture but a shared research claim: VLMs cannot be adequately understood from output scores alone. Whether the object of study is hidden states, benchmark artifacts, encoder compatibility, or primitive perception, the various VLM-Lens formulations all redirect attention from headline performance to the structure of what a vision-language model is actually encoding, using, and measuring [2510.02292][2601.02316][2605.01325][2409.14759].

Source: https://www.emergentmind.com/topics/vlm-lens