Lost in Embeddings: Information Loss in Vision-Language Models

Abstract: Vision--LLMs (VLMs) often process visual inputs through a pretrained vision encoder, followed by a projection into the LLM's embedding space via a connector component. While crucial for modality fusion, the potential information loss induced by this projection step and its direct impact on model capabilities remain understudied. We introduce two complementary approaches to examine and quantify this loss by analyzing the latent representation space. First, we evaluate semantic information preservation by analyzing changes in k-nearest neighbor relationships between image representations, before and after projection. Second, we directly measure information loss by reconstructing visual embeddings from the projected representation, localizing loss at an image patch level. Experiments reveal that connectors substantially distort the local geometry of visual representations, with k-nearest neighbors diverging by 40--60\% post-projection, correlating with degradation in retrieval performance. The patch-level embedding reconstruction provides interpretable insights for model behavior on visually grounded question-answering tasks, finding that areas of high information loss reliably predict instances where models struggle.

• The paper quantifies connector-induced information loss by introducing KNOR and patch-level reconstruction metrics to reveal geometric and semantic distortions in VLM connectors.
• It employs a dual-method approach—k-nearest neighbor overlap and per-patch reconstruction loss—to analyze how connector architectures affect downstream tasks like retrieval, captioning, and VQA.
• The study demonstrates that increased information loss in critical image regions correlates with degraded model performance, underscoring the need for adaptive connector designs.

Information Loss in Vision-LLM Connectors: Quantification and Implications

Introduction

This paper presents a systematic investigation into the information loss induced by connector modules in vision-LLMs (VLMs). Connectors, typically implemented as MLPs or attention-based modules, project high-dimensional visual features from pretrained vision encoders into the embedding space of LLMs. While these connectors are essential for modality fusion, their impact on the preservation of semantic and structural information in visual representations has not been rigorously quantified. The authors introduce two complementary methodologies—$k$-nearest neighbor overlap ratio (KNOR) and patch-level embedding reconstruction—to measure and localize information loss, and empirically analyze its consequences for downstream tasks such as retrieval, captioning, and visual question answering (VQA).

Quantifying Information Loss: Methodological Framework

The first approach, KNOR, evaluates the preservation of local geometric structure in the latent space by measuring the overlap of $k$-nearest neighbors for each image before and after the connector projection. A high overlap ratio indicates faithful retention of semantic relationships, while a low ratio signals substantial geometric distortion.

Figure 1: The k-nearest neighbors overlap ratio measures the overlap of an image's neighbors before and after projection. In this example, with k=3, the overlap ratio is 0.67 because two out of the three nearest neighbors are identical in both representation spaces.

The second approach involves training a high-capacity reconstruction model to recover the original vision encoder embeddings from the connector outputs. The per-patch reconstruction loss quantifies the degree of information loss at a fine-grained spatial level, enabling the identification of image regions where critical visual features are lost.

Figure 2: Input image with red answer mask.

Empirical Analysis: Structural and Localized Information Loss

Neighborhood Overlap and Retrieval Performance

Across three representative open-weight VLMs—LLaVA, Idefics2, and Qwen2.5-VL—the KNOR analysis reveals that connectors induce substantial reordering of local neighborhoods in the embedding space. For all models and datasets, the average overlap ratio remains below 0.62, with LLaVA achieving a maximum of 61.6% at $k=100$ and Qwen2.5-VL exhibiting the most severe loss (overlap ratio near 0.1 for small $k$).

Figure 3: Neighborhood overlap ratios across three datasets: SeedBench validation, a 10,000-sample subset of VQAv2 validation, and Vizwiz grounding VQA validation. Analysis using 10, 50, and 100 nearest neighbors shows overlap ratios below 0.62 for all models, suggesting connectors poorly preserve geometric relationships and neighbor rankings for the visual representations.

This geometric distortion correlates with degraded retrieval performance. For LLaVA and Idefics2, recall@5 on the CUB-200-2011 retrieval task drops by 41.4% and 18.8%, respectively, when using post-projection embeddings. Qwen2.5-VL, despite a low overlap ratio, paradoxically improves in semantic retrieval, likely due to continued pretraining of its vision encoder and aggressive patch merging, which fundamentally alters the representation space.

Figure 4: Five nearest neighbors of LLaVA image embeddings.

Patch-Level Reconstruction and Downstream Task Performance

Patch-level reconstruction loss is strongly predictive of downstream task failures in LLaVA and Idefics2. For image captioning, higher average reconstruction loss correlates with lower CIDEr scores. The per-sample Spearman correlation between reconstruction loss and CIDEr is negative and statistically significant for both models, with the highest-loss quartile of images exhibiting markedly worse captioning performance.

For VQA tasks requiring fine-grained visual grounding (e.g., VizWiz), high reconstruction loss in answer-relevant patches reliably predicts incorrect model responses. Visualization of these high-loss regions demonstrates that critical details (such as digits or text) are often lost in the connector projection, directly constraining the LLM's reasoning capabilities.

Figure 5: Correlation between reconstruction loss and question-answering accuracy on the VizWiz grounding VQA task. For LLaVA and Idefics2, all correlations have a p-value < \num{5e-5}.

Figure 6: Additional visualization of high reconstruction loss patches that contributes to model's failure on answering questions that requires recognizing text in the objects. Left: input images with answer-relevant regions in red masks. Middle: signed difference between post-projection embeddings norms and pre-projection embedding norms. Right: normalized norm differences overlay with the input image, with highest loss patches marked in yellow.

Analysis of Connector Architectures

The study distinguishes between feature-preserving connectors (e.g., LLaVA's MLP) and feature-compressing connectors (e.g., Idefics2's perceiver resampler, Qwen2.5-VL's patch merger). Feature-compressing connectors exhibit more severe geometric distortion and higher patch-level information loss, but may yield more semantically meaningful representations for certain tasks, as observed in Qwen2.5-VL.

Procrustes analysis further demonstrates that linear alignment is insufficient to recover the pre-projection structure from post-projection embeddings, especially for LLaVA and Qwen2.5-VL, indicating that the information loss is not merely a rigid transformation but involves non-linear collapse or entanglement.

Figure 7: Alignment visualization for LLaVA pre- and post-projection embeddings through PCA.

Implications and Future Directions

The findings have several implications for the design and evaluation of VLMs:

• Connector-induced information loss is substantial and measurable: Both global (KNOR) and local (reconstruction loss) metrics reveal that current connector architectures often fail to preserve the semantic and structural integrity of visual representations.
• Task performance is bottlenecked by connector fidelity: For tasks requiring fine-grained visual reasoning or grounding, information loss at the connector directly limits model accuracy.
• Connector design should be context-sensitive: The trade-off between semantic abstraction and structural preservation must be explicitly managed. Incorporating reconstruction loss as a regularization objective during pretraining may improve downstream performance.
• Visualization of patch-level loss provides interpretability: Localizing information loss enables targeted diagnosis of model failures and can inform the development of more robust connector architectures.

The authors suggest that future work should explore dynamic or adaptive projection mechanisms, improved feature selection for modality fusion, and the integration of information-preserving objectives into connector training.

Conclusion

This paper establishes a rigorous framework for quantifying and localizing information loss in VLM connectors, demonstrating that current architectures often induce significant geometric and semantic degradation in visual representations. The proposed metrics—KNOR and patch-level reconstruction loss—are shown to be predictive of downstream task performance and provide actionable insights for model development. These results underscore the necessity of rethinking connector design to balance semantic abstraction with the preservation of task-relevant visual information, and open avenues for principled improvements in multimodal model architectures.