A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models (2502.17516v1)

Abstract: The rise of foundation models has transformed machine learning research, prompting efforts to uncover their inner workings and develop more efficient and reliable applications for better control. While significant progress has been made in interpreting LLMs, multimodal foundation models (MMFMs) - such as contrastive vision-LLMs, generative vision-LLMs, and text-to-image models - pose unique interpretability challenges beyond unimodal frameworks. Despite initial studies, a substantial gap remains between the interpretability of LLMs and MMFMs. This survey explores two key aspects: (1) the adaptation of LLM interpretability methods to multimodal models and (2) understanding the mechanistic differences between unimodal LLMs and crossmodal systems. By systematically reviewing current MMFM analysis techniques, we propose a structured taxonomy of interpretability methods, compare insights across unimodal and multimodal architectures, and highlight critical research gaps.

• The paper surveys mechanistic interpretability techniques for multimodal foundation models, organizing them by model family, method type, and application areas.
• The survey finds many LLM interpretability techniques adapt to multimodal models, but challenges remain in interpreting visual embeddings and disentangling modality contributions.
• It discusses applications like model editing and hallucination detection, identifying limitations such as the lack of unified benchmarking for multimodal interpretability.

The paper provides an extensive survey of mechanistic interpretability methods as they are applied to multimodal foundation models, with a particular emphasis on models integrating image and text modalities. The authors organize the existing body of work into a clear taxonomy that is structured along three primary dimensions:

• Model Families:

The survey categorizes multimodal models into major families such as contrastive vision‐LLMs (e.g., CLIP and its variants), text-to-image diffusion models, and generative vision‐LLMs. Each family is characterized by its underlying architecture (e.g., transformer-based encoders versus convolutional U-Nets in diffusion models) and by the specific challenges that arise when processing heterogeneous inputs.

• Interpretability Techniques:

The authors distinguish between adaptations of techniques originally developed for LLMs and methods devised specifically for the multimodal domain. On the one hand, methods including linear probing, logit lens analysis, causal tracing, representation decomposition, and neuron-level attributions are examined for their adaptability and limitations when applied to multimodal systems. On the other hand, specialized approaches that exploit the unique aspects of multimodal architectures are discussed. These include:

• Text-based explanations of internal embeddings: Methods such as TextSpan provide human-interpretable descriptions of attention heads and internal components.
• Network dissection and cross-attention analysis: Techniques that correlate neuron activations with user-defined or automatically extracted semantic concepts are used to unveil cross-modal interactions.
• Training data attribution and feature visualization methods: The paper surveys approaches that trace generated outputs back to influential training samples and visualize spatial relevance through heatmaps aggregated over attention or gradient signals.
  • Downstream Applications:

The survey also addresses how the mechanistic insights obtained from these interpretability methods can be leveraged in practice. Notable applications include:

• In-context learning and task vector manipulation: The use of directional embeddings to compress context and steer model outputs.
• Model editing: Approaches that identify and intervene in specific layers (such as cross-attention modules) to modify factual knowledge, mitigate hallucinations, or adjust stylistic attributes in generated images.
• Hallucination detection and mitigation: Methods that analyze intermediate layer outputs and attention patterns to detect instances where the model’s output conflicts with ground truth, particularly in generative settings.
• Safety and privacy improvements: Techniques to ablate harmful or sensitive directions within the latent space, thereby improving the trustworthiness and robustness of multimodal systems.
• Enhancing compositionality: Frameworks aimed at ensuring that the models accurately capture the relationships between objects, attributes, and scenes, especially in text-to-image generation tasks.

The survey synthesizes a broad range of studies and reveals several key findings. For example, many LLM-derived interpretability techniques are shown to be adaptable to multimodal contexts with moderate modifications. However, unique challenges such as disentangling modality-specific contributions and interpreting visual embeddings in human-understandable terms persist. The work also highlights that while certain methods (such as linear probing and logit lens analysis) offer valuable insights, they often require supervised data or separate classifier training, which presents scalability challenges when applied across the heterogeneous datasets typical in multimodal settings.

The authors do not shy away from discussing limitations and open research directions. They point out, for instance, that higher-level phenomena like sequential batch model editing and the extraction of complete task-specific circuits remain underexplored. In addition, the survey identifies gaps in existing evaluation methodologies—most notably, a lack of unified benchmarking for mechanistic interpretability in multimodal models compared to the rich resources available in the LLM domain.

Overall, the paper is a comprehensive resource that not only organizes the many diverse techniques currently in use, but also critically examines how insights from mechanistic interpretability can inform model improvements. The discussion of both adapted and novel methods, alongside detailed applications and identified challenges, makes clear that while significant progress has been made, there remains substantial room for future research aimed at enhancing controllability, reliability, and safety in multimodal foundation models.