Task Vectors are Cross-Modal (2410.22330v1)

Abstract: We investigate the internal representations of vision-and-LLMs (VLMs) and how they encode task representations. We consider tasks specified through examples or instructions, using either text or image inputs. Surprisingly, we find that conceptually similar tasks are mapped to similar task vector representations, regardless of how they are specified. Our findings suggest that to output answers, tokens in VLMs undergo three distinct phases: input, task, and answer, a process which is consistent across different modalities and specifications. The task vectors we identify in VLMs are general enough to be derived in one modality (e.g., text) and transferred to another (e.g., image). Additionally, we find that ensembling exemplar and instruction based task vectors produce better task representations. Taken together, these insights shed light on the underlying mechanisms of VLMs, particularly their ability to represent tasks in a shared manner across different modalities and task specifications. Project page: https://task-vectors-are-cross-modal.github.io.

Overview of "Task Vectors are Cross-Modal"

The paper "Task Vectors are Cross-Modal" provides an in-depth investigation into the internal mechanisms of Vision-and-LLMs (VLMs), specifically focusing on how these models encode tasks. In essence, the authors explore how VLMs, which are capable of handling multi-modal inputs such as text and images, map various task specifications into a shared representation space termed "task vectors." This paper reveals that tasks expressed through different modalities—whether as text examples, image examples, or instructions—are surprisingly encoded into similar task representations, enabling cross-modal transferability.

Key Findings

1. Cross-Modal Task Vectors: The research identifies that VLMs encode tasks into a shared embedding space that transcends the specific input modality. This implies that a task vector derived from one modality (e.g., text) can effectively guide the VLM when applied to a different modality (e.g., image), thereby facilitating cross-modal transfer. The authors demonstrate this with tasks involving mapping countries to capitals, animals to their scientific names, and so on.
2. Token Representation Phases: When processing inputs and generating responses, VLMs undergo a consistent evolution in their token representations across three distinct phases: input, task, and answer. This pattern holds irrespective of the input modality, highlighting a potentially universal mechanism in token representation evolution across VLM layers.
3. Transfer Performance: The authors quantitatively evaluate the performance of task vectors transferred across modalities. For instance, transferring text-derived task vectors to image queries improves accuracy compared to using unimodal image examples alone. Moreover, ensembling instruction-based vectors with exemplar-based vectors enhances the sample efficiency of task representation.
4. Inter-Model Transfer: A notable exploration within the work is the transferability of task vectors from pre-trained language-only models (LLMs) to fine-tuned VLMs. The paper finds that task vectors in LLMs retain a high level of similarity with those in VLMs, allowing for effective cross-modal transfer from text-based queries processed by the LLM to image-based queries handled by the VLM.

Implications and Speculation

The insights from this paper have far-reaching implications for both theoretical understanding and practical applications in AI. The ability of VLMs to encode cross-modal, transferable task representations suggests that these models may be leveraging underlying commonalities between tasks across different modalities, potentially contributing to their generalist capabilities. This cross-modal task encoding could lead to more efficient AI systems that require fewer examples to generalize across different tasks and domains.

Theoretically, these findings challenge researchers to further explore the architectural and training methodologies that facilitate such cross-modal representations, potentially leading to breakthroughs in models of perception and cognition. Practically, the implications for AI development include the potential for creating more robust, efficient models capable of handling a wider range of inputs without exhaustive data requirements for each new task specification.

Conclusion

This paper contributes significantly to our understanding of multi-modal task processing in VLMs by unveiling the cross-modal nature of task vectors. By showing how task representations can transfer across modalities, the research opens new avenues for developing versatile multi-modal AI systems. Future work could investigate refining these models for better understanding diverse input contexts and task complexities, potentially leading to more integrated AI that mirrors human-like perception and problem-solving abilities.