CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion (2512.19535v1)

Abstract: Vision-LLMs (VLMs) are commonly trained by inserting image tokens from a pretrained vision encoder into the textual stream of a LLM. This allows text and image information to fully attend to one another within the model, but becomes extremely costly for high-resolution images, long conversations, or streaming videos, both in memory and compute. VLMs leveraging cross-attention are an efficient alternative to token insertion but exhibit a clear performance gap, in particular on tasks involving fine-grained visual details. We find that a key to improving such models is to also enable local text-to-text interaction in the dedicated cross-attention layers. Building on this, we propose CASA, Cross-Attention via Self-Attention, a simple and efficient paradigm which substantially reduces the gap with full token insertion on common image understanding benchmarks, while enjoying the same scalability as cross-attention models when applied to long-context multimodal tasks such as streaming video captioning. For samples and code, please see our project page at https://kyutai.org/casa .

• The paper presents CASA, a novel vision-language fusion method that integrates localized self-attention with cross-attention to restore text-to-text interactions.
• CASA achieves up to 74% memory reduction and over 80% training speedup compared to token-insertion methods, enabling efficient streaming and real-time inference.
• The architecture supports multiple integration modes (CASA@, CASA+, CASAV) and can retrofit pretrained models with minimal performance loss on fine-grained tasks.

Efficient Vision-Language Fusion via CASA: Cross-Attention via Self-Attention

Introduction and Motivation

This paper introduces CASA, a novel architecture for vision-language fusion that targets the longstanding trade-off between fusion fidelity and computational efficiency in large vision-LLMs (VLMs). Traditional approaches interleave visual tokens into LLM input streams ("token insertion"), enabling effective multimodal interaction via self-attention but incurring prohibitive memory and computational costs for high-resolution inputs or long sequences. Cross-attention-based fusion provides better scalability, particularly for streaming scenarios, but exhibits strong performance gaps on tasks requiring fine-grained visual reasoning. The core hypothesis is that the lack of text-to-text interaction in standard cross-attention fusion underlies these deficits. CASA augments cross-attention layers by allowing text tokens to self-attend within local windows, thereby reinstating text-to-text communication during multimodal fusion.

CASA Architecture

CASA extends standard cross-attention by enabling each text token to attend to a concatenation of image tokens and previous text tokens within a local attention window. This mechanism maintains causality and supports efficient incremental inference in streaming contexts. Importantly, image tokens are not propagated through LLM feedforward networks (FFNs) nor stored in the KV cache, drastically reducing compute and memory requirements compared to token insertion.

CASA is flexible and can be integrated in several architectural configurations:

• Parallel Fusion: CASA outputs are summed with self-attention at each LLM layer (CASA@).
• Prepending Fusion: CASA precedes every self-attention layer (CASA+).
• Replacement: CASA layers substitute a subset of self-attention layers (CASAV).

The modularity of CASA allows for both training new VLMs from text-only LLMs and retrofitting pretrained token-insertion VLMs by learning only the additional CASA layers.

Experimental Results

The evaluation covers standard VLM benchmarks: document understanding (DocVQA), chart reasoning (ChartQA, InfoVQA), OCR (OCRBench, TextVQA), general visual QA (AI2D, GQA, MME, RealWorldQA), and video QA (MVBench, VideoMME, NEXT-QA, PerceptionTest, MLVU). Models are compared under controlled settings for fusion technique, backbone size, and training data.

Key findings:

• All CASA variants substantially outperform classic cross-attention models (e.g., mPLUG-Owl3), including larger models;
• The gap to token-insertion fusion is considerably narrowed, especially for tasks demanding high-resolution or fine-grained spatial/textual detail;
• CASA+ and CASA@ incur an average performance drop of about 5-7% relative to full token-insertion, mainly on chart/infographic understanding tasks, but retain the bulk of insertion-based accuracy elsewhere;
• Adapting a frozen, pretrained insertion-based VLM (Qwen2.5-VL-3B) with CASA attains nearly base-model performance with significantly reduced memory and latency during multi-turn or streaming inference.

On live video captioning (using the demanding LiveSports3K and LiveCC benchmarks), CASA enables low-latency, real-time inference with near-constant memory consumption, successfully demonstrating scalability over hundreds of frames, where token insertion (even with heavy compression) exhausts GPU memory and incurs severe slowdowns.

Efficiency Analysis

CASA's efficiency gains derive from three sources:

1. No FFN propagation for image tokens: Attention and memory costs scale linearly with the number of images, not images × sequence length;
2. No KV cache updates for image tokens: Enables long text contexts without context length bottlenecks;
3. Blockwise attention: Training uses FlashAttention2's efficient blockwise attention, with image insertion points delimiting local windows.

Empirical profiling shows that CASA offers up to 74% memory reduction and over 80% training speedup versus token-insertion architectures processing the same number of image and text tokens.

Ablations and Architectural Insights

Several ablation studies clarify CASA's behavioral advantages:

• Self-attention is critical: Masking self-attention (i.e., removing text-to-text interaction in CASA) severely degrades performance (see Table 8).
• Placement of CASA layers: Uniform, sparse insertion of CASA layers throughout the transformer yields the best trade-off between performance and efficiency; excessive CASA layers are detrimental due to their limited global receptive field compared to self-attention.
• Image embedding updates: Forwarding image tokens into FFNs, as in recent cross-attention VLMs, introduces modest accuracy gains at substantial memory costs.
• Token compression (Q-Formers): Compression before token insertion helps general VQA but drastically impairs fine-grained image and text reading, and does not solve streaming scalability due to persistent KV cache growth.
• Implicit gating: Inclusion of self-attention in CASA layers acts as a natural gating mechanism, balancing image and text token contributions via the attention softmax, which explicit gating schemes fail to replicate.

Implications and Future Directions

CASA provides an effective and scalable alternative to token-insertion vision-language fusion, markedly improving the efficiency of VLMs, especially for streaming and long-context tasks. The implicit gating mechanism—enabled by localized self-attention—addresses the key weakness of standard cross-attention, allowing CASA to approach the performance of insertion-based models without the associated overhead.

Practical implications are especially pronounced for real-time video understanding, where CASA’s memory footprint and inference latency remain stable as video length grows, opening avenues for deployment in continuous perception, multi-turn dialog, and edge computing settings.

Theoretically, the results highlight the importance of maintaining rich intra-modal interaction when fusing modalities; simplistic cross-attention that neglects text-to-text dynamics cannot match insertion-based approaches on complex tasks. Further research could explore:

• Combining CASA with advanced image or video token compression for ultra-long context reasoning under limited resources;
• Application to cross-modal multi-document question answering and dialog, where complex interrelations among images/texts occur;
• Extending CASA-like fusion mechanisms to other non-vision modalities (e.g., audio, structured data).

Conclusion

CASA advances the design of efficient and performant vision-language fusion by augmenting cross-attention with localized self-attention among text tokens. Experiments demonstrate that CASA models close much of the gap to token-insertion architectures, especially on high-resolution and document-centric tasks, while achieving substantial gains in memory efficiency and inference speed. The design enables the construction of scalable, low-latency VLMs suitable for real-world multimodal interaction scenarios, particularly streaming video captioning.

This methodology signifies an important step toward practical, high-fidelity multimodal LLMs, offering a foundation for future explorations in both scaling and task specialization in multimodal AI systems.

---

Referenced paper: "CASA: Cross-Attention via Self-Attention for Efficient Vision-Language Fusion" (2512.19535)