Cross-Modal Query-Key Attention

• Cross-Modal Query-Key Attention (CMQKA) is a fusion mechanism that uses projection-based cross attention to connect distinct modality representations while enabling efficient compression and scalable computation.
• CMQKA underpins architectures like Mamba LLM and MolFM-Lite, delivering 20–25% faster training and improved accuracy in complex multimodal tasks.
• Its design supports bidirectional and hierarchical multimodal integration with local masking and binary scaling, achieving state-of-the-art performance in vision-language and audio-visual systems.

Cross-Modal Query-Key Attention (CMQKA) is an architectural mechanism for learning interactions between modalities—such as vision, text, audio, or molecular representations—using projection-based cross-attention. Unlike monomodal attention variants, CMQKA explicitly forms queries ($Q$) in one modality and keys/values ($K, V$) in another, enabling adaptive, parameterized fusion. In modern multimodal systems, it serves as a scalable alternative to full quadratic attention, provides token compression, and facilitates bidirectional or hierarchical information flow across diverse application domains including vision-language LLMs, molecular property prediction, and event-driven multimodal learning (Eom et al., 3 Jun 2026, Shah et al., 25 Feb 2026, Saleh et al., 31 Jan 2026, Wang et al., 2020).

1. Core Principles and Mathematical Definition

CMQKA is structurally a cross-attention module parameterized by learnable projection matrices, operating between two distinct modality representations. Formally, for query matrix $Q \in \mathbb{R}^{L \times d}$ and key/value matrices $K, V \in \mathbb{R}^{T \times d}$, projections $W^q, W^k, W^v \in \mathbb{R}^{d \times d_k}$ map to per-head subspaces (for $h$ heads, $d_k = d/h$). A single-head version computes: $$Q' = QW^q \in \mathbb{R}^{L \times d_k}, \quad K' = KW^k \in \mathbb{R}^{T \times d_k}, \quad V' = VW^v \in \mathbb{R}^{T \times d_k}$$

$$\text{Attention}(Q', K', V') = \text{softmax}\left(\frac{Q' {K'}^\top}{\sqrt{d_k}} + M \right) V' \in \mathbb{R}^{L \times d_k}$$

where $M \in \mathbb{R}^{L \times T}$ is an optional binary mask for locality.

In multi-head CMQKA, each head receives distinct projections and the outputs are concatenated and linearly mapped to form the final representation (Eom et al., 3 Jun 2026, Shah et al., 25 Feb 2026, Wang et al., 2020).

2. Architectures: Design Patterns and Variants

CMQKA is embedded in higher-level systems in several distinct forms:

• Vision-Language Compression Projector (Mamba LLM): A stack of 24 Q-Mamba blocks processes a learnable query bank and static visual features. Each block runs: unidirectional Mamba mixing, CMQKA with visual tokens, and a feed-forward channel mixer. The process adaptively compresses a 2D visual grid (e.g., 729 tokens) into a 1D sequence (e.g., 256 semantic tokens) eliminating manual scan orders. Local masking restricts each query to a spatial neighborhood, improving accuracy (Eom et al., 3 Jun 2026).
• Tri-Modal Molecule Fusion (MolFM-Lite): CMQKA cross-attends three modalities (1D SELFIES, 2D GIN, 3D SchNet). Attention is applied directionally (e.g., 1D queries attending to 2D or 3D keys/values), followed by residual addition and layer normalization. Output vectors are concatenated then processed by a two-layer MLP to produce fused molecular representations. The 3D encoder itself leverages internal conformer ensemble attention with Boltzmann weighting (Shah et al., 25 Feb 2026).
• Hierarchical Spiking Audio-Visual Learning (SNNergy): CMQKA realizes bidirectional, energy-efficient fusion between binary-spike encoding of vision and audio inputs. Projection and masking are performed with binary layers and operations, achieving $K, V$0 complexity per fusion block. CMQKA modules are deployed at multiple spatial resolutions in the SNNergy hierarchy, enabling both local and global cross-modal integration without quadratic cost (Saleh et al., 31 Jan 2026).
• Cross-Media Keyphrase Prediction (M³H-Att): Memory banks for text, vision, and image attributes are processed via pooled queries from one modality cross-attending to key/value banks from another. Multi-head attention layers are stacked, and co-attention outputs from each direction are fused for downstream classification and sequence generation (Wang et al., 2020).

3. Pipeline and Implementation Details

The general operational workflow of CMQKA is as follows:

1. Pre-encoding: Each modality is independently encoded into fixed-dimensional feature vectors using specialized backbones (e.g., SigLIP, GIN, SchNet, CNN, BiGRU).
2. Projection: Separate trainable matrices project $K, V$1, $K, V$2, $K, V$3 from each participating modality into attention subspaces.
3. Attention Computation: Scaled dot-product, often using multi-head decomposition.
4. Fusion/Update: Queried modality’s representation is updated via residual addition of attended outputs, often followed by layer normalization and channel mixing.
5. Stacking and Hierarchy: Multiple CMQKA blocks are stacked (e.g., 24 in Mamba, 4 in M³H-Att, 2–4 in SNNergy) and composed hierarchically or parallelly depending on the workflow.
6. Output: Final fused vectors serve as token sequences or compressed representations for LLMs, property predictors, or classifiers.

Hyperparameters such as query length ($K, V$4), number of blocks, head count ($K, V$5), and per-head dimension ($K, V$6) are systematically tuned for throughput and task-specific accuracy. For example, in Mamba-based LLM fusion, $K, V$7, $K, V$8, and 24 blocks yield the optimal throughput-to-accuracy tradeoff (Eom et al., 3 Jun 2026).

4. Computational Complexity and Scaling Properties

A key motivation for CMQKA is computational scalability:

• Quadratic Baseline: Standard Transformer cross-attention between two sequences of lengths $K, V$9 and $Q \in \mathbb{R}^{L \times d}$0 is $Q \in \mathbb{R}^{L \times d}$1.
• Projector Compression: By projecting $Q \in \mathbb{R}^{L \times d}$2 visual tokens to $Q \in \mathbb{R}^{L \times d}$3 semantic queries ($Q \in \mathbb{R}^{L \times d}$4), CMQKA reduces complexity to $Q \in \mathbb{R}^{L \times d}$5, with empirical speedup (citing the Vision-Language Mamba system) in training and inference times of 20–25%. Example figures: 5.94 s/iter (train) vs. 7.62 s/iter baseline, 0.099 s/iter (infer) vs. 0.129 s baseline for compressed vs. non-compressed projectors (Eom et al., 3 Jun 2026).
• Binary Linear Scaling: In SNNergy, binary CMQKA operations (masking, add/multiply) are $Q \in \mathbb{R}^{L \times d}$6 per block, with operational cost an order of magnitude lower than quadratic attention in high-resolution regimes (Saleh et al., 31 Jan 2026).
• Hierarchical Integration: Multiresolution CMQKA enables efficient fusion at multiple abstraction levels, circumventing prohibitive memory/runtime demands of conventional attention in deep multimodal systems.

5. Empirical Impact and Ablation Analyses

Quantitative and qualitative ablation studies consistently demonstrate the effectiveness of CMQKA:

• Vision-Language LLMs (Mamba projector): Projected visual token compression maintains or improves accuracy across VQA-v2, GQA, VizWiz, TextVQA, POPE, and MMBench, with only slight variability for different $Q \in \mathbb{R}^{L \times d}$7 values. Local masking further increases performance (Eom et al., 3 Jun 2026).

Model	L	VQA-v2	GQA	Train Time (s/iter)
Cobra	–	75.38	58.16	7.62
VL-Mamba	–	74.38	56.69	7.26
Ours (CMQKA)	256	75.01	58.10	5.94

• Molecule Property Prediction (MolFM-Lite): Tri-modal fusion via CMQKA yields a consistent 2–3% AUC gain (e.g., 0.956→0.929 on BBBP) over naive fusion, with conformer attention providing an additional ≈2% improvement (Shah et al., 25 Feb 2026).
• Audio-Visual Fusion (SNNergy): CMQKA-based SNNergy achieves state-of-the-art SNN accuracy on CREMA-D (78.38%), AVE (72.14%), and UrbanSound8K-AV (99.66%), with dramatic energy and memory efficiency gains compared to conventional architectures (Saleh et al., 31 Jan 2026).
• Social Media Keyphrase Prediction: Cross-modal multi-head attention (M³H-Att) increases F1 scores by 2–3 points over traditional co-attention, with qualitative heatmaps confirming that different heads specialize to complementary input regions and modalities (Wang et al., 2020).

6. Bidirectionality, Compression, and Masking Strategies

• Bidirectional/Multidirectional Attention: In both SNNergy and M³H-Att, CMQKA is instantiated in both directions, with each modality serving as both query and key/value bank. This extracts complementary features and facilitates symmetrical information exchange (Saleh et al., 31 Jan 2026, Wang et al., 2020).
• Compression and Pooling: Projected CMQKA compresses high-dimensional modalities (e.g., 2D visual grids, conformer ensembles) into semantic, fixed-length feature sequences through learnable compression, obviating handcrafted scan orders or pooling heuristics (Eom et al., 3 Jun 2026, Shah et al., 25 Feb 2026).
• Masking: Optional local or global mask matrices $Q \in \mathbb{R}^{L \times d}$8 restrict attention to spatial or temporal neighborhoods, improving locality, throughput, and empirical accuracy (local > global in vision-language LLMs) (Eom et al., 3 Jun 2026).

7. Broader Applications and Generalization

CMQKA’s abstract pattern—cross-attention with modality-specific query/key formation and learnable compression—has demonstrable benefits across domains:

• Multimodal LLM input compression (vision-language)
• Tri-modal molecular property fusion (incorporating conformer ensembles)
• Scalable spiking neural audiovisual learning (with hardware-aware efficiency)
• Multimodal social media content understanding with textual-visual-attribute fusion

By replacing or augmenting quadratic attention, CMQKA provides an extensible foundation for efficient, flexible, and accurate multimodal reasoning in next-generation AI systems (Eom et al., 3 Jun 2026, Shah et al., 25 Feb 2026, Saleh et al., 31 Jan 2026, Wang et al., 2020).