---
title: Mobile Multi-Query Attention (MQA)
url: https://www.emergentmind.com/topics/mobile-multi-query-attention-mqa
type: topic
---

# Mobile Multi-Query Attention (MQA)

Mobile Multi-Query Attention (MQA) refers to an attention mechanism within transformer architectures where multiple query heads attend to a single shared key and value projection, as opposed to the distinct key/value projections per head in standard multi-head attention. Designed primarily for lowering memory and computational costs, especially during autoregressive inference or on resource-constrained devices, MQA and its generalizations have become central to efficient transformer deployment, particularly in mobile and edge environments.

## 1. Formal Definition and Core Principles

The defining characteristic of Multi-Query Attention is the sharing of key and value heads across all query heads. Given an input sequence $X \in \mathbb{R}^{n \times d}$, standard multi-head attention (MHA) with $h$ heads computes
- $Q = X W^Q \in \mathbb{R}^{n \times h \times (d/h)}$,
- $K = X W^K \in \mathbb{R}^{n \times h \times (d/h)}$,
- $V = X W^V \in \mathbb{R}^{n \times h \times (d/h)}$.

Each head $i$ computes $A_i = \mathrm{softmax}\left(\frac{Q_i K_i^\top}{\sqrt{d/h}}\right) V_i$.

In MQA, there are $h$ distinct query projections but only a single, shared key and value projection per layer:
- $Q = X W^Q \in \mathbb{R}^{n \times h \times (d/h)}$
- $K_\text{shared} = X W^K_\text{shared} \in \mathbb{R}^{n \times 1 \times (d/h)}$
- $V_\text{shared} = X W^V_\text{shared} \in \mathbb{R}^{n \times 1 \times (d/h)}$

Attention for query head $i$:
$$
\mathrm{Attention}_i(Q_i, K_\text{shared}, V_\text{shared}) = \mathrm{softmax}(Q_i K_\text{shared}^\top / \sqrt{d/h}) V_\text{shared}
$$

This approach reduces both the number of key/value parameters and, crucially, the amount of cache storage required during inference by a factor of $h$, making it particularly attractive for deployment in memory-constrained settings and in mobile architectures [2305.13245, 2405.12981].

## 2. Motivations for MQA in Mobile and Efficient Inference

Transformers, especially at large scale, are memory-bound at inference time. The requirement to cache all key and value activations for each attention head at every decoding step becomes a limiting factor for sequence length, batch size, and model size in real-time or on-device scenarios. MQA directly addresses this by enabling:
- **Drastic reduction in per-token memory:** For $h$ heads, the KV-cache in MHA is $O(nhd/h) = O(nd)$; in MQA it is $O(n d/h)$ [2407.10855, 2305.13245].
- **Lower latency:** Reducing memory footprint relieves DRAM bandwidth bottlenecks that dominate on mobile NPUs and DSPs [2404.10518].

In MobileNetV4, Mobile MQA is implemented specifically to optimize memory traffic for visual transformers:
- Only $n+2$ projections (for $n$ query heads, one shared $K$, and one shared $V$) versus $3n$ in MHSA.
- Optionally downsamples $K,V$ spatially using depthwise strided convolution, further boosting operational intensity on mobile accelerators [2404.10518].

## 3. Empirical Performance, Trade-Offs, and Scaling Laws

The core trade-off with MQA is between memory efficiency and model quality:
- **Accuracy Degradation:** Sharing $K$/$V$ across all heads can impair representational fidelity, resulting in lower accuracy compared to MHA. For example, in T5-XXL, MQA achieves ∼46.6 average score versus 47.2 for full MHA on summarization, translation, and QA [2305.13245]. In Pythia-160M, MQA-12 shows ∼49.43% accuracy versus ∼52.79% for baseline [2406.09297].
- **Speed and Memory Gains:** MQA runs up to 6× faster on memory-bound inference and consumes 1.5 GB of cache versus 144 GB for full MHA in OPT-175B (for batch size $b=8$, $s=1024$, $h=96$) [2406.09297].

Scaling laws are favorable: MQA provides a robust reduction in inference-time cost that scales with both model and sequence length, making larger architectures feasible on fixed memory [2305.13245, 2405.12981].

## 4. Generalizations: GQA, WGQA, and Quality-Aware Sharing

Grouped-Query Attention (GQA) interpolates between MQA and MHA by sharing $G$ key/value projections across $h$ query heads, recovering part of the expressive power of MHA while retaining significant memory savings. Weighted Grouped-Query Attention (WGQA) augments GQA by learning per-head weights for the key and value averaging process, resulting in empirically superior results (WGQA improves GQA by ≈0.5% in T5-base on ROUGE and BLEU) [2407.10855].

Quality and Capacity-aware Grouped Query Attention (QCQA) algorithms further refine the head grouping by employing quality-aware (evolutionary) optimization for the grouping structure, yielding up to 20% higher accuracy than GQA for the same cache in Llama2-7B without fine-tuning, and 10.6% higher accuracy after fine-tuning at 50% KV-cache [2406.10247].

Low-Rank KV adaptation (LRKV) encompasses MQA as the $r=0$ special case, while with moderate low-rank head-specific adaptations, it recovers almost all head-diversity and downstream performance at much reduced cache size [2601.11471].

## 5. Fine-Tuning, Uptraining, and Deployment Considerations

MQA and its variants can be efficiently instantiated from existing multi-head checkpoints via a two-stage process:
- **Checkpoint Conversion:** Replace $h$ $K,V$ projections with their mean (for MQA) or group-wise mean (for GQA).
- **Uptraining:** Further pre-train for 5% of the original compute budget to restore most of the lost quality (≥97% of full MHA performance is typical) [2305.13245]. 

WGQA introduces $2h$ new scalars per attention layer for fine-tuning, initialized as mean-pooling coefficients, and recovers ∼0.5% of performance with no extra inference burden [2407.10855].

On mobile platforms, architectures like MobileNetV4 implement Mobile MQA by combining multi-query key/value sharing with INT8 quantization, depthwise convolution spatial reduction, and operator fusion, yielding 39–84% end-to-end speedup with only ≈0.03pt drop in top-1 accuracy versus MHSA [2404.10518]. Inference on memory-constrained hardware thus becomes possible for models and sequences infeasible with full MHA.

## 6. Limitations, Head Diversity, and Adaptive/Hybrid Schemes

Fully sharing keys/values (pure MQA) restricts head diversity: PCA-based measures show a drop from ∼94% rank in MHA to ∼91% for MQA. As a result, downstream task accuracy and perplexity also suffer, with MQA consistently worse than MHA or even GQA [2601.11471]. While query heads can compensate by specializing, this does not fully close the gap.

Hybrid methods such as Mixture of Attention Schemes (MoAS) propose dynamic, per-token routing between MHA, GQA, and MQA via a learned router. This strategy achieves validation loss competitive with MHA while still enabling potential for conditional memory efficiency, although at present hard branching is not enforced during training [2512.20650].

For deployment, if absolute minimal cache is required and slight accuracy loss is acceptable, pure MQA is optimal. For higher quality at moderate cost, GQA, WGQA, QCQA, or low-rank LRKV (with $r \approx d_h/2$) offer practical operating points. Cross-layer KV sharing (CLA, MLKV) can further halve or reduce cache requirements (e.g., CLA2 plus MQA yields a 32× cache reduction at ≤0.1 perplexity cost) [2405.12981, 2406.09297].

## 7. Practical Recommendations and Application Domains

MQA and its extensions are integral to modern transformer deployment in memory-bound or latency-sensitive contexts such as mobile devices, edge accelerators (EdgeTPU, ANE, DSP), and large-scale inference with strict throughput requirements. Recipe and deployment guidance include:
- Use MQA for maximum cache reduction when quality can be modestly sacrificed.
- Prefer GQA or WGQA with moderate group counts for best memory–quality trade-off.
- With off-the-shelf models, uptrain MQA/GQA variants from MHA checkpoints, using mean pooling and 5% retraining for performance recovery.
- Integrate spatial-downsampling and quantization for hybrid vision or multimodal scenarios on mobile [2404.10518].
- For advanced scenarios, use LRKV with moderate rank or QCQA for Pareto-optimal accuracy/memory on large language models.

Across the transformer landscape, MQA and its generalizations constitute the dominant paradigm for scaling attention under practical engineering constraints [2305.13245, 2407.10855, 2404.10518, 2406.10247, 2405.12981, 2601.11471].

Source: https://www.emergentmind.com/topics/mobile-multi-query-attention-mqa