---
title: Multi-Query Multi-Head Attention (MQMHA)
url: https://www.emergentmind.com/topics/multi-query-multi-head-attention-mqmha
type: topic
---

# Multi-Query Multi-Head Attention (MQMHA)

Multi-Query Multi-Head Attention (MQMHA) is a family of efficient attention architectures that generalizes classical multi-head attention (MHA) by allowing flexible sharing of key and value projections across queries. Originally introduced to address the bandwidth and latency bottlenecks of autoregressive Transformer inference, MQMHA—particularly its grouped-query forms—balances quality retention and hardware efficiency. Architectures in this family include standard MHA ($h$ query heads and $h$ separate key-value heads), multi-query attention (MQA; $h$ query heads, but a single shared key-value head), and grouped-query attention (GQA; $h$ query heads, $g$ intermediate key-value heads, $1 < g < h$). MQMHA is also instantiated in pooling modules for sequence embedding, particularly in speaker verification, where multiple learnable queries per head are used to aggregate feature statistics.

## 1. Formalization and Mathematical Structure

MQMHA defines the number of query heads as $h$ and the number of key-value heads as $g$ ($1 \leq g \leq h$). For token representations $X\in\mathbb{R}^{n\times d}$, standard MHA involves:

\[
Q = XW^Q, \quad K = XW^K, \quad V = XW^V
\]
with $W^{Q,K,V}\in\mathbb{R}^{d \times d}$, reshaped into $h$ sets of $d_k$-dimensional vectors per head ($d_k = d/h$). Each head computes:
\[
A_i = \operatorname{softmax}\!\left(\frac{Q_i K_i^\top}{\sqrt{d_k}}\right)V_i
\]
and the outputs are concatenated and linearly projected.

In MQA, all query heads have distinct projections ($Q_i$) but share key and value projections:
\[
K_\mathrm{shared} = XW^K_\mathrm{sq},\quad V_\mathrm{shared} = XW^V_\mathrm{sq}
\]
\[
A_i = \operatorname{softmax}\!\left(\frac{Q_i K_\mathrm{shared}^\top}{\sqrt{d_k}}\right)V_\mathrm{shared}
\]

GQA generalizes MQA by partitioning query heads into $g$ disjoint groups, each sharing its own key-value projection. For group assignment $G(i)\in\{1,\dots,g\}$ for head $i$:
\[
Q_i = XW^Q_i
\]
\[
K_j = XW^K_j,\, V_j = XW^V_j, \quad j=1,\dots,g
\]
\[
A_i = \operatorname{softmax}\!\left(\frac{Q_i K_{G(i)}^\top}{\sqrt{d_k}}\right)V_{G(i)}
\]
Special cases: $g=1$ yields MQA; $g=h$ recovers standard MHA [2305.13245, 2406.14963].

## 2. Derivation and Algorithmic Workflow

The essential workflow of grouped-query attention consists of:

1. **Query Projection:** For each head $i=1,\dots,h$, $Q_i$ is computed via $XW^Q_i$.
2. **Key-Value Projection and Grouping:** Compute $g$ distinct $K_j, V_j$ as $XW^K_j, XW^V_j$ for $j=1,\dots,g$. Assign each query head to a group; the mapping $G(i)$ can be fixed (neighbor grouping, even-sized groups) or data-informed (asymmetric, activation-informed grouping).
3. **Attention Computation:** Each head $i$ attends to its group’s key and value via the standard scaled dot-product formula.
4. **Aggregation and Output:** $A_i$ outputs are stacked and passed through an output projection.

Activation-informed grouping (e.g., AsymGQA) uses head activation similarity over a calibration set to construct groupings that maximize downstream accuracy, using a stochastic search and brief fine-tuning passes [2406.14963].

## 3. Uptraining and Conversion Methods

For language models already trained with MHA, conversion to GQA or MQA is achieved by:

- **Mean-Pooling Conversion:** For each GQA group, mean-pool the original $W^K_i$ and $W^V_i$ weights within the group to initialize $W^K_j$ and $W^V_j$. For MQA, mean-pool over all heads.
- **Minimal Uptraining:** Continue pre-training on the same data and schedule for $\alpha$ fraction (e.g., $\alpha=0.05$) of the original steps. This restores quality lost in the direct conversion.
- **Preservation of Output and Query Projections:** $W^Q$ and $W^O$ remain unchanged [2305.13245].

Empirically, mean-pooling outperforms selection or random initialization for grouped KV projections.

## 4. Empirical Benchmarking and Trade-Off Analysis

Extensive experiments on large language models (e.g., T5-XXL) and speaker verification systems demonstrate the efficiency-quality frontier:

- **Decoder Latency:** MQA achieves up to 12× reduction in per-token inference time compared to MHA, with typical BLEU drops ≤ 0.2 [1911.02150].
- **GQA (Intermediate $g$):** With $g=8$ (for $h=64$), GQA recovers ≈99% of MHA quality but requires only 20–30% of full KV bandwidth. MQA reaches minimum bandwidth but with greater quality loss.
- **Pooling Applications:** In speaker verification, MQMHA pooling with $H=16$ heads and $Q=4$ queries/head yields a 6% relative EER reduction over baseline statistics pooling, and up to 14% when combined with margin-based losses [2110.05042].
- **Group Size Recommendation:** Small $g$ ($2$–$8$) achieves the best trade-off, with diminishing hardware savings and increasing quality degradation as $g$ increases further. AsymGQA (activation-informed assignments) can halve KV cost with <1% downstream accuracy loss [2406.14963].
- **Uptraining Fraction:** Increasing uptraining from 5% to 10% further closes the gap between GQA/MQA and full MHA [2305.13245].

| Architecture     | KV Heads ($g$) | Relative Inference Time | Quality (Avg. Score) |
|------------------|----------------|------------------------|---------------------|
| MHA-XXL          | 64             | 2.531                  | 47.2                |
| GQA-8-XXL        | 8              | 0.514                  | 47.1                |
| MQA-XXL          | 1              | 0.489                  | 46.6                |

## 5. Implementation and Hardware Considerations

Grouping of query heads reduces key-value projection parameter count, FLOPs, and especially memory-bandwidth requirements for caching during incremental inference. At group size $g$, KV projections and cache usage are reduced by a factor of $h/g$. For deployment:

- **No Kernel Change:** Standard GQA can use existing attention kernel implementations with minor bookkeeping for group assignment.
- **Cache Access Patterns:** Sharing or grouping of K/V greatly reduces buffer reads/writes per decoding step.
- **Parallelizability:** Group-finding search is embarrassingly parallel across layers, and fine-tuning with low-rank adapters (LoRA) offers efficient convergence [2406.14963].

In pooling, MQMHA maintains linear $T$ time per utterance and $O(dQ)$ parameters, supporting efficient embedding extraction for long sequences [2110.05042].

## 6. Hyperparameter Selection and Practical Guidelines

The optimal configuration depends on both required throughput and task-specific accuracy.

- **If maximum quality is required:** Set $g=h$ (full MHA).
- **If KV bandwidth is limiting:** Use smallest $g$ consistent with acceptable accuracy loss (e.g., $g=2$–$4$ for known LLM workloads).
- **Mean-pooling** is the preferred conversion method for all $g$ [2305.13245].
- **Pool concatenation:** In pooling, always concatenate both weighted mean and standard deviation features.
- **Pooling heads and queries:** In MQMHA pooling, $H\in[8,16]$, $Q\in[2,4]$ are empirically robust [2110.05042].

On long-input tasks (summarization, QA), the bandwidth reduction is most pronounced.

## 7. Extensions and Future Research

Ongoing research explores dynamic, activation-informed grouping (AsymGQA), which refines group assignments based on pre-trained activation similarity for better quality at fixed hardware cost [2406.14963]. Other directions include:

- **Differentiable Grouping:** Using Gumbel-softmax to relax grouping into differentiable parameters for joint training.
- **Layer-wise Adaptive Grouping:** Choosing $g$ per layer to match local capacity demand or identify layer-level KV redundancy.
- **Combination with Sparsity/Low Rank:** Integrating GQA with sparse or low-rank projection techniques for further memory savings.
- **Pool-based Sequence Embedding:** Extending MQMHA pooling to additional domains beyond speaker verification.

The MQMHA family—abstracted through the tunable group parameter $g$—subsumes both classical MHA and aggressive MQA, enabling a controlled spectrum of quality-hardware trade-offs for Transformer models and related architectures [2305.13245, 2406.14963, 1911.02150, 2110.05042].

Source: https://www.emergentmind.com/topics/multi-query-multi-head-attention-mqmha