---
title: Grouped-Query Attention with Shared K/V Projections
url: https://www.emergentmind.com/topics/grouped-query-attention-with-shared-k-v-projections
type: topic
---

# Grouped-Query Attention with Shared K/V Projections

Grouped-Query Attention (GQA) with shared key/value (K/V) projections—alternatively termed Grouped-Head Attention—constitutes a class of attention mechanisms for Transformers in which multiple query heads are bundled into groups, each group sharing key and value projections. This design substantially reduces the memory and compute costs associated with the conventional multi-head attention (MHA) paradigm, primarily by lowering the number of distinct K/V projections and corresponding cache entries. Modern variants further optimize group formation via algorithmic or data-driven strategies, dynamically route tokens to per-group experts, introduce low-rank or latent-space compression, and exploit hardware-aware architecture. GQA with shared K/V projections underpins a range of advanced architectures for efficient autoregressive LLM inference and training.

## 1. Mathematical Formulation and Variants

**Standard GQA** partitions $H$ attention heads into $G < H$ groups, indexing each group as $g = 1, \dots, G$, each of size $s = H/G$. For input $X \in \mathbb{R}^{N \times D}$:

- *Queries:* $Q_{i} = X W^{Q}_i \in \mathbb{R}^{N \times d_h}$ for $i = 1 \dots H$. 
- *Keys/Values (shared):* $K^{(g)} = X W^K_g$, $V^{(g)} = X W^V_g$, each in $\mathbb{R}^{N \times d_h}$.
- Every head $i$ in group $g(i)$ computes
  $$
  \mathrm{Attention}_i = \mathrm{softmax}\biggl(\frac{Q_{i} (K^{(g(i))})^{\top}}{\sqrt{d_h}}\biggr)V^{(g(i))}
  $$
  
**Grouped-Tied Attention (GTA)** further ties the K/V projections within each group ($K = V$) and applies rotary positional encoding (RoPE) selectively, nearly halving memory relative to GQA, yet retaining expressiveness by only rotating a low-dimensional subspace [2505.21487].

**Dynamic GQA** variants, such as **Mixture-of-Experts Shared Group Attention (mixSGA)** and **MoSKA**, incorporate data-dependent token routing and allow tokens to select between multiple group sizes or shared K/V contexts based on learned or computed importance [2506.13541, 2511.06010].

**Latent/Low-Rank GQA**—notably in **LRKV** [2601.11471] and the CCA/CCGQA family [2510.04476]—compress the shared K/V representations into a lower-dimensional latent or low-rank space, with per-head or per-token decoding for additional memory and FLOP savings.

## 2. Implementation Algorithms and Grouping Strategies

### Static Grouping

The canonical GQA approach partitions heads evenly by index, with each group mean- or SVD-averaging the K/V parameters [2406.07056]. This can be formalized as:
$$
W^K_g = \text{Avg}_{i \in G_g}(W^K_i), \quad W^V_g = \text{Avg}_{i \in G_g}(W^V_i)
$$
A rigorous SVD-based approach optimizes K/V group projections to minimize cache reconstruction error on calibration data, and can accommodate RoPE by compressing the *actual* cached keys after positional modulation [2406.07056].

### Activation-Informed and Adaptive Grouping

- **AsymGQA** leverages calibration activations, measuring pairwise head similarity via activation statistics, and clusters heads accordingly—potentially with asymmetric group sizes [2406.14963].
- **QCQA** applies multi-objective (accuracy–memory) evolutionary search, using a “weight-sharing error” metric as a proxy for quality, supporting both equal- and arbitrary-cardinality groups [2406.10247].
- **Key-driven and Dynamic GQA** (KDGQA, DGQA) dynamically assign queries to groups based on key norms, updated per batch or via EMA, for vision models [2408.08454].
- **Mixture-of-expert GQA** (mixSGA) learns a per-token router over several pre-set group sizes, relying on an auxiliary loss for training–inference consistency and global weight sharing to avoid excessive parameter cost [2506.13541].

### Chunked and Sparse Shared Attention

**MoSKA** applies chunk-based partitioning of extremely long shared context, routing queries to relevant chunks via learned or heuristic chunk embeddings, then batches all selected requests for high-arithmetic-intensity batched GEMMs [2511.06010].

## 3. Computational Complexity and Hardware-Efficiency

### Cost Reductions

| Variant          | Distinct K/V Heads | KV Cache Size      | Projection Parameters   | KV Load per Token  |
|------------------|--------------------|--------------------|------------------------|--------------------|
| MHA              | $H$                | $2N H d_h$         | $3Hd d_h$              | $2H d_h$           |
| GQA (group $G$)  | $G$                | $2N G d_h$         | $H d d_h + 2G d d_h$   | $2G d_h$           |
| GTA              | $G$ (tied K/V)     | $(G+0.5) d_h$      | $H d d_h + G d d_h$    | $(G+0.5) d_h$      |
| CCGQA ($C_2$)    | $G’=H/C_2$         | $2N G’ d_h$        | $O(d^2/C_2)$           | $2G’ d_h$          |

- Arithmetic intensity improves G-fold over MHA in GQA, and $2G$-fold in GTA [2505.21487, 2511.06010].
- Prefill FLOPs remain unchanged for static GQA, can be lowered by latent compression (CCA/CCGQA): $1/C_2$ scaling [2510.04476].
- Cache memory: GQA achieves linear reduction in K/V cache with respect to $G$; advanced compression/low-rank approaches yield sublinear or multiplicative further savings [2510.04476, 2601.11471].
- Batchwise serving with shared attention (MoSKA) transforms per-request, memory-bound GEMVs into compute-bound batched GEMMs, enabling up to $538.7\times$ throughput increase for high-sharing workloads [2511.06010].

### Parallelism and Practical Optimizations

- GQA and GTA scale identically across tensor-parallel devices; group size $G$ limits duplication-free device count.
- Advanced paging, chunking, and custom fused kernels (e.g., in Opt-GPTQ) reduce fragmentation and maximize hardware utilization [2505.02351].
- MoSKA disaggregates inference across unique-attention and shared-attention nodes, pushing shared K/V serving into high-FLOP, batched compute pools [2511.06010].

## 4. Empirical Results and Comparative Analyses

Multiple evaluations confirm that GQA with shared K/V achieves substantial memory and compute gains, typically with minor (or no) perplexity/accuracy penalties if group formation and head compression are carefully managed:

- **LoRA-finetuned GQA with half/quarter heads** on Llama2-7/13B: drops in zero-shot PPL and accuracy are $<1\%$ (up to $2\%$ for aggressive 75% removal); gain is $\approx 50\%$–$170\%$ throughput [2406.07056].
- **Activation-informed (AsymGQA) and evolutionary (QCQA) groupings** yield up to $7.5\%$ (MMLU, AsymGQA) and $20\%$ (QCQA, no fine-tune) average accuracy improvements over naïve even grouping at constant cache/memory [2406.14963, 2406.10247].
- **MoSKA** achieves end-to-end throughput speedup of up to $538.7\times$ (full disaggregated system) over FlashAttention for $M=256$ concurrent requests with highly shared $16$M-token context [2511.06010].
- **CCA/CCGQA**: $8\times$ KV-cache compression with no drop in MoE model performance; prefill latency reduced by $1.7\times$, backward by $1.3\times$ compared to MHA [2510.04476].
- **mixSGA**: at a $50\%$ KV budget, $+2.3$ ROUGE-L over GQA, and $2$–$4$ lower perplexity on standard benchmarks [2506.13541].
- **GTA**: matches or exceeds GQA quality with roughly half the KV cache, $60.2\%$ zero-shot accuracy on FineWeb-Edu with $1,152$ vs. $2,048$ bytes per token per layer [2505.21487].

## 5. Design Considerations and Limitations

### Trade-Offs and Tuning

- Larger group sizes $G$ (fewer KV heads): higher memory/computation savings, but increasing risk of quality degradation due to underdiversified heads [2406.14963].
- Activation- or quality-aware grouping (AsymGQA, QCQA): preserves performance at high compression rates by clustering heads with similar statistics [2406.14963, 2406.10247].
- Dynamic/statistical routing is crucial when token or group importance skews heavily across context or tasks (mixSGA, MoSKA) [2506.13541, 2511.06010].
- Latency/throughput is a function of batching/group-size and chunk size in shared-K/V architectures; $M \approx 64–256$ is recommended to maximize speedup with minimal extra latency [2511.06010].

### Limitations

- Extreme grouping ($G\ll H$) risks notable loss in fine-grained attention diversity and potential collapse of attention patterns, particularly in model layers highly sensitive to context [2406.14963, 2601.11471].
- MoSKA and similar approaches rely on high instance or request sharing—benefits diminish as shared context fraction decreases [2511.06010].
- For structured tasks or non-text modalities, static or even grouping may underperform without careful adaptation (e.g. vision transformers show gains from DGQA/PGQA [2408.08454]).

## 6. Extensions, Hybrids, and Research Directions

- **MoE-driven and token-wise dynamic GQA** (mixSGA, MoSKA) continue to evolve, mixing adaptive grouping, sparse expert selection, and weight sharing to approach the memory–quality Pareto front [2506.13541, 2511.06010].
- **Low-rank and latent GQA** (LRKV, CCGQA) provide a tunable spectrum from full sharing to per-head expressivity, leveraging latent or SVD-based representations for further compression with minimal loss [2601.11471, 2510.04476].
- **Chunked/shared-KV batching frameworks** (MoSKA) exemplify infrastructural co-design, pairing algorithmic attention optimization with physical/node-level disaggregation [2511.06010].
- **Quality-guided head grouping** (QCQA) demonstrates the use of multi-objective search and cheap surrogate objectives (e.g., weight-sharing error) as scalable alternatives to full retraining for grouping optimization [2406.10247].

## 7. Selected Comparative Table

| Method                | Parameter Overhead    | Cache Compression           | Sample Accuracy Gain/Drop                     | Notes                                  |
|-----------------------|----------------------|----------------------------|-----------------------------------------------|----------------------------------------|
| Static GQA            | $O(2Gd^2/H)$         | $1/G$                      | $<1-2\%$ loss @ 50% heads (LLaMA2-7B)         | Strong baseline [2406.07056]           |
| AsymGQA/QCQA          | +search, $O(1)$      | $>1/G$ (same)              | $+7.5\%$ (MMLU), $+20\%$ (QCQA vs GQA, no FT) | Grouping informed by activations       |
| Opt-GQA w/ GPTQ       | +paging + quant      | $1/G$                      | $+2.6\%$ tokens/sec, $75\%$ mem cut           | Hardware/page/fused kernel optimized   |
| MoSKA (Shared KV)     | infra + router $O(1)$| $N/A$                      | $538.7\times$ throughput at $M=256$           | Requires high shared-context fraction  |
| mixSGA                | +router $O(DE)$      | Adaptive per-token         | $+2.3$ ROUGE-L, $2-4$ lower PPL               | Token-wise expert/group routing        |
| CCGQA/CCA             | +down/up-proj, conv  | $1/C_2$                    | Lossless up to $8\times$ compression (MoE)    | Latent and grouped hybrid              |
| LRKV                  | +low-rank $O(Hdr)$   | $1-\phi(r/H)$ (tunable)    | $18-30\%$ fewer tokens to reach target BPB    | Interpolates MHA ↔ GQA                 |

All claims supported by: [2511.06010], [2506.13541], [2406.10247], [2601.11471], [2406.07056], [2406.14963], [2505.21487], [2505.02351], [2506.17286], [2510.04476], [2408.08454].

Source: https://www.emergentmind.com/topics/grouped-query-attention-with-shared-k-v-projections