Multi-Head Attention RAG Overview

• MHA-RAG is a framework that leverages multi-head attention to control retrieval, context selection, and prompt synthesis in generation tasks.
• It integrates variants such as head reweighting, soft-prompt synthesis, and per-head retrieval keys to address diverse design challenges.
• Empirical evaluations show significant gains in accuracy, retrieval efficiency, and minimal inference overhead across multiple benchmarks.

Multi-Head Attention Retrieval-Augmented Generation (MHA-RAG) denotes a family of retrieval-augmented generation designs in which multi-head attention is used as the principal control surface for retrieval, context selection, prompt synthesis, or generation-time context use. In the current literature, the label is used in several non-identical senses: PEAR defines MHA-RAG as methods that improve how a Transformer’s multi-head attention uses retrieved context by explicitly modulating individual head contributions; a later work uses the same name for a framework that encodes retrieved exemplars into a compact soft prompt with one virtual token per head; and MRAG states that “Multi-Head Attention RAG (MHA-RAG)” is equivalent to “Multi-Head RAG (MRAG)” and performs retrieval with head-specific embeddings rather than a single collapsed vector (Tan et al., 2024, Jain et al., 6 Oct 2025, Besta et al., 2024). Across these usages, the recurring design choice is to preserve, weight, or repurpose head-specific structure instead of relying only on conventional single-vector retrieval or unmodified attention aggregation.

1. Terminological scope and research variants

The term is not attached to a single canonical pipeline. Some papers use MHA-RAG for generator-internal interventions, some for prompt construction from retrieved exemplars, and some for retrieval-stage mechanisms. Related work extends the same design space to attention-guided pruning, training-free long-document retrieval from attention maps, query-aware graph attention, retrieval-aware prompting, and entropy-controlled attention (Tan et al., 2024, Jain et al., 6 Oct 2025, Besta et al., 2024, Fang et al., 13 Mar 2025, Fu et al., 12 Feb 2026, Agrawal et al., 25 Jul 2025, Ye et al., 2024, Wang et al., 7 Aug 2025).

Line of work	Use of multi-head attention	Representative paper
Head reweighting in the generator	Downweight suppressive heads with learned scalars	(Tan et al., 2024)
Soft-prompt synthesis	Encode retrieved exemplars into $H$ virtual prompt tokens	(Jain et al., 6 Oct 2025)
Multi-aspect retrieval	Index per-head embeddings and fuse head-wise searches	(Besta et al., 2024)
Attention-guided pruning	Aggregate head attention from a focal token to score context tokens	(Fang et al., 13 Mar 2025)
Long-document retrieval	Use selected-layer attention maps and entity-aware scope expansion	(Fu et al., 12 Feb 2026)
Query-aware graph retrieval	Use multi-head graph attention and query-guided pooling for reranking	(Agrawal et al., 25 Jul 2025)

This distribution of meanings matters because empirical claims are architecture-specific. “Zero inference overhead” is a defining property of PEAR, not of all MHA-RAG systems; “exemplar-order invariance” is central to the soft-prompt formulation, not to MRAG; and “training-free” describes MRAG and AttentionRetriever, but not the prompt-encoder variant or R$^2$AG. A common misconception is therefore to treat MHA-RAG as a single method rather than a research area organized around head-level control.

2. Recurrent mathematical patterns

One recurring formulation begins from standard Transformer attention. For layer $l$ and head $h$,

$$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$

and the head output is

$$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$

The ordinary layer aggregation is

$O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$

PEAR turns this into an explicit MHA-RAG template by introducing head weights $\alpha_{l,h}$: $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$ In PEAR, $\alpha_{l,h}$ equals a learned scalar $^2$0 for heads in a suppressive set $^2$1, and $^2$2 otherwise (Tan et al., 2024).

A second pattern treats each head output as a prompt token rather than as an internal correction. In the soft-prompt MHA-RAG formulation, for head $^2$3,

$^2$4

$^2$5

The $^2$6 vectors in $^2$7 are prepended to the frozen LM as $^2$8 virtual prompt tokens (Jain et al., 6 Oct 2025).

A third pattern uses per-head activations as retrieval keys. MRAG extracts the last decoder block’s head outputs before the output projection $^2$9, and for a chunk $l$0 defines

$l$1

where $l$2 is the last token. Retrieval is then performed independently in each head-specific embedding space and fused with head-importance weights (Besta et al., 2024).

These formulations show that MHA-RAG is not defined by a single optimization objective. It is defined by where head structure is surfaced: inside the layer as a reweighting variable, at the prompt boundary as virtual tokens, or at the retrieval boundary as multiple aspect-specific keys.

3. Generator-side head control

PEAR is the most explicit generator-internal instantiation. It identifies “suppressive” heads on a context-copying proxy task, using causal mediation analysis at position $l$3 on a duplicated token sequence $l$4. For each head, PEAR replaces $l$5 with a mean vector and scores the intervention by

$l$6

Larger $l$7 indicates stronger suppression of copying. The top-$l$8 heads by this score form $l$9; the reported choices are $h$0 for Llama2-7B-chat-4k, $h$1 for OPT-6.7B-2k, and $h$2 for Baichuan-13B-chat-4k. Only these $h$3 scalars are trained, with the base model frozen, using 500 proxy samples with $h$4, AdamW at learning rate $h$5, $h$6, one epoch, BF16 precision, and an A100-PCIE-40GB. After training, each $h$7 is folded into the corresponding block of $h$8, so inference incurs no extra operations, time, or memory (Tan et al., 2024).

The empirical profile of PEAR is unusually sharp. On 2WikiMultiHopQA, MuSiQue-Full, and Qasper with Llama2-7B-chat-4k, exact match rises from $h$9 for the base model to $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$0, for an average increase from $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$1 to $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$2. Practical measurements show matched inference cost with the base model: $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$3 s / $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$4 GB on 2WikiMultiHopQA, $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$5 s / $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$6 GB on MuSiQue, and $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$7 s / $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$8 GB on Qasper. The same paper reports robustness across positional schemes on an MDQA experiment: Llama2-7B-chat-4k improves from $$Q^{(l,h)} = X W_Q^{(l,h)}, \quad K^{(l,h)} = X W_K^{(l,h)}, \quad V^{(l,h)} = X W_V^{(l,h)},$$9 to $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$0, OPT-6.7B-2k from $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$1 to $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$2, and Baichuan-13B-4k from $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$3 to $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$4, winning $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$5 position comparisons. On MMLU, Llama2-7B-chat-4k changes only from $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$6 to $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$7, indicating negligible loss of parametric knowledge (Tan et al., 2024).

BEE-RAG reformulates the same generator-side problem in terms of entropy invariance rather than suppressive heads. It modifies attention with a chunk-level balancing factor $$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$8, either as a per-chunk temperature or as a logit-side bias, and analyzes the entropy

$$a^{(l,h)}=\operatorname{softmax}\!\left(\frac{Q^{(l,h)}K^{(l,h)\top}}{\sqrt{d_k}}\right)V^{(l,h)}.$$9

as context length grows. Its MHA-RAG extension allows either shared $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$0 across heads or per-head $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$1, with the stated objective of keeping head-wise entropy close to a target regime while reducing attention dilution. The zero-shot variant, Zero-BEE, raises Qwen-2.5-7B average EM/F1 from $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$2 to $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$3, while Light-BEE raises the same model from LoRA’s $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$4 to $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$5. Only $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$6 parameters are updated in Light-BEE (Wang et al., 7 Aug 2025).

A more structural extension is Interleaved Head Attention, which is not itself a full RAG pipeline but is proposed as a RAG-relevant replacement for standard MHA in reasoning-heavy generators. IHA creates $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$7 pseudo-heads per head, mixes $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$8, $O^{(l)}=[a^{(l,1)},\ldots,a^{(l,H)}]W_O^{(l)}.$9, and $\alpha_{l,h}$0 across all heads before attention, and induces up to $\alpha_{l,h}$1 attention patterns per head with parameter overhead $\alpha_{l,h}$2. The paper attributes to standard MHA a fundamental limitation: $\alpha_{l,h}$3 heads produce exactly $\alpha_{l,h}$4 independent attention matrices with no communication between heads during attention computation. In RAG-oriented evaluation, IHA improves Multi-Key retrieval on RULER by $\alpha_{l,h}$5-$\alpha_{l,h}$6 across $\alpha_{l,h}$7k-$\alpha_{l,h}$8k contexts, and after fine-tuning on OpenThoughts improves GSM8K by $\alpha_{l,h}$9 and MATH-500 by $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$0 under Majority Vote over full attention (Duvvuri et al., 24 Feb 2026).

4. Prompt synthesis and retrieval-aware conditioning

The 2025 MHA-RAG framework uses retrieved exemplars not as long textual demonstrations but as a compact soft prompt generated by a dedicated multi-head attention module. The query is encoded as $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$1, each exemplar $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$2 as $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$3, and each head produces one prompt token $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$4 via scaled dot-product attention. Because the head output is a symmetric function of the exemplar set, the resulting prompt is invariant to exemplar order. The generator receives only $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$5 tokens rather than $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$6, where $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$7 is the tokenized exemplar length. The number of heads $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$8 is therefore both the number of virtual prompt tokens and the main capacity hyperparameter. When $$O^{(l)}_{\text{MHA-RAG}}= [\alpha_{l,1}a^{(l,1)},\ldots,\alpha_{l,H}a^{(l,H)}]W_O^{(l)}.$$9, increasing $\alpha_{l,h}$0 brings little benefit; when $\alpha_{l,h}$1, increasing $\alpha_{l,h}$2 generally improves accuracy, with diminishing returns beyond approximately $\alpha_{l,h}$3-$\alpha_{l,h}$4 depending on the task and model (Jain et al., 6 Oct 2025).

This formulation is explicitly evaluated on BACE, BBBP, ClinTox, and PubMedQA using Effective Accuracy, the geometric mean of TPR and TNR. The paper reports an average improvement of $\alpha_{l,h}$5 points over standard RAG across models and tasks, together with an approximately $\alpha_{l,h}$6 reduction in inference GFLOPs across $\alpha_{l,h}$7. The reported order-invariance result is equally stark: the standard deviation under exemplar permutation is $\alpha_{l,h}$8 for MHA-RAG, versus $\alpha_{l,h}$9-$^2$00 or $^2$01-$^2$02 points for baselines depending on the table cited. Representative gains include Qwen3-4B on ClinTox, from $^2$03 to $^2$04, Qwen3-0.6B on BBBP, from $^2$05 to $^2$06, and Llama3.2-3B on BBBP, from $^2$07 to $^2$08. The paper also notes that PubMedQA may show a modest drop relative to text RAG at some scales because multi-thousand-token documents are compressed into at most $^2$09 prompt vectors, although the method remains much cheaper and still outperforms document-free prompting (Jain et al., 6 Oct 2025).

R$^2$10AG occupies a nearby but distinct point in the design space. It does not synthesize a soft exemplar set from head outputs; instead, it passes retrieval-side signals into generation through a small attention module, R$^2$11-Former, and retrieval-aware prompting. For each retrieved document, a feature token $^2$12 is constructed from relevance, precedent similarity, and neighbor similarity. A Transformer encoder with multi-head self-attention contextualizes these tokens into retrieval-information embeddings $^2$13, which are projected into the LM embedding space and prepended as one “retrieval token” per document. The trainable additions are small: the paper reports approximately $^2$14M parameters for R$^2$15-Former and $^2$16M including the projection layer, while keeping the $^2$17B-parameter LLaMA2-7B frozen. The stated inference overhead is approximately $^2$18 (Ye et al., 2024).

The reported improvements are substantial in low-resource frozen-model settings. On NQ-30, accuracy rises from $^2$19 to $^2$20; on HotpotQA, accuracy/F1 rises from $^2$21 to $^2$22; on MuSiQue, from $^2$23 to $^2$24; and on 2WikiMultiHopQA, from $^2$25 to $^2$26. Attention visualization in the same work shows elevated attention on retrieval-information tokens and earlier emphasis on relevant documents, supporting the paper’s claim that retrieval information functions as an anchor for downstream generation (Ye et al., 2024).

5. Retrieval, pruning, and graph-based use of head-specific signals

MRAG is the clearest retrieval-stage use of the MHA-RAG label. Instead of indexing a single $^2$27-dimensional vector from the final decoder block, it stores $^2$28 head-specific vectors of dimension $^2$29 each for Mistral-7B-based embedding models. For head $^2$30, an offline importance score is computed as $^2$31, where $^2$32 is the average L2 norm and $^2$33 the average cosine distance within the head-specific space. Query-time retrieval performs one ANN search per head, retrieves top $^2$34 candidates per head, and assigns each candidate a weight

$^2$35

with duplicate documents fused by summation. On $^2$36-aspect synthetic queries, the paper reports more than $^2$37 average improvement in exact retrieval success ratio $^2$38 and more than $^2$39 average improvement in category retrieval success ratio $^2$40 relative to standard RAG. Across aspect counts $^2$41, MRAG improves the weighted retrieval success ratio $^2$42 by $^2$43-$^2$44 on average, while remaining near-parity with standard RAG on single-aspect queries (Besta et al., 2024).

AttentionRAG moves head-specific information into context compression. It rewrites the query as a next-token prediction task with an answer-hint prefix, yielding a focal token whose attention to context tokens can be read directly from the model’s matrices. In the multi-head extension, token-level pruning scores are aggregated as

$^2$45

followed by top-$^2$46, thresholding, or budget-based selection at the token or sentence level. The method skips chunks whose predicted focal token is “none” and otherwise retains the sentences containing top-scoring tokens. On LongBench and BABILong, the paper reports up to $^2$47 context compression while outperforming LLMLingua methods by around $^2$48 in key metrics. Specific LongBench results include $^2$49WikiMQA at EM $^2$50, LLM Judge $^2$51, and compression ratio $^2$52, and HotpotQA at EM $^2$53, LLM Judge $^2$54, and compression ratio $^2$55 with Llama-3.1-8B-Instruct as generator and the large AttentionRAG variant as compressor (Fang et al., 13 Mar 2025).

AttentionRetriever uses selected layers of pretrained LLM attention maps directly as long-document retrieval signals. Its attention-based sentence score is computed by averaging over heads and taking a maximum over selected layers, query tokens, and sentence tokens. Entity-based scope expansion then ranks entities by the average relevance of the sentences in which they appear, combining an attention view and an embedding view before retrieving the union of paragraphs containing top sentences and top entities. The system is training-free. On single-document retrieval, AttentionRetriever-LLaMA-3.2-3B reports average F1 approximately $^2$56 across six datasets, compared with average $^2$57 for GritLM; on multi-document retrieval it is competitive but not dominant, with average approximately $^2$58 for the LLaMA-3.2-3B variant (Fu et al., 12 Feb 2026).

A graph-attentive retriever extends head-specific scoring to structured episode graphs. Query-Aware Graph Neural Networks for RAG construct per-episode graphs with bidirectional sequential edges and top-5 semantic nearest-neighbor edges retained above cosine threshold $^2$59. The encoder uses $^2$60 query-aware, edge-aware graph-attention heads with residuals, layer normalization, and dropout, followed by query-guided pooling and logistic fusion of FAISS, graph, and GNN scores. Reported Recall@5 improves from $^2$61 to $^2$62 on LPM and from $^2$63 to $^2$64 on TED, with larger relative gains on harder multi-hop queries (Agrawal et al., 25 Jul 2025).

6. Empirical regularities, misconceptions, and limitations

Several regularities recur across these methods. Head count or head selection is rarely monotonic. In PEAR, $^2$65 is best on Llama2-7B-chat-4k, $^2$66 under-corrects suppressive behavior, and $^2$67 damages non-suppressive heads. In soft-prompt MHA-RAG, increasing $^2$68 is useful mainly when multiple exemplars are retrieved, and returns diminish beyond approximately $^2$69-$^2$70. AttentionRAG likewise shows a precision–recall trade-off between chunk size, top-$^2$71, and compression rate; larger chunks and smaller token budgets increase compression but may hurt recall (Tan et al., 2024, Jain et al., 6 Oct 2025, Fang et al., 13 Mar 2025).

A second regularity is that efficiency claims depend on where head structure is inserted. PEAR achieves literal zero inference overhead by folding learned scalars into $^2$72. Soft-prompt MHA-RAG reduces LM-side cost because the generator sees only $^2$73 prompt vectors instead of long textual exemplar strings. R$^2$74AG adds only a small side network and reports approximately $^2$75 latency overhead. By contrast, AttentionRAG performs an additional compression stage, and IHA increases attention cost by a factor $^2$76 for global application unless a hybrid local/global schedule is used (Tan et al., 2024, Jain et al., 6 Oct 2025, Ye et al., 2024, Duvvuri et al., 24 Feb 2026).

The literature also identifies several failure modes. PEAR notes proxy mismatch, static input-agnostic coefficients, and degradation when $^2$77 is too large or $^2$78 becomes too aggressive. Soft-prompt MHA-RAG depends on retrieval quality, may lose fine-grained evidence when long documents are compressed into at most $^2$79 vectors, and does not remove fairness issues induced by biased exemplars. AttentionRAG can over-prune if the hint prefix is ill-formed or if $^2$80 is too small. AttentionRetriever depends on sufficiently informative attention maps, robust entity extraction, and context-extension behavior that may vary across backbones. Query-aware graph retrieval remains sensitive to edge-threshold calibration and domain shift. BEE-RAG adds sensitivity to the entropy target $^2$81 and to the scaling of $^2$82 as context length increases (Tan et al., 2024, Jain et al., 6 Oct 2025, Fang et al., 13 Mar 2025, Fu et al., 12 Feb 2026, Agrawal et al., 25 Jul 2025, Wang et al., 7 Aug 2025).

The forward trajectory suggested by these papers is coherent even though the terminology is heterogeneous. Proposed extensions include dynamic $^2$83 conditioned on queries or retrieved contexts, retrieval-aware head selection that combines causal head scores with retriever confidence, multi-objective proxy tasks for more complex RAG behaviors, dynamic head allocation, per-head temperature or gating, cross-document relation types in graph attention, and tighter end-to-end coupling between retrieval and generation (Tan et al., 2024, Jain et al., 6 Oct 2025, Agrawal et al., 25 Jul 2025). Taken together, these proposals indicate that MHA-RAG is evolving from a narrow idea of “more heads for retrieval” into a broader research program: using the structure of multi-head attention itself as a retrieval interface, a compression oracle, a prompt constructor, and a generator control mechanism.