QK Adapter: Efficient Transformer Compression

• QK Adapter is a modular neural component that compresses per-token query and key representations into efficient chunk-level features, enabling scalable transformer inference.
• It employs lightweight two-layer FFNs—the Q-Adapter and K-Adapter—integrated in parallel with transformer layers to maintain near-lossless performance.
• Empirical results show significant reductions in computational load and memory usage, while achieving over 98% accuracy retention in both long-context and short-text tasks.

A QK Adapter is a modular neural component designed to compress and aggregate per-token query (Q) and key (K) representations in transformer architectures into more efficient, chunk-level features, dramatically reducing attention computation and key-value cache overhead. The QK Adapter encompasses both a Q-Adapter and a K-Adapter, commonly implemented as lightweight two-layer feed-forward networks (FFNs), and is typically integrated into standard transformer layers in a parallel, plug-and-play fashion. Originating from system-level requirements for scalable LLM inference and efficient adaptation to specialized input representations, QK Adapters enable near-lossless retention of model performance while introducing significant computational and memory savings through chunking and selective attention techniques (Ouyang et al., 28 Sep 2025, Akylzhanov, 29 Mar 2026).

1. Motivation and Problem Statement

Transformer self-attention’s computational and storage complexity scales quadratically with the input sequence length $n$, as the attention matrix $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$ is $n \times n$. This O($n^2$) scaling is prohibitive for long-context applications, especially when, semantically, only a small subset of groups of tokens (chunks) are actually relevant (“key”) to inference over an entire context.

Standard approaches cache and attend to every token, resulting in inefficiencies in both compute and memory, particularly in long-text or agglutinative linguistic settings where tokenization yields high sequence lengths, as in Kazakh (Ouyang et al., 28 Sep 2025, Akylzhanov, 29 Mar 2026). The Q-Adapter and K-Adapter—collectively referred to as the QK Adapter—were proposed to overcome these issues by compressing Q and K representations to chunk-level abstractions and learning sparse, chunk-selective attention, providing an O($n c$) attention scoring path (with $c \ll n$) and drastically reducing the size of the key-value (KV) cache required at inference (Ouyang et al., 28 Sep 2025).

2. Architectural Design and Placement

The QK Adapter is instantiated in each transformer layer $\ell$, operating in parallel to the usual Q and K projections. The core design features are:

• Q-Adapter (FFN$_Q^\ell$): Maps per-token queries $Q^\ell \in \mathbb{R}^{n \times d}$ to compressed chunk-attentive representations $\bar{Q}^\ell \in \mathbb{R}^{n \times d_k}$.
• K-Adapter (FFN$A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$0): Maps selected per-chunk keys $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$1 (gathered at chunk boundaries detected by a Chunk Adapter) to $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$2.

Both FFNs are two-layer bottleneck networks with $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$3, and employ standard nonlinearities (ReLU, GELU).

The dataflow per layer $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$4 is as follows:

1. Compute standard Q and K projections: $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$5, $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$6.
2. Identify semantic chunk boundaries via the Chunk Adapter and index boundary tokens.
3. Extract $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$7.
4. Compress: $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$8, $A = \operatorname{Softmax}(Q K^\top / \sqrt{d_k})$9.
5. Compute student chunk-level attention scores via dot-product and softmax: $n \times n$0 (Ouyang et al., 28 Sep 2025).

3. Mathematical Formulation

The compression operations are defined as: $n \times n$1

$n \times n$2

where $n \times n$3 is a nonlinear activation and $n \times n$4, $n \times n$5 are learned parameters shaping from hidden to bottleneck sizes.

The student chunk attention is

$n \times n$6

To guide training, the (frozen) teacher attention is computed as standard token-level attention, then aggregated to chunk-level by

$n \times n$7

with $n \times n$8 enumerating tokens in chunk $n \times n$9 (Ouyang et al., 28 Sep 2025).

4. Training via Attention Distillation

QK Adapters are trained via a layer-wise Kullback–Leibler divergence objective aligning the student chunk attentions $n^2$0 to teacher chunk-attentions $n^2$1: $n^2$2 where $n^2$3 is the number of transformer layers and KL divergence is evaluated per-token, per-chunk (Ouyang et al., 28 Sep 2025). Only the QK Adapters (and the Chunk Adapter) are updated, with all backbone weights frozen.

This formulation ensures that the student attends to the same key chunks as indicated by the full-attention teacher, preserving performance while reducing the number of active attended positions and memory retained.

5. Chunk Detection, Inference Scheduling, and Integration

A separate Chunk Adapter, attached to the bottom transformer layer, segments the input sequence. For each token, it predicts chunk boundaries with a threshold $n^2$4 via a lightweight FFN and sigmoid activation; training minimizes binary cross-entropy.

Inference alternates standard forward computation with selective top-$n^2$5 chunk selection:

• At chunk boundaries ($n^2$6), for each layer, select top-$n^2$7 chunk indices according to $n^2$8, then aggregate voted top-$n^2$9 globally.
• Only the key/value states corresponding to these chunks are included in the active KV-cache.
• Intra-Chunk Attention Consistency (ICAC) leverages the empirical observation that tokens within a chunk attend to the same set of top-$n c$0 chunks, so cache updates are only triggered at detected boundaries, minimizing frequent recomputation.

6. Practical Hyperparameters and Implementation

The QK Adapter’s compression bottleneck is narrow (e.g., $n c$1 vs $n c$2). Local chunking windows, $n c$3 (number of selected chunks), and chunk boundary thresholds are tuned for specific workloads; for example, with 4 K input, 15 local chunks are typical, and 45% of $n c$4 selected for top-$n c$5 attention. Training employs Adam with $n c$6, $n c$7, and a cosine-annealing learning rate schedule, usually on frozen backbones over billions of tokens (Ouyang et al., 28 Sep 2025).

7. Empirical Results and Impact

QK Adapters enable substantial efficiency gains with negligible performance loss:

• On long-text benchmarks (e.g., LongBench), accuracy retention is 98.64% (e.g., 43.53 vs. 44.13), while KV-cache utilization is reduced to 48.58%.
• On Needle-in-a-Haystack (64 K context), QK Adapter-augmented ChunkLLM uses less than 55% of the cache and surpasses SepLLM on positions beyond 12 K.
• On book-length (PG19, 120 K tokens), up to 4.48$n c$8 inference speedup is achieved, with a minor perplexity increase (14.41 $n c$9 16.23).
• On standard short-text tasks (MMLU, SciQ), 99.5–99.8% accuracy is retained with only 45% cache usage.
• Middle transformer layers recover over 80% of ground-truth key chunks with $c \ll n$0 (for a 4 K context) (Ouyang et al., 28 Sep 2025).

This suggests that QK Adapter-based compression provides a trade-off that is near-optimal for both long and short-context inference in practice.

A structurally analogous compression philosophy appears in byte-level adaptation for language-specific tasks. For example, in ByteKaz (Akylzhanov, 29 Mar 2026), a patch-based, byte-level adapter interfaces with a frozen Qwen2.5-7B to overcome BPE inefficiencies in agglutinative languages, using dense patch embeddings and cross-attention to pool sub-token bytes and ultimately project into (and out of) the main transformer body, with the adapter’s parameter count kept to $c \ll n$14.3% of the total model size.

8. Broader Context and Significance

QK Adapters exemplify a paradigm where large foundation models are made more resource-efficient and domain-flexible by introducing targeted, trainable bottlenecks for specific operations (e.g., attention, tokenization adaptation) without sacrificing the generalization and scalability of the frozen backbone. This approach enables chunk-selective, hierarchical attention systems, which reduce computational overhead and memory footprint, and is of increasing relevance for memory-constrained, long-context, or linguistically diverse application domains.

The concept’s adaptability, verified for both self-attention acceleration (Ouyang et al., 28 Sep 2025) and byte-level language adaptation (Akylzhanov, 29 Mar 2026), indicates further opportunities for modular adapters in both inference optimization and cross-lingual generalization in LLMs.