Grouped Query Experts (GQE) in Deep Learning

Updated 4 July 2026

Grouped Query Experts (GQE) are a family of mixture-of-experts architectures that apply conditional computation over structured groups for efficient specialization.
They use routing strategies like top-k selection and load balancing to activate a limited subset of query heads while keeping the key-value path dense.
Empirical results show that GQE improves efficiency and accuracy across domains such as long-context language modeling, product search, and multi-dialect query generation.

Grouped Query Experts (GQE) denotes a class of mixture-of-experts designs in which routing is organized around groups rather than a single undifferentiated expert pool. In current arXiv usage, the term appears most explicitly in a self-attention formulation that places an MoE layer on top of grouped-query attention (GQA), selecting a small number of query-head experts per token while leaving the key-value path dense and unchanged (Tripathi et al., 18 Jun 2026). A second recent usage appears in dynamic token-wise KV optimization, where tokens are routed to heterogeneous grouped-attention experts with different KV group sizes (Song et al., 16 Jun 2025). Related grouped-expert architectures predate this terminology in product search ranking, where experts are tied to query-category structure (Xiao et al., 2020), and in multi-dialect query generation, where experts are partitioned into dialect groups plus a shared group (Lin et al., 2024). A plausible implication is that GQE is best understood as a family of grouped routing strategies rather than a single canonical layer.

1. Conceptual scope and design space

The unifying idea behind GQE is conditional computation over a grouped structure. The grouping axis differs by application: attention-head groups in long-context Transformers, category hierarchies in product search, or dialect partitions in database query generation. In each case, routing is not fully free-form; it is constrained by an externally defined or architecturally induced grouping.

Variant	Grouping axis	Core routing object
Product-search grouped experts (Xiao et al., 2020)	query sub-category and top-category hierarchy	Noisy Top-K gate over experts
GQE on GQA self-attention (Tripathi et al., 18 Jun 2026)	query heads within GQA groups	per-token top- $k$ query-head selection
mixSGA GQE (Song et al., 16 Jun 2025)	experts with different KV group sizes	token-wise expert-choice routing
MoMQ grouped experts (Lin et al., 2024)	database dialect groups	group-level and expert-level routers

This design space differs from standard dense layers in two respects. First, only a subset of experts is executed for each token or example. Second, the grouping structure usually encodes a prior: shared KV heads in GQA, categorical proximity in e-commerce taxonomies, or dialect-specific syntax in multi-dialect generation. This suggests that GQE combines sparse activation with an explicit inductive bias about how specialization should be organized.

2. GQE as a query-side MoE on grouped-query attention

In "Grouped Query Experts: Mixture-of-Experts on GQA Self-Attention" (Tripathi et al., 18 Jun 2026), GQE is defined as a query-side MoE layer built on top of grouped-query attention. Standard dense multi-head self-attention has per-layer asymptotic cost $O(N^2\cdot d + N\cdot d^2)$ , and for long contexts the $N^2\cdot d$ term dominates. GQA reduces key/value projection and cache cost by sharing key/value projections across groups of query heads, but it does not change the quadratic attention cost $O(N^2\cdot d)$ and does not reduce the query-side cost $O(N\cdot d\cdot H)$ .

GQE preserves the GQA key/value path completely unchanged and injects sparsity only into the query side. Let $H$ query heads be partitioned into $G$ KV groups, with $M = H/G$ candidate query-head experts per group. For each token representation $x_i \in \mathbb{R}^d$ and each group $g$ , a routing network produces a score vector

$O(N^2\cdot d + N\cdot d^2)$ 0

followed by

$O(N^2\cdot d + N\cdot d^2)$ 1

The router selects the top- $O(N^2\cdot d + N\cdot d^2)$ 2 experts per group,

$O(N^2\cdot d + N\cdot d^2)$ 3

and only those experts are executed.

The output construction is not purely hard routing. For each selected expert, the model computes an attention output $O(N^2\cdot d + N\cdot d^2)$ 4. It also computes one always-on shared head $O(N^2\cdot d + N\cdot d^2)$ 5 for stability and one weighted-sum slot,

$O(N^2\cdot d + N\cdot d^2)$ 6

with

$O(N^2\cdot d + N\cdot d^2)$ 7

The hard-selected outputs, the weighted-sum output, and the shared-head output are concatenated into $O(N^2\cdot d + N\cdot d^2)$ 8 slots and projected by $O(N^2\cdot d + N\cdot d^2)$ 9.

The resulting query-side cost becomes

$N^2\cdot d$ 0

while key/value cost remains $N^2\cdot d$ 1 and attention cost remains $N^2\cdot d$ 2. For the configuration $N^2\cdot d$ 3, $N^2\cdot d$ 4, $N^2\cdot d$ 5, $N^2\cdot d$ 6, the relative query-compute factor is

$N^2\cdot d$ 7

corresponding to 44% query compute saved. The paper’s central claim is therefore not that GQE removes the quadratic attention bottleneck, but that it reduces active query-head computation while keeping the KV-cache benefits of GQA.

3. Heterogeneous grouped-attention experts for token-wise KV optimization

In "Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization" (Song et al., 16 Jun 2025), GQE refers to a different construction embedded in the mixSGA architecture. Here the goal is not to sparsify query heads within a fixed GQA layout, but to route each token to one of several grouped-attention experts with different KV group sizes. The method retains all tokens and allocates finer-granularity KV to "important" tokens and coarser KV to "less important" ones.

Let the original model have $N^2\cdot d$ 8 KV heads and embedding dimension $N^2\cdot d$ 9. The model introduces $O(N^2\cdot d)$ 0 experts indexed by $O(N^2\cdot d)$ 1, where expert $O(N^2\cdot d)$ 2 has group size $O(N^2\cdot d)$ 3 and therefore $O(N^2\cdot d)$ 4 KV heads. All experts share the same key/value projection matrices $O(N^2\cdot d)$ 5 and $O(N^2\cdot d)$ 6 from the pretrained model. The construction proceeds by slicing per-head projections,

$O(N^2\cdot d)$ 7

averaging within grouped heads,

$O(N^2\cdot d)$ 8

and then repeating each grouped KV head $O(N^2\cdot d)$ 9 times,

$O(N\cdot d\cdot H)$ 0

Because each token is assigned exclusively to one expert, the per-token KV is

$O(N\cdot d\cdot H)$ 1

Routing is driven by a learned importance scorer,

$O(N\cdot d\cdot H)$ 2

During training-phase prefill, capacity ratios $O(N\cdot d\cdot H)$ 3 with $O(N\cdot d\cdot H)$ 4 determine $O(N\cdot d\cdot H)$ 5 tokens for expert $O(N\cdot d\cdot H)$ 6, assigned greedily by descending scores while skipping already assigned tokens. During decode, routing is per token via $O(N\cdot d\cdot H)$ 7. To align training and inference, the method adds a prefill-decode consistency loss,

$O(N\cdot d\cdot H)$ 8

The resource trade-off is explicit. Expert $O(N\cdot d\cdot H)$ 9 uses $H$ 0 of the full per-token KV size, so the average KV per token under density ratios $H$ 1 is

$H$ 2

With ratios $H$ 3 for experts with group sizes $H$ 4 heads, the budget becomes

$H$ 5

of the original KV. Unlike pruning methods, this construction does not evict tokens; it changes the granularity of their KV representation.

4. Routing constraints, regularization, and expert specialization

A persistent issue in GQE-like systems is expert collapse: routers may overuse a small subset of experts, or experts may converge to similar behaviors. Different papers address this with different auxiliary objectives.

In the product-search model of "Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint" (Xiao et al., 2020), the gate network uses only the embedding of the sub-category ID $H$ 6, with

$H$ 7

followed by top- $H$ 8 masking and softmax. The final output is

$H$ 9

Two regularizers enforce specialization and hierarchy consistency. The adversarial loss penalizes similarity between outputs of top- $G$ 0 experts and uniformly sampled disjoint adversarial experts,

$G$ 1

while the hierarchical soft constraint (HSC) ties the inference gate to a top-category constraint gate:

$G$ 2

The total loss is

$G$ 3

The stated purpose is to force experts to disagree, improve clustering of gate outputs under different categories, and allow smaller sibling categories to share training signal through softly aligned gate distributions.

The GQA-based GQE model uses a standard load-balancing auxiliary loss to spread assignments across experts and reports that training becomes unstable without the differentiable renormalized weighted-sum slot; without that slot, the router receives no stable gradient and performance drops by about 0.7 points (Tripathi et al., 18 Jun 2026). The same paper also reports that removing the always-on shared head causes specialization to collapse.

mixSGA uses a different stabilization principle: one-hot training-inference consistency. Its auxiliary loss aligns the scorer’s $G$ 4 with the training-phase assignment mask, and removing this loss collapses routing and drops ROUGE-L from approximately 21.20 to approximately 7.35 (Song et al., 16 Jun 2025).

MoMQ extends the grouped-expert pattern to multi-dialect generation with a two-stage router: a dialect router

$G$ 5

and a top- $G$ 6 expert router within each dialect group, with $G$ 7 (Lin et al., 2024). Its fine-tuning objective adds a Dialect Router Loss with label smoothing and an Expert Balance Loss,

$G$ 8

to discourage expert collapse. Across these systems, routing quality is not an incidental detail; it is a primary determinant of whether grouped specialization emerges at all.

5. Empirical results across domains

Recent GQE-related systems have been evaluated in long-context language modeling, dynamic KV optimization, product search ranking, and multi-dialect query generation.

System	Setting	Reported result
Adv+HSC-MoE (Xiao et al., 2020)	JD search ranking	AUC 0.8212 vs DNN 0.8131; online +2.18% CVR, +3.97% RPM
GQE on GQA (Tripathi et al., 18 Jun 2026)	250M-scale model, 30B tokens	56.04 average vs GQA 55.86; 1.15× speedup at 2k, ~1.7–1.8× at 32k–1M
mixSGA GQE (Song et al., 16 Jun 2025)	50% KV budget	Wikitext-2 PPL 20.46 vs GQA 22.66; Llama3.2-3B ROUGE-L 25.57 vs GQA 23.26
MoMQ (Lin et al., 2024)	multi-dialect query generation	Qwen2-7B EX 49.15% vs vanilla LoRA 43.48%

The long-context GQE paper reports that, at a fixed 30B token budget and roughly 250M parameters, GQE matches or slightly exceeds the all-active GQA baseline while activating only 9 of 16 total query-attention heads per token (Tripathi et al., 18 Jun 2026). Its ablations also show that weighted concatenation without the renormalized slot yields 55.18 average score, hard concatenation only yields 55.43, and the full design reaches 56.04.

mixSGA reports gains under equal KV budgets across several model families. In supervised fine-tuning at 50% KV budget, Llama3.1-8B improves from GQA approximately 27.40 to mixSGA approximately 28.47 ROUGE-L, and OPT-355m improves from approximately 21.11 to approximately 21.43 (Song et al., 16 Jun 2025). In continued pretraining on TinyLlama-1.1B at 50% KV, Wikitext-2 perplexity improves from 22.66 to 20.46, HellaSwag from 36.70 to 37.00, and Winogrande from 55.90 to 56.30.

The product-search model shows that explicit category-aware specialization can also pay off in industrial ranking. On in-house JD search data with 26.7M train and 2.1M test examples, Adv+HSC-MoE reaches offline AUC 0.8212, compared with 0.8175 for vanilla MoE and 0.8131 for a single DNN; on Amazon Reviews with 199M examples, AUC rises from 0.7031 to 0.7402 (Xiao et al., 2020). The gains are described as especially large on small-data categories such as Clothing.

MoMQ shows the same pattern in a different modality: shared experts and multi-level routing are most valuable under data imbalance. In full-data settings, Qwen2-1.5B improves in EX from 33.36% to 35.66%, Qwen2-7B from 43.48% to 49.15%, and Qwen1.5-14B from 40.29% to 45.78%; in high-resource imbalance scenarios, MoMQ improves by 3–6 points EX, especially on low-resource dialects such as Cypher and nGQL (Lin et al., 2024).

6. Limitations, misconceptions, and open directions

A common misconception is that all GQE methods target the same computational bottleneck. This is not the case. In the GQA-based formulation, GQE reduces only active query-head computation; the attention cost remains $G$ 9 and the key/value path is unchanged (Tripathi et al., 18 Jun 2026). In mixSGA, query, key, and value projections and dot-products are unchanged in total complexity; the main savings arise from lower average KV allocation, while a naïve implementation still runs at approximately 3–4% lower tokens/sec than static GQA at the same KV budget (Song et al., 16 Jun 2025). In product search and multi-dialect generation, the primary objective is not attention efficiency at all, but structured specialization under heterogeneous data distributions (Xiao et al., 2020, Lin et al., 2024).

Another misconception is that grouped routing necessarily implies token discard. mixSGA explicitly distinguishes itself from pure KV pruning methods by never discarding tokens and instead coarsening their representation (Song et al., 16 Jun 2025). Conversely, GQE on GQA preserves all GQA KV-cache benefits but sparsifies only the query side (Tripathi et al., 18 Jun 2026). These are materially different operating points.

The main open problems described in the papers center on router optimization and scaling behavior. The GQA-based paper notes that larger expert pools with fixed $M = H/G$ 0 may need larger token budgets to fully train the router (Tripathi et al., 18 Jun 2026). mixSGA currently fixes capacity ratios uniformly across all layers and identifies learned layer-wise ratios, better batched kernels, and fairness or adversarial robustness in routing as future work (Song et al., 16 Jun 2025). MoMQ reports that too many experts, specifically more than 64, under-train in 3 epochs, indicating that expert count and optimization budget must be matched carefully (Lin et al., 2024). The earlier category-aware model suggests another continuing direction: explicit structural priors, such as hierarchical soft constraints, can be used to turn grouped routing into implicit data pooling for low-resource subgroups (Xiao et al., 2020).

Taken together, these results indicate that GQE is not a single architecture but a general principle: sparse expert activation becomes more effective when routed within a meaningful grouping structure. The exact grouping axis—query heads, KV granularities, product taxonomies, or dialect families—determines what is shared, what is specialized, and which bottleneck is being optimized.