MQMHA: Multi-Query Multi-Head Pooling

• The paper demonstrates that MQMHA enhances representational diversity by computing multiple attention distributions across different feature subspaces and temporal queries.
• MQMHA employs head splitting, attention scoring, and weighted statistics to transform frame-level features into interpretable sequence-level embeddings.
• Empirical results show significant improvements in speech emotion recognition, speaker verification, and anti-spoofing tasks versus traditional pooling methods.

Multi-Query Multi-Head Attentive Statistics Pooling (MQMHA) is a temporal feature aggregation mechanism designed to create sequence-level embeddings from frame-level features. By extending earlier attentive statistics pooling methods, MQMHA learns multiple, parallel attention distributions—both across distinct feature subspaces (“heads”) and via multiple independent temporal queries per head. This enhances representational diversity, interpretability, and empirical performance in applications such as speech emotion recognition, speaker verification, and anti-spoofing.

1. Mathematical Formulation and Forward Computation

MQMHA operates on a sequence of frame-level feature vectors, $X \in \mathbb{R}^{B \times T \times K}$, where $B$ is batch size, $T$ is the (possibly padded) sequence length, and $K$ is the feature dimension. The method relies conceptually and practically on splitting the feature dimension into $H$ equal groups (heads), each of size $K' = K/H$. For each head $h=1,...,H$ and query $q=1,...,Q$, an attention score network $F_{n,p}^{(q,h)}$ computes per-frame scalar “energy” scores.

The core computational steps are as follows (Leygue et al., 18 Jun 2025, Zhao et al., 2021, Park et al., 9 Dec 2025):

1. Head Splitting: $X$ is divided into $B$0 feature groups per frame: $B$1.
2. Attention Scoring: For each group $B$2 and each query $B$3, compute $B$4.
   • $B$5 is a single-layer (n=1) or two-layer MLP (n=2), parametrized per $B$6.
3. Temporal Masking: Apply sequence mask $B$7, setting non-valid frames to $B$8.
4. Attention Weights: For each $B$9, obtain time-normalized attention weights via softmax:

$T$0

1. Weighted Statistics: Compute query- and head-specific statistics:
   • Weighted mean: $T$1
   • Weighted standard deviation: $$\sigma_{b}^{(q,h)} = \sqrt{ \sum_{t} \omega_{b,t}^{(q,h)} (X_{b,t}^{(h)} \odot X_{b,t}^{(h)}) - (\mu_{b}^{(q,h)} \odot \mu_{b}^{(q,h)}) }$$T$2$T$3$

Several instantiations (notably for anti-spoofing (Park et al., 9 Dec 2025)) utilize scaled dot-product attention with learned query vectors and value projections per head.

2. Relationship to Prior Attention Pooling Methods

MQMHA generalizes a variety of earlier pooling mechanisms:

Method	Heads $T$4	Queries $T$5	Attention Layers	Distinction
Attentive Statistics (AS)	1	1	1–2 layer MLP	Global, single attention
Self-Attentive (SA)	1	>1	2-layer MLP	Multiple queries, no splitting
Multi-Head Attentive (MHA)	>1	1	Linear	Head-wise, single query per head
MQMHA	>1	>1	Linear or MLP	Multiple queries & heads

By enabling both $T$6 (temporal diversity) and $T$7 (feature subspace diversity), MQMHA subsumes special cases and allows the model to capture richer, complementary weighting patterns, enhancing the embedding’s expressiveness and robustness (Zhao et al., 2021).

3. Applications and Empirical Performance

Speech Emotion Recognition (SER)

MQMHA has been shown to substantially outperform static and classical pooling strategies for SER. On the MSP-Podcast benchmark (Leygue et al., 18 Jun 2025):

• MQMHA ($T$8, $T$9) achieved a dev macro-F1 of 0.3912, compared to 0.3559 for average pooling and 0.3884 for attentive single-head statistics.
• MQMHA yielded a 3.5-point macro F1 gain over average pooling.
• When analyzed, attentive pooling prioritized non-linguistic vocalizations, hyperarticulated phonemes, and diphthongs, with a model attention pattern resembling human perception.

Speaker Verification

In deep x-vector architectures for speaker verification (ResNet-34 backbone), MQMHA reduced VoxCeleb1-O EER from 1.01% (mean+std pooling) to 0.9465%; combined with an inter-topK penalty, EER reached 0.9305%. Ablation consistently found optimal results for $K$0, $K$1, with further increases not beneficial (Zhao et al., 2021).

Spoof-Aware Speaker Verification

For anti-spoofing in the WildSpoof Challenge, MQMHA was applied to features extracted from HiFi-GAN and BigVGAN discriminators. Aggregating these using MQMHA achieved a-DCF = 0.1363, a ∼4% relative reduction versus no sub-judge and ∼3% versus simple statistics pooling (Park et al., 9 Dec 2025).

4. Hyperparameter Choices and Architectural Considerations

Common hyperparameters include:

• Number of heads $K$2 (e.g., 2, 4, 16)
• Number of queries per head $K$3 (e.g., 2, 4)
• Attention scorer network: linear (n=1) or 2-layer MLP (n=2)
• Hidden dimension $K$4, if using MLP (e.g., 256)
• Value and key dimensions ($K$5, $K$6), especially in dot-product attention configurations

Increasing $K$7 and $K$8 up to a moderate point improves performance; excessive splitting (e.g., $K$9 or $H$0) tends to degrade accuracy (Zhao et al., 2021). In practice, a single linear attention layer per $H$1 is usually sufficient. Implementation in anti-spoofing pipelines follows similar patterns, with per-block projections and dropout for regularization.

5. Interpretability and Attention Analysis

MQMHA provides frame-level, query-specific attention distributions, offering several interpretability and localization advantages (Leygue et al., 18 Jun 2025):

• On SER tasks, approximately 15% of frames account for 80% of cumulative attention mass, revealing strong temporal localization (Pareto-like).
• Correlation between raw audio energy and MQMHA attention is modest ($H$2), showing the model targets emotionally salient (not merely high-energy) regions.
• Phoneme-level Bayesian analysis found that spoken noise, hyperarticulated vowels (e.g., AW1, AY1), and diphthongs were salient. This reflects pooling of non-linguistic and syllabic prominence cues analogous to human perceptual strategies.

A plausible implication is that MQMHA naturally lends itself to applications requiring fine-grained temporal explainability or diagnosis of feature salience.

6. Practical Impact and Computational Considerations

MQMHA introduces parameter overhead proportional to $H$3, but this is typically minor relative to backbone networks. Training remains stable with standard optimizers (SGD, Adam), using dropout on attention weights for regularization. Synchronization is required if multiple input streams (e.g., discriminator layers with varying $H$4) are pooled in parallel (Park et al., 9 Dec 2025). In real-world deployments, MQMHA can be efficiently batched and run in parallel across queries and heads.

The mechanism has enabled state-of-the-art results in competitive benchmarks for both speaker verification and spoof-aware speaker verification, with consistent empirical gains over all major attentive and non-attentive pooling baselines (Zhao et al., 2021, Park et al., 9 Dec 2025).