CA-MHFA: Context-Aware Multi-Head Pooling

• The paper introduces CA-MHFA, a pooling backend that factorizes SSL layer representations into separate key and value streams to better capture content and speaker cues.
• CA-MHFA uses multi-head, context-aware attention by employing local temporal windows and grouped learnable queries to effectively aggregate frame-level features.
• Empirical results demonstrate improved performance in speaker verification, emotion recognition, and spoof detection with minimal additional parameters.

Searching arXiv for the primary CA-MHFA paper and closely related pooling work to ground the article in current arXiv records. Context-Aware Multi-Head Factorized Attentive Pooling (CA-MHFA) is a lightweight, attention-based pooling backend designed to adapt large self-supervised speech models such as WavLM, HuBERT, and wav2vec 2.0 to speaker verification and related speech classification tasks. It replaces heavy TDNN/ResNet style extractors with a compact module that factorizes self-supervised layer representations into “key” and “value” streams and uses grouped, learnable queries with local temporal context to perform multi-head attentive pooling over time. In the formulation introduced for SSL-based speaker verification, CA-MHFA is explicitly described as a context-aware, multi-head, factorized attentive pooling mechanism, and the reported backend size is approximately $2.3$M parameters (Peng et al., 2024).

1. Definition, scope, and motivation

CA-MHFA was introduced for SSL-based speaker verification in order to address several limitations attributed to existing SSL back-ends. The stated issues are that many SSL-based speaker verification systems compute pooling weights using each frame independently in the time dimension, add redundant frame-level back-ends on top of already strong SSL encoders, rely on a single set of layer weights when combining SSL layers, and are less explored outside speaker verification tasks such as emotion recognition and anti-spoofing (Peng et al., 2024).

The central motivation is twofold. First, context-awareness is introduced at the pooling stage because, even if SSL encoders already model long-range dependencies, pooling still determines which frames matter for the final utterance embedding. Second, factorization is introduced because different SSL layers capture different mixtures of phonetic/content and speaker cues, so a single layer weighting scheme cannot disentangle “where to attend” from “what to aggregate” (Peng et al., 2024).

Within this design, the key stream is described as more content-oriented and is used only to compute attention weights, whereas the value stream is described as more speaker-oriented and is aggregated into the utterance representation. This separation is the defining “factorized” component of CA-MHFA. The “context-aware” component comes from computing attention with a local temporal window around each frame rather than from isolated frame-wise scores. The “multi-head” component comes from grouping learnable global queries into multiple heads that share keys and values but differ in query kernels (Peng et al., 2024).

A plausible implication is that CA-MHFA should be understood less as a generic attention layer than as a specialized pooling backend: it operates after the SSL encoder, consumes variable-length frame sequences, and outputs a fixed-dimensional embedding suitable for speaker verification, emotion recognition, or spoof detection.

2. Architecture and mathematical formulation

The full SSL-based pipeline begins with a pre-trained SSL model that produces layer-wise hidden sequences

$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$

Here $T$ is the number of frames, $F$ is the hidden dimensionality, and $N$ is the number of SSL layers. CA-MHFA then performs a frame-level extraction step with compression and factorization: $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$ The resulting keys and values satisfy $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$, where the two normalized layer-weight sets $\{\omega_n^k\}$ and $\{\omega_n^v\}$ and the two projection matrices $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$ are distinct (Peng et al., 2024).

CA-MHFA uses a global, input-agnostic query matrix

$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$0

where $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$1 is the number of groups or heads and each group $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$2 contains $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$3 query vectors

$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$4

Keys and values are shared across all groups; only the grouped queries differ. This shared-$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$5, shared-$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$6 design is presented as a source of parameter efficiency and regularization, because multi-headedness is realized solely via $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$7 rather than via separate per-head projections for all three streams (Peng et al., 2024).

The context-aware attention score for frame $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$8 in group $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$9 is

$T$0

with context radius $T$1. The numerator averages dot products between the query kernel and a local key patch, and the denominator normalizes over all time indices. Once the attention weights are obtained, each head pools the value sequence as

$T$2

and the heads are concatenated: $T$3 The pooled representation is then passed through a linear layer and $T$4 normalization to produce the final embedding, and speaker verification training uses AAM-softmax with scale $T$5 and margin $T$6, followed later by $T$7 in large-margin stages (Peng et al., 2024).

3. Context-awareness, factorization, and relation to simpler pooling

The distinguishing operation in CA-MHFA is that attention for frame $T$8 depends on a temporal window of length $T$9 centered at $F$0, rather than on $F$1 alone. In the paper’s interpretation, this is equivalent to applying a convolution-like filter of shape $F$2 over the key sequence, with each head corresponding to a different learned temporal kernel (Peng et al., 2024).

This mechanism differs from mean pooling, self-attentive pooling, attentive statistics pooling, and non-contextual multi-head attentive pooling in a specific way. Mean pooling assigns uniform weights $F$3 to all frames. Self-attentive pooling uses a learned attention vector but does not incorporate explicit local windows in attention computation. Previous MHFA back-ends use separate layer-weighted projections for keys and values but treat frames independently when computing attention. CA-MHFA is exactly MHFA when $F$4, and it reduces to a single-head attentive design when $F$5 and $F$6 (Peng et al., 2024).

The factorized component appears at two levels. First, the SSL layer aggregation is factorized into separate key and value mixtures, allowing the model to use one representation for attention computation and another for aggregation. Second, the multi-head design shares keys and values across heads while varying only the grouped queries. The paper explicitly argues that applying temporal convolution to the value branch as well degrades performance, which is interpreted as evidence that contextualization is especially beneficial in the content/key subspace while the speaker-oriented value subspace benefits from stability (Peng et al., 2024).

Related arXiv work provides a broader conceptual backdrop for this design. “Enhancing Sentence Embedding with Generalized Pooling” formulates vector-based multi-head attention pooling over contextual token representations and shows that max pooling, mean pooling, and scalar self-attention are special cases; this suggests a broader family in which CA-MHFA can be placed as a context-conditioned, multi-head pooling operator rather than as an isolated architecture (Chen et al., 2018). In speaker verification, “Double Multi-Head Attention for Speaker Verification” adds a second self-attention layer over head summaries, showing that explicit head re-weighting can improve discriminability relative to simple concatenation (India et al., 2020). A separate line of work on low-rank multi-head attention uses a global context vector and factorized bilinear scoring to construct context-aware multi-head pooling with linear scaling in the number of heads, offering an alternative factorization strategy for sequence pooling (Mehta et al., 2019).

4. Empirical performance on speaker verification and other speech tasks

The principal speaker verification results are reported on VoxCeleb. For WavLM_Large + CA-MHFA, the paper reports EERs of $F$7, $F$8, and $F$9 on Vox1-O, Vox1-E, and Vox1-H, respectively; with LMF/QMF calibration, the corresponding EERs are $N$0, $N$1, and $N$2 (Peng et al., 2024). For WavLM_Base_Plus + CA-MHFA, the reported EERs are $N$3, $N$4, and $N$5, and with additional LMF/QMF calibration they become $N$6, $N$7, and $N$8 (Peng et al., 2024).

The paper also compares CA-MHFA with previous SSL back-ends and supervised systems. WavLM_Large + MHFA is reported at $N$9, $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$0, and $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$1, while WavLM_Large + ECAPA-TDNN is reported at $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$2, $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$3, and $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$4 in the paper’s original numbers and $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$5, $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$6, and $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$7 in a WeSpeaker implementation. Conventional supervised baselines include ECAPA-TDNN with $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$8M parameters and $\begin{split} \mathbf{K} &= \left(\sum_{n=0}^{N} \omega_{n}^k \mathbf{z}_{n}\right)\mathbf{S}^k,\ \mathbf{V} &= \left(\sum_{n=0}^{N} \omega_{n}^v \mathbf{z}_{n}\right)\mathbf{S}^v. \end{split} \tag{1}$9G FLOPs, ResNet221 with $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$0M parameters and $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$1G FLOPs, and ResNet293 with $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$2M parameters and $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$3G FLOPs (Peng et al., 2024).

Two efficiency claims are emphasized. First, MHFA with $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$4 and no context ($\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$5) has approximately $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$6M parameters, while CA-MHFA with $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$7 and $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$8 has approximately $\mathbf{K}, \mathbf{V} \in \mathbb{R}^{T \times D}$9M parameters, so the added contextual queries contribute only about $\{\omega_n^k\}$0M parameters. Second, WavLM Large + CA-MHFA is reported at $\{\omega_n^k\}$1G FLOPs for a 2-second input, which is the same figure reported for WavLM Large + MHFA (Peng et al., 2024).

Hyper-parameter analysis on WavLM Base Plus shows that increasing the number of heads and using moderate context windows improves speaker verification performance, especially on Vox1-E and Vox1-H. The reported sequence includes MHFA with $\{\omega_n^k\}$2: $\{\omega_n^k\}$3, MHFA with $\{\omega_n^k\}$4: $\{\omega_n^k\}$5, CA-MHFA with $\{\omega_n^k\}$6: $\{\omega_n^k\}$7, CA-MHFA with $\{\omega_n^k\}$8: $\{\omega_n^k\}$9, CA-MHFA with $\{\omega_n^v\}$0: $\{\omega_n^v\}$1, CA-MHFA with $\{\omega_n^v\}$2: $\{\omega_n^v\}$3, CA-MHFA with $\{\omega_n^v\}$4: $\{\omega_n^v\}$5, and CA-MHFA with $\{\omega_n^v\}$6: $\{\omega_n^v\}$7 (Peng et al., 2024).

The same backend is evaluated under frozen-SSL conditions across speaker verification, emotion recognition, and anti-spoofing. On WavLM Large for speaker verification, the reported Vox1-O EERs are $\{\omega_n^v\}$8 for x-vector, $\{\omega_n^v\}$9 for ECAPA-TDNN, $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$0 for MHFA, and $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$1 for CA-MHFA. On emotion recognition with WavLM Large, mean pooling yields $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$2, MHFA yields $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$3, and CA-MHFA yields $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$4. On ASVspoof 2019 LA with WavLM Large, the reported EERs are $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$5 for LLGF, $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$6 for MHFA, and $$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$7 for CA-MHFA (Peng et al., 2024).

Setup	Result	Source
WavLM_Large + CA-MHFA + LMF/QMF	$$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$8 EER	Vox1-O/E/H
WavLM_Large + CA-MHFA	$$\mathbf{S}^k, \mathbf{S}^v \in \mathbb{R}^{F \times D}$$9 EER	Vox1-O/E/H
WavLM_Base_Plus + CA-MHFA	$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$00 EER	Vox1-O/E/H
WavLM_Large + MHFA	$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$01 EER	Vox1-O/E/H

These results are presented in the paper as evidence that CA-MHFA generalizes across multiple SSL models and multiple downstream tasks while retaining a small backend size (Peng et al., 2024).

5. Historical and methodological antecedents

CA-MHFA belongs to a longer trajectory of attentive pooling research in which contextual encoders, multi-head pooling, and factorization are progressively combined. In NLP sentence embedding, vector-based generalized pooling introduced a contextual encoder $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$02, per-head vector-valued attention matrices $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$03, and explicit diversity penalties on parameters, attention matrices, or sentence embeddings. That formulation already framed multi-head pooling as a mechanism in which each head captures “different aspects of the sentence,” and it explicitly treated max pooling, mean pooling, and scalar self-attention as special cases (Chen et al., 2018).

In speaker verification, Double Multi-Head Attention pooling extended a previous self multi-head attention mechanism by adding a second self-attention layer over the per-head context vectors. The reported gains were $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$04 and $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$05 relative improvement in terms of EER compared to Self Attention pooling and Self Multi-Head Attention, respectively, on VoxCeleb2. This is relevant because it established that head-level selection can be as important as frame-level weighting in speaker embedding extraction (India et al., 2020).

Low-rank factorization for compact multi-head self-attention provided another antecedent by defining multi-head attention through a global context vector and a factorized bilinear form. That work interprets the resulting attention matrix $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$06 as a multi-head pooling operator $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$07, while emphasizing linear rather than quadratic scaling in sequence length. This suggests that CA-MHFA’s shared-key/shared-value grouped-query design is part of a broader family of compact, factorized attention-pooling mechanisms (Mehta et al., 2019).

Beyond speech and text, analogous ideas appear in vision and multimodal fusion. A non-local self-attentive pooling method for CNN feature maps uses patch embedding, multi-head self-attention, spatial-channel restoration, sigmoid activation, and exponential soft-max to construct context-aware pooling weights during down-sampling, showing that context-aware multi-head pooling is not specific to sequence data (Chen et al., 2022). In VQA, generalized multimodal factorized high-order pooling uses low-rank factorization and cascaded multiplicative blocks to capture high-order multimodal interactions, illustrating a different but related meaning of “factorized pooling” centered on expressive low-rank fusion (Yu et al., 2017).

This broader literature suggests that CA-MHFA occupies the intersection of three previously separate design trends: contextual weighting, multi-head decomposition, and compact factorized parameterization.

6. Limitations, ablations, and design implications

The paper’s ablations identify several practical constraints. Moderate context lengths improve performance, but gains saturate: $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$08 or $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$09 is better than $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$10, while $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$11 offers no further improvement and can slightly degrade some results. Increasing the number of heads from $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$12 to $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$13 to $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$14 improves performance, especially on Vox1-E and Vox1-H, but increases backend parameters from $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$15M to $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$16M. Applying temporal convolution to the value branch as well, denoted CA-MHFA$$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$17, yields $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$18, which is worse than using context only in keys (Peng et al., 2024).

The cross-task experiments also show that context-aware pooling is not uniformly superior in every configuration. On ASVspoof 2019 LA, the paper notes a minor regression for HuBERT Base where MHFA gives $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$19 EER and CA-MHFA gives $$\mathbf{Z} = \{\mathbf{z}_0,\dots,\mathbf{z}_N\}, \quad \mathbf{z}_n \in \mathbb{R}^{T \times F}.$$20. This indicates that the gains depend on the upstream model and task, even though the broader trend is favorable (Peng et al., 2024).

The architecture also relies on a strong SSL backbone; the paper does not report absolute performance for low-resource or non-SSL setups. A plausible implication is that CA-MHFA should be viewed primarily as a backend specialization for rich frame-level encoders rather than as a substitute for upstream representation learning.

Several future directions are suggested by the reported findings. The paper points toward adaptive or input-dependent context lengths, more sophisticated sharing patterns or regularization across query groups, extension to multi-channel or multi-modal inputs, and combination with parameter-efficient tuning methods such as adapters, LoRA, and prompts (Peng et al., 2024). More generally, earlier multi-head pooling work suggests that explicit diversity constraints or head-level re-weighting may remain useful when grouped queries become numerous or when head specialization begins to collapse (Chen et al., 2018, India et al., 2020).

Taken together, these observations position CA-MHFA as a compact backend that introduces local temporal awareness into SSL pooling without abandoning the efficiency constraints that motivated earlier factorized attentive pooling designs.