--- title: 'SA Merge: Selective Attention Fusion' url: https://www.emergentmind.com/topics/selective-attention-merge-sa-merge type: topic --- # SA Merge: Selective Attention Fusion Selective Attention Merge (SA Merge) denotes two algorithmically distinct approaches situated at the intersection of transformer attention efficiency and parameter-adaptive representation fusion. In recent literature, SA Merge refers, first, to a domain-adaptive model merging technique for Speech Foundation Models (SFMs), in which attention-layer “task vectors” from multiple fine-tuned models are fused via exponentially weighted schedules to enhance low-resource ASR [2501.08468]. Second, SA Merge designates a correlation-aware sparse attention framework for length-efficient transformers, in which query/key regions with maximal semantic similarity are selectively attended and then merged for computational tractability and accuracy preservation [2410.04211]. Both approaches target resource-constrained scenarios—either data-limited or hardware-limited—and are characterized by non-uniform attention-parameter fusion. ## 1. Definitions and Mathematical Formalism ### SA Merge for Speech Model Fusion Given a pretrained SFM $\mathcal{M}_0$, a child-speech–adapted version $\mathcal{M}_1$, and an adult-speech–adapted version $\mathcal{M}_2$, attention-layer task vectors are defined for transformer layer $i$ as: \[ \tau_{1,i}^{Q,K,V} = W_{1,i}^{Q,K,V} - W_{0,i}^{Q,K,V};\quad \tau_{2,i}^{Q,K,V} = W_{2,i}^{Q,K,V} - W_{0,i}^{Q,K,V} \] The merged model’s task vector is: \[ \tau_{SA,i}^{Q,K,V} = \lambda_i \tau_{1,i}^{Q,K,V} + (1-\lambda_i) \tau_{2,i}^{Q,K,V} \] where $\lambda_i = \lambda^{\alpha_i}$, with global mixing factor $\lambda$ and decay $\alpha_i$. The resulting attention matrices are: \[ W_{SA,i}^{Q,K,V} = W_{0,i}^{Q,K,V} + \tau_{SA,i}^{Q,K,V} \] ### SA Merge for Sparse Attention Extension Inputs $X \in \mathbb{R}^{B\times H \times N \times d}$ are segmented into query and key regions. Semantic tokens $Q_s'$, $K_s'$ are obtained (e.g., via mean pooling), and region-wise affinity is \[ A_s = Q_s'(K_s')^T \] For each query region, the $k$ top-correlated key regions are selected, indices merged across $m$ adjacent query regions, and a final multi-query attention is computed over the consolidated key/value set. This yields $O(Nk)$ time/memory and tunable compression. ## 2. Algorithms and Implementation Protocols ### Speech SFM Task-Vector Merge Construction of $\mathcal{M}_{SA}$ proceeds as follows: 1. For each transformer layer $i$, extract $W_{0,i}^{Q,K,V}$, $W_{1,i}^{Q,K,V}$, $W_{2,i}^{Q,K,V}$. 2. Compute task vectors $\tau_{1,i}^{\cdot}$ and $\tau_{2,i}^{\cdot}$. 3. Exponentiate mixing ratio: $\lambda_i = \lambda^{\alpha_i}$. 4. Merge $Q/K/V$ deltas and reconstruct $W_{SA,i}^{\cdot}$. 5. All non-attention parameters are sourced from $\mathcal{M}_1$. Model families used include Whisper (all variants), Wav2Vec 2.0-base, HuBERT-base, and WavLM-base. Tooling is provided via HuggingFace Transformers, fairseq, and MergeKit [2501.08468]. ### Correlation-Aware Sparse Attention Pipeline Selection and merge stages are implemented as: 1. Segment $X$ into $n_{sq}$ query and $n_{sk}$ key regions. 2. Pool region tokens to generate $Q_s'$, $K_s'$. 3. Compute dot-product correlations and select top-$k$ key regions per query-region. 4. For every $m$ neighboring query regions, unique-merge their selection indices, keep top-$n$ key/value regions. 5. For each merged block, compute multi-head attention with gathered $K/V$ regions. 6. Positional encoding augmentation is performed post-selection using CRD-NTK (cyclic/randomly truncated/dynamically growing NTK positional embeddings) [2410.04211]. ## 3. Empirical Results and Baselines ### Low-Resource ASR with SA Merge WER reduction for Whisper-small on MyST is recorded as: | Train Subset (h) | Fine-tuned WER | SA Merge WER | Relative Reduction | |------------------|----------------|--------------|-------------------| | 1 | 10.64% | 10.40% | −2.3% | | 5 | 10.05% | 9.85% | −2.0% | | 10 | 9.94% | 9.80% | −1.4% | | full | 9.34% | 8.85% | −5.2% | Data augmentation plus SA Merge sets a new SOTA of 8.69% with SpecAugment [2501.08468]. ### Efficient Long-Context Fine-Tuning For Llama2-7B, SA Merge achieves context extension to up to $1$M tokens with stable perplexity and exact passkey recall ($100\%$ at 4M). GPU resource use is reduced by $\geq64\times$ compared to full attention [2410.04211]. ## 4. Analytical Insights and Ablation Studies ### Layerwise Fusion for Acoustic-Linguistic Feature Adaptation High mixing ratios $\lambda_i$ in lower layers preferentially preserve acoustic/phonetic adaptation, while upper layers employ broader-source linguistic patterns. Distinct from uniform merging, the exponential $\lambda_i$ schedule emulates transformer feature stratification [2501.08468]. Comparative benchmarking against Lerp, Slerp, TA, RegMean, TIES, and DARE+TA demonstrates statistical superiority (Whisper-small, $p<0.05$). ### Sparse Selection Coverage Tradeoff Merging query regions enables shared access to top-K key-value regions, mitigating isolated context starvation and enhancing long-sequence generalization. Segment/merge factors ($s_q,s_k,m,k,n$) allow controllable compute–accuracy balances [2410.04211]. ### Task-Vector Orthogonality Cosine similarity analysis reveals signal-processing–based augmentation vectors (PP, SP, VTLP, SpecAug) are highly aligned ($>0.8$) while synthetic TTS vectors are orthogonal ($0.1$–$0.2$), implying complementary robustness when combined [2501.08468]. ## 5. Practical Applications and Limitations ### Model Fusion for Low-Resource Domains SA Merge demonstrates efficacy for child ASR benchmarks where pretraining data is scarce. By isolating attention-layer adaptation, parameter efficiency is achieved without disruptive confounding of non-attention layers. Extensions to dysarthric/accented speech and multilingual adaptation are logical next steps [2501.08468]. ### Sparse Attention for Commodity Hardware SA Merge enables inference and fine-tuning of 7B+ parameter models with $>32$K tokens on single A100s, outperforming LongLoRA/Longformer in resource usage. Positional encoding augmentation is critical for extrapolation ($1$M+$ tokens) [2410.04211]. ### Limitations and Future Enhancements - Hyperparameter schedules ($\lambda$, $\alpha$, region/merge/sparsity factors) currently require grid or manual search. - Non-attention parameter merging remains unexplored. - CRD-NTK positional augmentation could be further developed by integrating relative positional encodings. - Routing complexity for extreme context lengths still presents bottlenecks. ## 6. Connections to Related Frameworks SA Merge’s speech-domain instantiation is conceptually analogous to techniques such as Task Arithmetic and DARE+TA but is distinguished by its selective, exponentially scheduled fusion specific to attention matrices. The sparse attention variant advances beyond BigBird, Longformer, Routing Transformers, and Biformer by leveraging single-pass, correlation-driven selection rather than fixed local/global windows or clustering. Both frameworks illustrate the trend toward targeted adaptation of transformer attention for domain specificity and computational scalability. ## 7. Summary of Impact and Research Directions Selective Attention Merge constitutes an algorithmic advance in both speech foundation model adaptation and length-efficient transformer attention. It offers up to $14\%$ relative WER reduction over conventional fine-tuning (child ASR, Whisper-small) and, separately, unlocks $1$M+ context-length inference on a single A100 with competitive PPL and passkey recall for LLMs. A plausible implication is that selective domain and sparse attention fusion—when combined with principled positional augmentation—will become standard practice in settings where either data or hardware are severely limited. Key future directions include per-head adaptive schedules, extension of merging to non-attention submodules, and learned selection controllers for sparse attention routing [2501.08468, 2410.04211]. Source: https://www.emergentmind.com/topics/selective-attention-merge-sa-merge