---
title: 'MFA-Conformer: Multi-scale Feature Aggregation'
url: https://www.emergentmind.com/topics/multi-scale-feature-aggregation-conformer-mfa-conformer
type: topic
---

# MFA-Conformer: Multi-scale Feature Aggregation

The Multi-scale Feature Aggregation Conformer (MFA-Conformer) is an architectural framework for sequential representation learning that employs Convolution-augmented Transformer (Conformer) blocks with explicit multi-scale feature aggregation. Distinguished by its systematic concatenation of intermediate outputs from all Conformer blocks prior to pooling and downstream classification, the MFA-Conformer delivers empirically validated gains in tasks that require robust extraction of local and global cues, including automatic speaker verification (ASV) and anti-spoofing countermeasures. The model's design facilitates efficient joint modeling of varying temporal context ranges, leading to superior recognition performance and efficiency compared to popular architectures such as ECAPA-TDNN and baseline Conformer systems [2203.15249, 2307.01546, 2410.05037].

## 1. Core Architectural Components

MFA-Conformer is built on the processing pipeline: input features → convolutional subsampling → a stack of Conformer blocks → multi-scale feature concatenation → pooling → linear projection for embedding.

- **Input Preprocessing:** Spoken utterances are transformed into sequences of 80-dimensional log-Mel filterbank (FBANK) features, typically with a 25 ms window and 10 ms hop [2203.15249, 2410.05037, 2307.01546].
- **Convolutional Subsampling Layer:** Two-dimensional (or sometimes one-dimensional) strided convolutions reduce temporal and/or frequency resolution to mitigate computational cost. For instance, a (2,2) strided 2D convolution halves the time and frequency dimensions [2203.15249].
- **Conformer Encoder Stack:** A sequence of $L$ Conformer blocks, each having dimension $d$, jointly model global (self-attention) and local (convolution module) dependencies. MFA-Conformer typical configurations use $L=6$ (ASV) or $L=16$ (countermeasure) blocks [2203.15249, 2307.01546].
- **Multi-scale Feature Aggregation:** Instead of outputting only from the last block, MFA-Conformer concatenates all block outputs (post-layernorm) along the feature/channel dimension to form a dense multi-scale representation. This captures feature hierarchies for downstream pooling and projection [2203.15249, 2307.01546, 2410.05037].
- **Pooling and Projection Head:** The aggregated feature map undergoes attentive statistics pooling (ASP), yielding the weighted mean and standard deviation across time frames, followed by a fully connected layer (with batch normalization) to realize a fixed-dimensional speaker or artifact embedding [2203.15249, 2410.05037, 2307.01546].

## 2. Detailed Module Specifications

### Conformer Block Construction

Each Conformer block consists of:
- Pre-norm Macaron-style feed-forward modules split before and after the main sequence operations, each with half-step residuals.
- Multi-Head Self-Attention (MHSA) with relative positional encodings, using 4 heads and scaled dot-product for $d$, typically 44–256.
- Convolution module (pointwise convolution + GLU, followed by depthwise convolution and batch normalization, then Swish nonlinearities).
- All sub-modules are wrapped with residual connections and layer normalization [2203.15249, 2410.05037, 2307.01546].

### Multi-scale Aggregation Formulation

Let $h_i \in \mathbb{R}^{d \times T'}$ denote the output from the $i$-th block. The multi-scale aggregation step is:

\[
F_{\text{agg}} = \operatorname{Concat}(h_1, h_2, \dots, h_L) \in \mathbb{R}^{(d \cdot L) \times T'}
\]

followed by layer normalization and ASP pooling. Segment-level embedding vectors are produced by concatenating the mean and standard deviation across the time axis:

\[
e = [\tilde \mu; \tilde \sigma] \in \mathbb{R}^{2 \cdot (dL)}
\]
where $\tilde \mu$ and $\tilde \sigma$ are computed using learned attention weights over frames [2203.15249, 2410.05037].

### Embedding Projection and Loss Functions

The action of the pooling head and projection can be summarized as:

- Each pooled vector is batch-normalized and linearly projected to dimensionality (typically $192$ for speaker embeddings) [2203.15249, 2410.05037].
- A variant aggregates blockwise pooled vectors $z_1, \dots, z_L$ into an $L \times 192$ vector for additional projection [2410.05037].
- Losses include Additive Margin Softmax (AM-Softmax) [2203.15249, 2410.05037], with options for blockwise supervision using contrastive loss variants (see Section 5).

## 3. Empirical Performance and Model Comparison

Across ASV and anti-spoofing tasks, MFA-Conformer achieves state-of-the-art or near state-of-the-art performance with competitive parameter counts and superior inference speed.

| Model/System                  | #Params   | EER (VoxCeleb1-O, %) | EER (FAD-Clean SEEN/UNSEEN, %)    | Inference RT Factor |
|------------------------------|-----------|----------------------|------------------------------------|--------------------|
| ResNet34                     | 23.2M     | 1.03                 | —                                  | 0.0088             |
| ECAPA-TDNN                   | 20.8M     | 0.82                 | —                                  | 0.0180             |
| MFA-Conformer (1/2)          | 20.5M     | **0.64**             | —                                  | 0.0121             |
| MFA-Conformer (ASR-pre)      | 13M       | —                    | **0.05 / 27.32**                   | —                  |
| MFA-Conformer (ASV-pre)      | 13M       | —                    | 0.14 / 25.62                       | —                  |
| MFA-Conformer + MFCon loss   | —         | 2.41†                | —                                  | —                  |

†Best result with combined contrastive supervision and AM-Softmax [2410.05037].

On ASV benchmarks, MFA-Conformer with 1/2 subsampling exhibits a 21% relative equal error rate (EER) reduction versus ECAPA-TDNN, while being 32% faster in inference [2203.15249]. For anti-spoofing, MFA-Conformer with ASR pretraining attains 0.05% EER (SEEN) and 27.32% EER (UNSEEN) on FAD-clean, outperforming baseline SSL and CNN models [2307.01546].

Ablation studies consistently show that removing multi-scale aggregation or the convolution module causes the greatest EER increase, confirming their essential contributions [2203.15249, 2307.01546].

## 4. Application Domains and Transfer Strategies

While originally proposed for speaker verification, the MFA-Conformer has been adapted for anti-spoofing countermeasures and demonstrates robust cross-lingual generalization [2307.01546]. In this context:
- Pretraining the Conformer encoder with CTC-based ASR or AAM-Softmax-based ASV improves downstream anti-spoofing robustness.
- Fine-tuning proceeds via a two-stage strategy: freezing the encoder to train the classifier head, then joint training, stabilizing convergence for small target datasets.
- The model achieves competitive error rates across multiple corpora, including FAD (Chinese) and ASVspoof-2019/2021 (English) [2307.01546].

## 5. Recent Extensions: Contrastive Multi-Scale Supervision

Building on the foundation established by multi-scale aggregation, subsequent work has introduced multi-scale feature contrastive (MFCon) loss functions to further enhance intermediate feature discriminability [2410.05037]. This involves:

- Applying supervised contrastive loss to blockwise embeddings $z_i$ post-pooling.
- The total loss becomes a combination of AM-Softmax applied to the final embedding and averaged supervised contrastive losses across all blocks.
- Ablations show that separate pooling and projection for each block achieves the best EER; sharing parameters degrades performance.
- The combined use of MFCon and AM-Softmax on the concatenated speaker embedding yields an additional 9.05% relative EER improvement compared to MFA-Conformer baseline on VoxCeleb1-O.

## 6. Analytic and Ablation Results

Key ablation outcomes on VoxCeleb1-O or FAD-clean:
- Eliding multi-scale aggregation nearly doubles EER (+94%).
- Removing the convolution module results in >100% EER increase.
- Substituting Conformer blocks with Transformer blocks (removing convolution) degrades EER from 1.29% to 2.50% on SITW.Dev.
- For anti-spoofing, concatenating all block outputs (16 in total) yields 20–30% lower pooled EER than using only the final block, indicating diverse depthwise representations capture complementary cues [2307.01546].

## 7. Future Research Directions

Anticipated advancements include:
- Streaming-capable and causal MFA-Conformer variants for low-latency inference.
- Integration of domain-adversarial adaptation to bolster robustness across deployment conditions.
- Exploration of quantized and lightweight models for edge/on-device deployment in ASV and anti-spoofing.
- Further research on fusion with raw waveform models, leveraging attack-wise error tendency (ET) statistics to guide model combination for forensic or adversarial contexts [2203.15249, 2307.01546].

The MFA-Conformer paradigm establishes a principled multi-scale sequence modeling approach for audio, validated across speaker verification and anti-spoofing tasks, and motivates continued investigation into depthwise aggregation and intermediate supervision for sequential embedding extraction [2203.15249, 2307.01546, 2410.05037].

Source: https://www.emergentmind.com/topics/multi-scale-feature-aggregation-conformer-mfa-conformer