Conformer Encoder: Global and Local Modeling

• Conformer Encoder is a hybrid model combining multi-head self-attention with convolution modules to capture both long-range dependencies and local acoustic patterns.
• It employs a macaron-style architecture with dual feed-forward layers that enhance residual learning, stability, and multi-task adaptability.
• Extensions such as efficient, memory-augmented, and sparse variants demonstrate its versatility across ASR, audio-visual enhancement, anti-spoofing, and time-series applications.

The Conformer encoder is a convolution-augmented Transformer encoder that combines multi-head self-attention with a dedicated convolution module to model both global and local structure in sequential data. In speech processing, it has become a dominant encoder design because self-attention captures long-range dependencies while convolution captures short-range acoustic regularities such as phonetic transitions, syllabic rhythms, and other local spectro-temporal patterns (Yao et al., 2023, Peng et al., 2023). Although it originated as an automatic speech recognition backbone, subsequent work has adapted the same encoder principle to multilingual ASR, long-form transduction, speech enhancement, audio-visual fusion, anti-spoofing, keyword spotting, audio retrieval, and even irregular time-series embedding.

1. Canonical block structure and mathematical form

A canonical Conformer block uses a macaron-style arrangement in which two feed-forward sublayers surround the attention and convolutional core. One formulation used in multilingual ASR is

$\tilde{x}_1 = x + \frac{1}{2}\,\mathrm{FFN}(x),$

$$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$

$$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$

$$y = \mathrm{LN}\!\left(\tilde{x}_3 + \frac{1}{2}\,\mathrm{FFN}(\mathrm{LN}(\tilde{x}_3))\right).$$

This pattern combines residual connections, layer normalization, and the “half-step” FFN scaling that is characteristic of the macaron design (N, 2021). An equivalent presentation in ESPnet-based speech systems writes the layer sequence as FFN $\rightarrow$ MHSA $\rightarrow$ Conv $\rightarrow$ FFN, again with residual paths and final normalization (Peng et al., 2023).

The attention sublayer is standard scaled dot-product attention, typically extended with multi-head projections:

$$\mathrm{Att}(Q,K,V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$

In Conformer practice, relative positional encoding is common, particularly in ASR, because it improves sequence modeling over long speech utterances (Peng et al., 2023). However, the encoder is not tied to a single positional scheme. Some robust CHiME-4 systems use absolute sinusoidal positional encoding, added as $x + \mathrm{PE}/\sqrt{d_{\mathrm{attn}}}$, while some multi-rate HydraFormer results improved when positional encoding was removed entirely (Yang et al., 2022, Xu et al., 2024).

The convolution module provides the locality bias that distinguishes Conformer from a plain Transformer encoder. A standard implementation uses a pointwise $1\times1$ convolution with expansion factor $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$0, a gated linear unit, a 1-D depthwise convolution, batch normalization, Swish activation, and a final pointwise projection (N, 2021). In ESPnet-style descriptions, this is the local branch that complements MHSA’s global aggregation and is specifically motivated by formants, phonetic transitions, and other short-range speech regularities (Peng et al., 2023).

Not all descendants preserve the exact canonical recipe. DCUC-Net keeps the macaron structure but places a single LayerNorm at the end of the block rather than explicitly stating pre-norm at each intermediate sublayer (Ahmed et al., 2023). AstroCo, designed for irregular stellar light curves, uses a three-sublayer order MHSA $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$1 Convolution $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$2 Gated FFN rather than the original dual-FFN macaron form (Tan et al., 29 Sep 2025). These deviations do not negate the defining principle: a Conformer encoder couples global self-attention with an explicit local convolutional path.

2. Role inside end-to-end ASR systems

In end-to-end ASR, the Conformer encoder usually transforms low-level acoustic features into a sequence of high-level embeddings that feed one or more decoding heads. A representative multilingual system for six Indian languages uses 40-dimensional log-Mel filterbanks with a 25 ms window and 10 ms frame shift, applies speech perturbation and SpecAugment, uses two initial convolutional layers for $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$3 subsampling, and then stacks 12 Conformer blocks with 8 attention heads, $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$4, and FFN hidden size 2048 (N, 2021). The resulting encoder sequence is shared by a grapheme decoder, a phoneme decoder, a language classifier, and a CTC branch.

That multilingual formulation also illustrates how naturally the Conformer encoder fits multi-task training. The system optimizes CTC over the grapheme stream, attention-decoder losses for phoneme and grapheme prediction, and a cross-entropy language-identification loss on an utterance-level pooled encoder representation. The combined objective is

$$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$5

with $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$6, $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$7, $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$8, and $$\tilde{x}_2 = \tilde{x}_1 + \mathrm{MHSA}(\mathrm{LN}(\tilde{x}_1)),$$9 (N, 2021). This encoder-centered multi-tasking is important because it shows that Conformer is not merely a feature extractor feeding one decoder; it can be the shared representational backbone for phonetic, graphemic, and language-level supervision.

Contextualized streaming transducer systems extend the same idea across utterance boundaries. In one Conformer-Transducer design, preceding utterance encoder outputs are cached, then compressed into a compact $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$0 history matrix by an attention-pooling layer,

$$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$1

and these pooled context vectors are appended to the current block’s keys and values (Cui et al., 2023). This preserves cross-utterance information without concatenating all earlier frames.

Long-form ASR pushes the same encoder-decoder interface in another direction. A memory-augmented Conformer inserts a differentiable Neural Turing Machine between a 12-block Conformer encoder and a 6-block Transformer decoder, so that encoder states can be recurrently written to and read from external memory before the decoder attends to them (Carvalho et al., 2023). This architecture leaves the Conformer stack itself intact while addressing the documented degradation of attention-based models on long utterances.

The encoder can also be used in more conventional hybrid acoustic modeling. On CHiME-4, a robust monaural ASR system replaces a recurrent WRBN encoder with a shallow two-block Conformer encoder operating after a WRCNN-like front-end, retaining utterance-wise normalization and iterative speaker adaptation (Yang et al., 2022). This suggests that the Conformer encoder is compatible not only with pure encoder-decoder objectives, but also with senone-posterior acoustic modeling.

3. Extensions beyond canonical ASR

Work outside ASR has largely reused the same Conformer principle—global MHSA plus local convolution—but changed the input domain and the downstream head.

In audio-visual speech enhancement, DCUC-Net places a two-block Conformer stack between a complex U-Net encoder and decoder. Audio is processed in the complex STFT domain with ConvSTFT using window length 400 samples, hop 160, and FFT length 512; visual lip features are extracted with ResNet-18, temporally upsampled, concatenated with the real-valued audio features, and projected into the Conformer model space before fusion (Ahmed et al., 2023). In that setting, replacing Conformer with BLSTM or Transformer fusion yields lower PESQ: 1.62 for BLSTM, 1.79 for Transformer, and 1.84 for Conformer on the COG-MHEAR AVSE Challenge 2023 development set (Ahmed et al., 2023).

In time-frequency speech enhancement, DPCFCS-Net uses Dual-Path Conformer modules not in the front-end encoder proper but in the enhancement layer that follows a convolutional encoder. The model alternates Conformer processing along time and frequency axes, with $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$2 DPCF blocks, 4 attention heads, and 64-dimensional embeddings entering the Conformer stage (Wang, 2023). This preserves the central Conformer insight but reorients the attention axis from purely temporal modeling to alternating sub-band and full-band modeling.

In anti-spoofing, MFA-Conformer uses a 16-block Conformer encoder pretrained on ASR or ASV, then aggregates features from multiple encoder depths through Multi-scale Feature Aggregation before attentive statistics pooling and classification (Wang et al., 2023). The encoder retains 4 heads, convolutional feature dimension 176, FFN feature dimension 704, and $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$3 front-end downsampling. On the FAD clean seen set, the ASR-pretrained MFA-Conformer reaches a best EER of 0.04% (Wang et al., 2023).

Streaming keyword spotting preserves the standard block order but adds trainable binary gates to the residual branches of FFN, MHSA, and Conv modules. In continuous speech on Librispeech top-1000 words, the gated Conformer maintains performance while skipping 30% of MACs on test-clean and 26% on test-other; on Google Speech Commands over noise, it skips 42% on speech regions and 97% on non-speech regions (Bittar et al., 2023). This use is notable because it treats Conformer blocks as selectively executable modules rather than mandatory fixed-depth computation.

Audio retrieval and non-speech time-series modeling show that the encoder has generalized well beyond recognition. For audio fingerprinting, pretrained Conformer encoders with 2, 3, or 4 blocks generate 128-dimensional L2-normalized embeddings from 3-second stabilized log-mel spectrograms; the Large model reaches top-1 retrieval accuracy 98.8 on FMA-large without noise (Altwlkany et al., 15 Aug 2025). For irregular stellar light curves, AstroCo uses a Conformer-style encoder with 12 layers, embedding dimension 256, 4 heads, convolution kernel size 32, and a gated FFN, reducing reconstruction error by about 70 percent relative to Astromer v1 and about 61 percent relative to Astromer v2 on MACHO R-band (Tan et al., 29 Sep 2025).

Taken together, these applications indicate that the Conformer encoder is best understood as a general sequence encoder template rather than an ASR-specific construction.

4. Efficiency, scaling, and structural variants

The principal cost of a canonical Conformer encoder is the quadratic attention term. Efficient Conformer addresses this directly with progressive downsampling and grouped attention, reducing attention complexity from $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$4 to $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$5 for sequence length $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$6, hidden dimension $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$7, and group size $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$8 (Burchi et al., 2021). On LibriSpeech, its 13M-parameter CTC model reaches 3.6%/9.0% WER without a LLM and 2.7%/6.7% with an external n-gram LM, while being 29% faster at inference and 36% faster to train than the paper’s CTC Conformer baseline (Burchi et al., 2021).

Deep Sparse Conformer replaces dense MHSA with ProbSparse self-attention using Transformer-XL-style relative positional encoding and uses DeepNorm to stabilize very deep stacks. The stated sparse attention complexity is $$\tilde{x}_3 = \tilde{x}_2 + \mathrm{ConvModule}(\mathrm{LN}(\tilde{x}_2)),$$9 in time and memory, and the design was trained up to 100 encoder layers (Wu, 2022). On CSJ-500h, a 12-layer variant reports CERs of 5.52%, 4.03%, and 4.50% on the three evaluation sets, while an ensemble of five variants improves these to 4.16%, 2.84%, and 3.20% (Wu, 2022).

Practical Conformer targets deployment constraints more directly. It replaces lower Conformer blocks with conv-only blocks, strategically reduces model size, and substitutes RNNAttention-Performer for standard attention in upper blocks (Botros et al., 2023). The baseline 12-block encoder has 120M parameters, 247–248M FLOPs, and 21.8 ms TPU latency; the optimized version has 56M parameters, 93M FLOPs, and 3.2 ms TPU latency, a 6.8$$y = \mathrm{LN}\!\left(\tilde{x}_3 + \frac{1}{2}\,\mathrm{FFN}(\mathrm{LN}(\tilde{x}_3))\right).$$0 latency reduction, though single-pass WER degrades from 6.5% to 7.7% before recovery in a cascaded second pass (Botros et al., 2023).

Key-frame mechanisms alter the encoder dynamically rather than redesigning attention globally. By attaching an intermediate CTC head after the first encoder stage, the model identifies non-blank frames as key frames and either restricts attention to them (KFSA) or drops blank frames before the second encoder (KFDS) (Fan et al., 2023). The paper reports that more than 60% of frames can be discarded during training and inference, with AISHELL-1 and LibriSpeech results that are comparable to or better than vanilla Conformer and Efficient Conformer.

HydraFormer tackles a different systems problem: the lack of a single universally optimal subsampling rate. HydraSub introduces multiple branch-specific convolutional subsamplers, each feeding the same shared Conformer encoder and BiTransformer decoder, so that inference can select rates such as 4, 6, or 8 without training separate full models (Xu et al., 2024). The multi-rate model has 51.7M parameters, only modestly above the roughly 48–49M single-rate baselines, and its reported real-time factor decreases monotonically as the subsampling rate increases (Xu et al., 2024).

These developments have also motivated successor architectures. Zipformer explicitly presents itself as “a faster and better encoder for automatic speech recognition” than Conformer, citing the fixed frame rate and memory cost of standard Conformer stacks; on a 30-second LibriSpeech input, the reproduced Conformer-L encoder requires 294.2 GFLOPs versus 107.7 GFLOPs for Zipformer-L (Yao et al., 2023). E-Branchformer keeps the macaron logic but replaces the sequential MHSA$$y = \mathrm{LN}\!\left(\tilde{x}_3 + \frac{1}{2}\,\mathrm{FFN}(\mathrm{LN}(\tilde{x}_3))\right).$$1Conv composition with parallel global and local branches, and reports comparable or better performance than Conformer across 15 ASR, 2 ST, and 3 SLU benchmarks, with better training stability (Peng et al., 2023).

5. Empirical behavior, robustness, and failure modes

Across tasks, the empirical case for the Conformer encoder rests on its ability to combine local and global modeling. In the multilingual Indian-language ASR system, the proposed Conformer-GRP+PHN+LID model reaches a weighted-average WER of 26.70, compared with 28.41 for the corresponding Transformer-GRP+PHN+LID model, and the paper attributes the roughly 6% relative WER improvement to better local-global modeling via the convolution module (N, 2021). The same study shows incremental gains from phoneme auxiliary supervision and from an encoder-level language classifier, indicating that Conformer-derived representations are responsive to structured multi-task regularization.

Robustness, however, is domain-dependent rather than automatic. In the same multilingual ASR work, blind-test Marathi performance degrades sharply to 96.75 WER because of mismatched channel conditions, even though the regular test-set Marathi WER is 17.94 (N, 2021). This is evidence against the common simplification that convolutional augmentation alone guarantees channel robustness.

Long-context behavior is another nontrivial issue. Memory-augmented Conformer-NTM models show substantial improvements on very long utterances: on LibriSpeech train-960, no-LM WER on concat-clean drops from 10.0 to 4.3, and with a 6-gram LM from 9.3 to 3.9, when external memory is inserted between encoder and decoder (Carvalho et al., 2023). This suggests that standard Conformer encoders, while strong on ordinary utterances, can benefit materially from explicit long-range state mechanisms.

A more pathological phenomenon appears in global attention-based encoder-decoder systems. “The Conformer Encoder May Reverse the Time Dimension” reports that the encoder may internally flip the time order of frames, which manifests as monotonically decreasing decoder cross-attention weights (Schmitt et al., 2024). In the analyzed model, the flip emerges in a specific Conformer block where mean $$y = \mathrm{LN}\!\left(\tilde{x}_3 + \frac{1}{2}\,\mathrm{FFN}(\mathrm{LN}(\tilde{x}_3))\right).$$2 norms are approximately 60 for self-attention, 23 for the input residual, 13 for convolution, and 15 for the second FFN, so the block output becomes dominated by MHSA content (Schmitt et al., 2024). The paper reports flipping in 12 out of 13 random seeds for global Conformer-LSTM AED setups with BPE-1k and no CTC auxiliary loss, and shows that adding a CTC auxiliary loss prevents flipping in all runs considered there while also improving WER from 2.6/2.8 to 2.4/2.6 on clean and from 6.1/6.0 to 5.4/5.8 on other (Schmitt et al., 2024).

This failure mode matters conceptually. It shows that the Conformer encoder is not simply a monotonic acoustic compressor whose internal order is guaranteed by architecture alone; its interaction with decoder cross-attention and training dynamics can produce non-intuitive representational geometry.

6. Research directions and continuing relevance

Recent work suggests that the most productive research trajectory is not to discard the Conformer encoder outright, but to specialize or constrain it for particular operating regimes. Cross-utterance context pooling compresses history into fixed-size contextual summaries rather than attending over all previous frames (Cui et al., 2023). External memory addresses long-form degradation without redesigning the encoder stack itself (Carvalho et al., 2023). Dynamic skipping, key-frame selection, and multi-rate HydraSub front-ends all make the encoder more input-adaptive at inference time (Bittar et al., 2023, Fan et al., 2023, Xu et al., 2024).

Another direction is transfer. Anti-spoofing work shows that ASR- and ASV-pretrained Conformer encoders can be repurposed effectively in a countermeasure setting, while audio retrieval work shows that self-supervised contrastive pretraining produces Conformer encoders that are robust to temporal misalignment, noise, reverb, and time-stretching (Wang et al., 2023, Altwlkany et al., 15 Aug 2025). A plausible implication is that the encoder’s local-global inductive bias is especially valuable when downstream supervision is sparse or highly task-specific.

At the same time, comparative studies make clear that Conformer is no longer unchallenged as the default sequence encoder. E-Branchformer reports broader training stability and generally stronger benchmark performance, while Zipformer targets better speed-memory-accuracy trade-offs through multi-frame-rate processing and attention-weight reuse (Peng et al., 2023, Yao et al., 2023). These results do not diminish Conformer’s importance; rather, they define the baseline that new speech encoders are expected to match or surpass.

The present literature therefore positions the Conformer encoder as both a mature canonical design and a generative template. Its central insight—the explicit combination of self-attentive global aggregation with convolutional local modeling—has remained intact across multilingual ASR, streaming transduction, enhancement, anti-spoofing, keyword spotting, retrieval, and irregular time-series representation. The active research problem is no longer whether this combination works, but how to control its cost, stabilize its dynamics, adapt its temporal resolution, and couple it to richer forms of memory and supervision.