---
title: 'FastConformer: Efficient Speech Encoder Variant'
url: https://www.emergentmind.com/topics/fastconformer
type: topic
---

# FastConformer: Efficient Speech Encoder Variant

FastConformer is a memory- and computation-efficient variant of the Conformer and a drop-in efficient variant of the Conformer encoder for speech applications, introduced by Rekesh et al. as a redesign of Conformer-based sequence modeling for efficient training and inference [2305.05084]. In published systems it serves as the encoder backbone for automatic speech recognition (ASR), speech-to-text translation (AST), streaming ASR, multilingual ASR/AST, and self-supervised speech representation learning, with recurring design choices that include aggressive 8× subsampling, depth-wise separable convolutional front-ends, and alternative attention mechanisms for long-context efficiency [2408.13106][2409.05601].

## 1. Definition and lineage

FastConformer inherits the basic Conformer organization introduced by Gulati et al. in which each layer applies a feed-forward module, self-attention, a convolutional module, a second feed-forward module, and layer normalization with residual connections. The defining changes reported by Rekesh et al. are a fully redesigned 8× downsampling front-end instead of 4×, depth-wise separable convolutions in the subsampling layers, reduced channel count and smaller kernels in the front-end, and an optional post-training replacement of full-context attention with Longformer-style local attention plus a global token [2305.05084].

The architecture was quickly generalized beyond a single implementation. In the 2024 Canary work, FastConformer denotes an encoder for a multilingual attention encoder-decoder system in which the downsampling factor is increased from 4 to 8 and the attention kernel is replaced by a linearized attention mechanism whose memory and compute grow as $O(L\cdot d)$ rather than $O(L^2\cdot d)$ [2406.19674]. In later reports, the same name is also used for variants with standard scaled dot-product multi-head self-attention, hybrid local-global attention, or limited-context streaming masks, indicating that the term identifies an efficiency-oriented Conformer family rather than a single immutable block definition [2305.05084][2409.05601].

A common simplification is to equate FastConformer with only one acceleration trick. The literature instead ties its identity to a cluster of design moves: stronger temporal subsampling, lighter convolutional processing, and context mechanisms that make longer sequences, larger models, or lower-latency inference feasible on practical hardware [2305.05084][2409.05601].

## 2. Encoder block and subsampling design

A representative FastConformer block, as specified in the Romanian ASR report, is
$$
x_0 = \mathrm{LayerNorm}(x),\qquad
y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),
$$
$$
y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad
y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),
$$
$$
y_4 = y_3 + \tfrac12\cdot \mathrm{FFN}(\mathrm{LayerNorm}(y_3)),\qquad
\mathrm{Output} = \mathrm{LayerNorm}(y_4),
$$
with pre-norm, residual connections around FFN, MHSA, and ConvModule, and dropout inside the FFN and convolution module [2511.03361]. This is consistent with the macaron-style arrangement described in earlier FastConformer reports, even when the attention and front-end details differ [2305.05084].

The convolutional module is typically described as a pointwise expansion, gated linear unit, depthwise convolution, normalization and nonlinearity, and a final pointwise projection. In the long-sequence FastConformer XXL description, the convolution path is given as pointwise conv $\rightarrow$ GLU $\rightarrow$ depthwise conv $\rightarrow$ batch-norm $\rightarrow$ swish non-linearity $\rightarrow$ pointwise conv [2409.05601]. Related reports use nearly the same template, with minor differences in activation or normalization placement [2507.13977][2509.14128].

The most stable architectural signature is the aggressive front-end reduction in sequence length. Rekesh et al. describe three depth-wise separable 1D convolutional subsampling layers, each with stride 2, kernel size 9, and 256 output channels, so that a sequence sampled every 10 ms becomes one frame every 80 ms after subsampling [2305.05084]. In the long-sequence speech-recognition and translation study, a 60 s utterance sampled at 16 kHz and featurized with 25 ms windows and 10 ms hop becomes approximately $6000$ raw frames and then approximately $750$ feature-frames after 8× subsampling, which is the basis for fitting full attention at that scale into GPU memory [2409.05601].

Published FastConformer systems span a wide parameter range. Rekesh et al. define model sizes L, XL, and XXL with 120 M, 600 M, and 1.1 B parameters respectively, without changing the core block [2305.05084]. The long-sequence FastConformer-XXL configuration is reported with $d=1024$, $d_{ff}=4096$, $16$ heads, and $48$ blocks for approximately $1.1$ billion parameters [2409.05601]. At smaller scales, the Romanian system uses a 17-layer, approximately 110 M-parameter encoder [2511.03361], while the Arabic models use the NeMo `fastconformer_hybrid_large` configuration with 18 FastConformer blocks, model dimension $512$, FFN hidden size $2048$, 8 attention heads, and convolution kernel size $31$ [2507.13977].

## 3. Attention backends and context regimes

The literature reports several distinct attention regimes under the FastConformer name. In Rekesh et al., the base model uses standard multi-head self-attention during full-context training and can then be converted post-training to limited-context attention with a single global token. In that regime, token $i$ attends only to neighbors in $[i-C,i+C]$, while a special global token attends to all positions and is attended by all positions; the resulting local-only cost is $O(N\cdot d\cdot C)$ when $C\ll N$, rather than $O(N^2\cdot d)$ [2305.05084].

The long-sequence FastConformer XXL work uses a different hybrid formulation. Each Conformer block divides the sequence into local blocks of length $W$ and selects every $G$-th frame as a global summary key, yielding a complexity of $O(L\cdot W\cdot d)+O((L/G)^2\cdot d)$ instead of dense quadratic attention at every layer. For the reported setting with $L_{sub}\approx 750$, $W=64$, and $G=32$, the attention cost is stated to be much smaller than $(750)^2$ while preserving global access through sparse summary positions [2409.05601].

Canary introduces yet another variant. The 2024 data-efficient multilingual model describes FastConformer with a linearized attention mechanism
$$
\hat{A}(Q,K,V)=\phi(Q)\,[\phi(K)^\top V],
$$
with $\phi(x)=\mathrm{elu}(x)+1$, specifically to reduce the quadratic dependence on sequence length in the encoder [2406.19674]. By contrast, the 2025 Canary-1B-v2 report summarizes a FastConformer encoder with standard scaled dot-product multi-head self-attention and states that no explicit sinusoidal or rotary embeddings are added, with positional information carried by convolutional subsampling and learned content interactions [2509.14128]. The coexistence of these descriptions indicates that attention implementation is application-dependent rather than fixed across all FastConformer systems.

Streaming adaptations further specialize the context mechanism. The stateful streaming FastConformer constrains the left and right context in every layer and introduces activation caches for depth-wise convolutions and self-attention keys and values, so that chunk-aware inference matches parallel full-context output exactly while avoiding duplicate compute [2312.17279]. This design is explicitly intended to eliminate the training-versus-inference context mismatch that affects many streaming models [2312.17279].

## 4. Training objectives, decoders, and optimization regimes

FastConformer has been paired with multiple decoder families. The Polish case study states that the output of the encoder can be decoded using RNN-T or CTC loss, with RNN-T by default [2603.02246]. Rekesh et al. report FastConformer with RNN-T for ASR, a Transformer decoder for speech translation, and limited-context post-training for long-form transcription [2305.05084]. Later systems broaden this pattern further through explicitly hybrid decoders.

Two hybrid formulations recur. In streaming ASR, a shared FastConformer encoder branches into both CTC and RNNT heads and is trained with
$$
\mathcal{L}_{\mathrm{total}}=\alpha\,\mathcal{L}_{\mathrm{CTC}}+\mathcal{L}_{\mathrm{RNNT}},\qquad \alpha=0.3,
$$
after which either head may be used at inference time [2312.17279]. In the long-sequence and Romanian systems, the encoder is trained with a Token-Duration Transducer and CTC combination,
$$
L_{\mathrm{final}}=L_{\mathrm{TDT}}+\lambda\cdot L_{\mathrm{CTC}},\qquad \lambda=0.3,
$$
or equivalently $L=\lambda L_{CTC}+(1-\lambda)L_{TDT}$ with $\lambda=0.3$, enabling greedy TDT, ALSD, and CTC beam search with a 6-gram token language model as alternative decoding strategies [2409.05601][2511.03361].

Data preparation is often as important as the encoder itself. The long-sequence speech recognition and translation study replaces lowercased partial segments with sentence-level punctuation-and-capitalization (PnC) segments constructed so that every segment begins with an uppercase letter and ends with one of `.` `?` or `!`. Training data are binned into duration windows $[0,20)$, $[20,40)$, and $[40,60)$ seconds, and models are fine-tuned for 25k steps with 5k warm-up steps, peak learning rate $3\times 10^{-4}$, and A100 80 GB batch sizes of 16, 8, and 2 respectively [2409.05601].

Large multilingual systems extend this with staged pretraining. Canary-1B-v2 uses a two-stage process of 150k and 100k pre-training steps on a total 1.7M hours of data, followed by a 10k-step fine-tuning stage on a high-quality subset with dynamic data balancing [2509.14128]. The earlier Canary report emphasizes dynamic bucketing, stochastic weighted multiplexing across datasets, synthetic data from machine translation, and noise-robust fine-tuning, while the NEST framework uses fixed random projection rather than clustering-based quantization and a generalized noisy speech augmentation that teaches the model to disentangle the main speaker from noise or other speakers [2406.19674][2408.13106].

## 5. Empirical performance across ASR and AST settings

On short-utterance ASR, Rekesh et al. report that Fast Conformer + RNNT reduces LibriSpeech `test-other` WER from 5.19% for Conformer + RNNT to 4.99%, while encoder compute drops from 143.2 GMAC to 48.7 and inference speed increases from 169 to 467 samples/s, corresponding to an approximately 2.8× encoder speed-up [2305.05084]. In the same report, long-form maximum processable length on an A100 at batch 1 increases from 15 min for full-context Conformer to 25 min for full-context FastConformer, and to 675 min, or 11.25 h, for FastConformer with local attention [2305.05084].

For long-sequence ASR with restored punctuation and capitalization, the 2024 FastConformer XXL study reports Greedy WER on Earnings-21 of 30.6 for partial PnC (0–20 s), 24.5 for complete PnC (0–20 s), 23.3 for complete PnC (0–40 s), and 23.1 for complete PnC (0–60 s); on Earnings-22 the corresponding values are 32.8, 25.5, 24.8, and 25.0 [2409.05601]. The same study states that sentence-level PnC gives approximately 20–22% relative WER reduction versus partial segments, that extending to 40 s yields another approximately 5% relative gain, and that past 40 s the gains plateau [2409.05601]. For speech translation on MuST-C de→en, the reported valid/test BLEU values are 22.1/21.5 for 0–20 s, 23.1/22.4 for 0–40 s, and 22.5/21.9 for 0–60 s, again showing a plateau beyond 40 s [2409.05601].

Streaming results show a different performance frontier. On LibriSpeech `test-other`, the cache-aware stateful model achieves 7.1 WER for CTC and 6.3 for RNNT at 1360 ms encoder-induced latency, compared with buffered baselines of 8.0 and 11.3 at 1500 ms and 2000 ms respectively [2312.17279]. Multi-latency training improves robustness across target latencies: on the same benchmark, multi-latency RNNT improves from 6.4/5.9/5.4 to 6.2/5.5/5.2 at 40/240/520 ms, while multi-latency CTC improves from 7.9/7.3/6.2 to 7.6/6.5/6.0 [2312.17279].

Multilingual and language-specific systems show that the architecture remains competitive at very different scales. Canary-1B-v2 reports English ASR with average WER 7.15% at RTFx 749, compared with Whisper-large-v3 at 7.44% and RTFx 145; its average multilingual ASR WER over 24 languages is 8.1%, compared with 9.9% for Whisper-large-v3, and AST X→En reaches 79.3 COMET [2509.14128]. The Romanian FastConformer system, trained on over 2,600 h and decoded with a 6-gram LM, reports WERs of 1.73 on RSC-eval, 8.12 and 10.75 on SSC-eval1/2, 3.92 on CDEP-eval, 3.29 on CV21-RO, 8.85 on Fleurs-RO, and 23.40 on USPDATRO, with up to 27% relative WER reduction compared to previous best-performing systems [2511.03361]. In Arabic, the MSA FastConformer model reports 8.52 on MASC, 7.97 on MCV, and 5.08 on FLEURS, outperforming the prior open and proprietary benchmarks listed in that report [2507.13977]. In Polish read speech, the comparative study reports FastConformer at 5.99% WER on Mozilla Common Voice [2603.02246].

## 6. Applications, ecosystem, and limitations

FastConformer has become a general backbone rather than only an ASR encoder. In NEST, a self-supervised FastConformer with 8× sub-sampling is used for speech recognition and translation, speaker verification, diarization, and spoken language understanding; reported SUPERB results include 94.94% speaker identification accuracy, 3.85% speaker verification EER, 2.28 diarization DER, 1.95 phoneme recognition PER, and 3.49% ASR CTC WER for NEST-L, while NEST-XL reaches 32.42 average BLEU on En→De/Es/Fr AST [2408.13106]. This suggests that the architecture’s efficiency gains are not restricted to token transcription.

The software ecosystem is strongly tied to NVIDIA NeMo. The long-sequence FastConformer XXL work states that model code, training recipes, and 1.1 B-parameter weights are available in NVIDIA NeMo and on HuggingFace, and that training used NVIDIA NeMo toolkit v1.x, PyTorch, and Apex AMP on A100 80 GB GPUs with mixed precision [2409.05601]. NEST, the Arabic models, and later multilingual FastConformer systems likewise report public code, checkpoints, or both through NeMo-linked releases [2408.13106][2507.13977][2509.14128].

Several limitations are also explicit in the literature. The most direct is that longer context is not automatically better: in the punctuated long-sequence study, ASR and AST gains plateau beyond 40 s despite 60 s training being feasible, and the authors explicitly attribute this to a likely need for architectural improvements to exploit more than 40 s of context [2409.05601]. A second limitation is deployment trade-off rather than raw accuracy. The Romanian report notes that the 6-gram LM occupies approximately 2 GB RAM and that TDT+ALSD provides marginal WER gains at a substantial latency cost, motivating future work on smaller LMs and optimized search algorithms [2511.03361]. A third limitation concerns domain transfer: the Polish case study reports FastConformer on Common Voice but does not report FastConformer on medical interviews, while concluding more broadly that purpose-built, domain-tuned systems outperform generic open models in noisy clinical dialogue [2603.02246].

Future directions in the cited work follow from these constraints. The Arabic report recommends self-supervised pre-training on large unlabeled Arabic speech pools, adapter-based fine-tuning for additional dialects, and extensions to relative positional encoding windows and convolution kernel sizes for ultra-long recitations [2507.13977]. The Romanian report identifies streaming and on-device variants as a next step [2511.03361]. Taken together, these directions indicate that FastConformer remains an active architectural substrate for scaling, streaming, multilingual transfer, and domain specialization rather than a closed design point.

Source: https://www.emergentmind.com/topics/fastconformer