FastConformer: Efficient Speech Encoder Variant

• FastConformer is a memory- and computation-efficient variant of the Conformer encoder that uses aggressive 8× subsampling, reduced channels, and depth-wise separable convolutions for rapid inference.
• It employs redesigned convolutional front-ends and hybrid or linearized attention mechanisms to reduce computational cost and enable processing of longer sequences.
• FastConformer serves as the backbone for diverse speech applications including ASR, AST, streaming recognition, and multilingual models, delivering significant speed-ups and improved scalability.

FastConformer is a memory- and computation-efficient variant of the Conformer and a drop-in efficient variant of the Conformer encoder for speech applications, introduced by Rekesh et al. as a redesign of Conformer-based sequence modeling for efficient training and inference (Rekesh et al., 2023). In published systems it serves as the encoder backbone for automatic speech recognition (ASR), speech-to-text translation (AST), streaming ASR, multilingual ASR/AST, and self-supervised speech representation learning, with recurring design choices that include aggressive 8× subsampling, depth-wise separable convolutional front-ends, and alternative attention mechanisms for long-context efficiency (Huang et al., 2024, Koluguri et al., 2024).

1. Definition and lineage

FastConformer inherits the basic Conformer organization introduced by Gulati et al. in which each layer applies a feed-forward module, self-attention, a convolutional module, a second feed-forward module, and layer normalization with residual connections. The defining changes reported by Rekesh et al. are a fully redesigned 8× downsampling front-end instead of 4×, depth-wise separable convolutions in the subsampling layers, reduced channel count and smaller kernels in the front-end, and an optional post-training replacement of full-context attention with Longformer-style local attention plus a global token (Rekesh et al., 2023).

The architecture was quickly generalized beyond a single implementation. In the 2024 Canary work, FastConformer denotes an encoder for a multilingual attention encoder-decoder system in which the downsampling factor is increased from 4 to 8 and the attention kernel is replaced by a linearized attention mechanism whose memory and compute grow as $O(L\cdot d)$ rather than $O(L^2\cdot d)$ (Puvvada et al., 2024). In later reports, the same name is also used for variants with standard scaled dot-product multi-head self-attention, hybrid local-global attention, or limited-context streaming masks, indicating that the term identifies an efficiency-oriented Conformer family rather than a single immutable block definition (Rekesh et al., 2023, Koluguri et al., 2024).

A common simplification is to equate FastConformer with only one acceleration trick. The literature instead ties its identity to a cluster of design moves: stronger temporal subsampling, lighter convolutional processing, and context mechanisms that make longer sequences, larger models, or lower-latency inference feasible on practical hardware (Rekesh et al., 2023, Koluguri et al., 2024).

2. Encoder block and subsampling design

A representative FastConformer block, as specified in the Romanian ASR report, is

$$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$

$$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$

$$y_4 = y_3 + \tfrac12\cdot \mathrm{FFN}(\mathrm{LayerNorm}(y_3)),\qquad \mathrm{Output} = \mathrm{LayerNorm}(y_4),$$

with pre-norm, residual connections around FFN, MHSA, and ConvModule, and dropout inside the FFN and convolution module (Pirlogeanu et al., 5 Nov 2025). This is consistent with the macaron-style arrangement described in earlier FastConformer reports, even when the attention and front-end details differ (Rekesh et al., 2023).

The convolutional module is typically described as a pointwise expansion, gated linear unit, depthwise convolution, normalization and nonlinearity, and a final pointwise projection. In the long-sequence FastConformer XXL description, the convolution path is given as pointwise conv $\rightarrow$ GLU $\rightarrow$ depthwise conv $\rightarrow$ batch-norm $\rightarrow$ swish non-linearity $\rightarrow$ pointwise conv (Koluguri et al., 2024). Related reports use nearly the same template, with minor differences in activation or normalization placement (Grigoryan et al., 18 Jul 2025, Sekoyan et al., 17 Sep 2025).

The most stable architectural signature is the aggressive front-end reduction in sequence length. Rekesh et al. describe three depth-wise separable 1D convolutional subsampling layers, each with stride 2, kernel size 9, and 256 output channels, so that a sequence sampled every 10 ms becomes one frame every 80 ms after subsampling (Rekesh et al., 2023). In the long-sequence speech-recognition and translation study, a 60 s utterance sampled at 16 kHz and featurized with 25 ms windows and 10 ms hop becomes approximately $O(L^2\cdot d)$0 raw frames and then approximately $O(L^2\cdot d)$1 feature-frames after 8× subsampling, which is the basis for fitting full attention at that scale into GPU memory (Koluguri et al., 2024).

Published FastConformer systems span a wide parameter range. Rekesh et al. define model sizes L, XL, and XXL with 120 M, 600 M, and 1.1 B parameters respectively, without changing the core block (Rekesh et al., 2023). The long-sequence FastConformer-XXL configuration is reported with $O(L^2\cdot d)$2, $O(L^2\cdot d)$3, $O(L^2\cdot d)$4 heads, and $O(L^2\cdot d)$5 blocks for approximately $O(L^2\cdot d)$6 billion parameters (Koluguri et al., 2024). At smaller scales, the Romanian system uses a 17-layer, approximately 110 M-parameter encoder (Pirlogeanu et al., 5 Nov 2025), while the Arabic models use the NeMo fastconformer_hybrid_large configuration with 18 FastConformer blocks, model dimension $O(L^2\cdot d)$7, FFN hidden size $O(L^2\cdot d)$8, 8 attention heads, and convolution kernel size $O(L^2\cdot d)$9 (Grigoryan et al., 18 Jul 2025).

3. Attention backends and context regimes

The literature reports several distinct attention regimes under the FastConformer name. In Rekesh et al., the base model uses standard multi-head self-attention during full-context training and can then be converted post-training to limited-context attention with a single global token. In that regime, token $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$0 attends only to neighbors in $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$1, while a special global token attends to all positions and is attended by all positions; the resulting local-only cost is $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$2 when $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$3, rather than $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$4 (Rekesh et al., 2023).

The long-sequence FastConformer XXL work uses a different hybrid formulation. Each Conformer block divides the sequence into local blocks of length $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$5 and selects every $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$6-th frame as a global summary key, yielding a complexity of $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$7 instead of dense quadratic attention at every layer. For the reported setting with $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$8, $$x_0 = \mathrm{LayerNorm}(x),\qquad y_1 = x + \tfrac12\cdot \mathrm{FFN}(x_0),$$9, and $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$0, the attention cost is stated to be much smaller than $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$1 while preserving global access through sparse summary positions (Koluguri et al., 2024).

Canary introduces yet another variant. The 2024 data-efficient multilingual model describes FastConformer with a linearized attention mechanism

$$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$2

with $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$3, specifically to reduce the quadratic dependence on sequence length in the encoder (Puvvada et al., 2024). By contrast, the 2025 Canary-1B-v2 report summarizes a FastConformer encoder with standard scaled dot-product multi-head self-attention and states that no explicit sinusoidal or rotary embeddings are added, with positional information carried by convolutional subsampling and learned content interactions (Sekoyan et al., 17 Sep 2025). The coexistence of these descriptions indicates that attention implementation is application-dependent rather than fixed across all FastConformer systems.

Streaming adaptations further specialize the context mechanism. The stateful streaming FastConformer constrains the left and right context in every layer and introduces activation caches for depth-wise convolutions and self-attention keys and values, so that chunk-aware inference matches parallel full-context output exactly while avoiding duplicate compute (Noroozi et al., 2023). This design is explicitly intended to eliminate the training-versus-inference context mismatch that affects many streaming models (Noroozi et al., 2023).

4. Training objectives, decoders, and optimization regimes

FastConformer has been paired with multiple decoder families. The Polish case study states that the output of the encoder can be decoded using RNN-T or CTC loss, with RNN-T by default (Pietroń et al., 25 Feb 2026). Rekesh et al. report FastConformer with RNN-T for ASR, a Transformer decoder for speech translation, and limited-context post-training for long-form transcription (Rekesh et al., 2023). Later systems broaden this pattern further through explicitly hybrid decoders.

Two hybrid formulations recur. In streaming ASR, a shared FastConformer encoder branches into both CTC and RNNT heads and is trained with

$$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$4

after which either head may be used at inference time (Noroozi et al., 2023). In the long-sequence and Romanian systems, the encoder is trained with a Token-Duration Transducer and CTC combination,

$$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$5

or equivalently $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$6 with $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$7, enabling greedy TDT, ALSD, and CTC beam search with a 6-gram token LLM as alternative decoding strategies (Koluguri et al., 2024, Pirlogeanu et al., 5 Nov 2025).

Data preparation is often as important as the encoder itself. The long-sequence speech recognition and translation study replaces lowercased partial segments with sentence-level punctuation-and-capitalization (PnC) segments constructed so that every segment begins with an uppercase letter and ends with one of . ? or !. Training data are binned into duration windows $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$8, $$y_2 = y_1 + \mathrm{MHSA}(\mathrm{LayerNorm}(y_1)),\qquad y_3 = y_2 + \mathrm{ConvModule}(\mathrm{LayerNorm}(y_2)),$$9, and $$y_4 = y_3 + \tfrac12\cdot \mathrm{FFN}(\mathrm{LayerNorm}(y_3)),\qquad \mathrm{Output} = \mathrm{LayerNorm}(y_4),$$0 seconds, and models are fine-tuned for 25k steps with 5k warm-up steps, peak learning rate $$y_4 = y_3 + \tfrac12\cdot \mathrm{FFN}(\mathrm{LayerNorm}(y_3)),\qquad \mathrm{Output} = \mathrm{LayerNorm}(y_4),$$1, and A100 80 GB batch sizes of 16, 8, and 2 respectively (Koluguri et al., 2024).

Large multilingual systems extend this with staged pretraining. Canary-1B-v2 uses a two-stage process of 150k and 100k pre-training steps on a total 1.7M hours of data, followed by a 10k-step fine-tuning stage on a high-quality subset with dynamic data balancing (Sekoyan et al., 17 Sep 2025). The earlier Canary report emphasizes dynamic bucketing, stochastic weighted multiplexing across datasets, synthetic data from machine translation, and noise-robust fine-tuning, while the NEST framework uses fixed random projection rather than clustering-based quantization and a generalized noisy speech augmentation that teaches the model to disentangle the main speaker from noise or other speakers (Puvvada et al., 2024, Huang et al., 2024).

5. Empirical performance across ASR and AST settings

On short-utterance ASR, Rekesh et al. report that Fast Conformer + RNNT reduces LibriSpeech test-other WER from 5.19% for Conformer + RNNT to 4.99%, while encoder compute drops from 143.2 GMAC to 48.7 and inference speed increases from 169 to 467 samples/s, corresponding to an approximately 2.8× encoder speed-up (Rekesh et al., 2023). In the same report, long-form maximum processable length on an A100 at batch 1 increases from 15 min for full-context Conformer to 25 min for full-context FastConformer, and to 675 min, or 11.25 h, for FastConformer with local attention (Rekesh et al., 2023).

For long-sequence ASR with restored punctuation and capitalization, the 2024 FastConformer XXL study reports Greedy WER on Earnings-21 of 30.6 for partial PnC (0–20 s), 24.5 for complete PnC (0–20 s), 23.3 for complete PnC (0–40 s), and 23.1 for complete PnC (0–60 s); on Earnings-22 the corresponding values are 32.8, 25.5, 24.8, and 25.0 (Koluguri et al., 2024). The same study states that sentence-level PnC gives approximately 20–22% relative WER reduction versus partial segments, that extending to 40 s yields another approximately 5% relative gain, and that past 40 s the gains plateau (Koluguri et al., 2024). For speech translation on MuST-C de→en, the reported valid/test BLEU values are 22.1/21.5 for 0–20 s, 23.1/22.4 for 0–40 s, and 22.5/21.9 for 0–60 s, again showing a plateau beyond 40 s (Koluguri et al., 2024).

Streaming results show a different performance frontier. On LibriSpeech test-other, the cache-aware stateful model achieves 7.1 WER for CTC and 6.3 for RNNT at 1360 ms encoder-induced latency, compared with buffered baselines of 8.0 and 11.3 at 1500 ms and 2000 ms respectively (Noroozi et al., 2023). Multi-latency training improves robustness across target latencies: on the same benchmark, multi-latency RNNT improves from 6.4/5.9/5.4 to 6.2/5.5/5.2 at 40/240/520 ms, while multi-latency CTC improves from 7.9/7.3/6.2 to 7.6/6.5/6.0 (Noroozi et al., 2023).

Multilingual and language-specific systems show that the architecture remains competitive at very different scales. Canary-1B-v2 reports English ASR with average WER 7.15% at RTFx 749, compared with Whisper-large-v3 at 7.44% and RTFx 145; its average multilingual ASR WER over 24 languages is 8.1%, compared with 9.9% for Whisper-large-v3, and AST X→En reaches 79.3 COMET (Sekoyan et al., 17 Sep 2025). The Romanian FastConformer system, trained on over 2,600 h and decoded with a 6-gram LM, reports WERs of 1.73 on RSC-eval, 8.12 and 10.75 on SSC-eval1/2, 3.92 on CDEP-eval, 3.29 on CV21-RO, 8.85 on Fleurs-RO, and 23.40 on USPDATRO, with up to 27% relative WER reduction compared to previous best-performing systems (Pirlogeanu et al., 5 Nov 2025). In Arabic, the MSA FastConformer model reports 8.52 on MASC, 7.97 on MCV, and 5.08 on FLEURS, outperforming the prior open and proprietary benchmarks listed in that report (Grigoryan et al., 18 Jul 2025). In Polish read speech, the comparative study reports FastConformer at 5.99% WER on Mozilla Common Voice (Pietroń et al., 25 Feb 2026).

6. Applications, ecosystem, and limitations

FastConformer has become a general backbone rather than only an ASR encoder. In NEST, a self-supervised FastConformer with 8× sub-sampling is used for speech recognition and translation, speaker verification, diarization, and spoken language understanding; reported SUPERB results include 94.94% speaker identification accuracy, 3.85% speaker verification EER, 2.28 diarization DER, 1.95 phoneme recognition PER, and 3.49% ASR CTC WER for NEST-L, while NEST-XL reaches 32.42 average BLEU on En→De/Es/Fr AST (Huang et al., 2024). This suggests that the architecture’s efficiency gains are not restricted to token transcription.

The software ecosystem is strongly tied to NVIDIA NeMo. The long-sequence FastConformer XXL work states that model code, training recipes, and 1.1 B-parameter weights are available in NVIDIA NeMo and on HuggingFace, and that training used NVIDIA NeMo toolkit v1.x, PyTorch, and Apex AMP on A100 80 GB GPUs with mixed precision (Koluguri et al., 2024). NEST, the Arabic models, and later multilingual FastConformer systems likewise report public code, checkpoints, or both through NeMo-linked releases (Huang et al., 2024, Grigoryan et al., 18 Jul 2025, Sekoyan et al., 17 Sep 2025).

Several limitations are also explicit in the literature. The most direct is that longer context is not automatically better: in the punctuated long-sequence study, ASR and AST gains plateau beyond 40 s despite 60 s training being feasible, and the authors explicitly attribute this to a likely need for architectural improvements to exploit more than 40 s of context (Koluguri et al., 2024). A second limitation is deployment trade-off rather than raw accuracy. The Romanian report notes that the 6-gram LM occupies approximately 2 GB RAM and that TDT+ALSD provides marginal WER gains at a substantial latency cost, motivating future work on smaller LMs and optimized search algorithms (Pirlogeanu et al., 5 Nov 2025). A third limitation concerns domain transfer: the Polish case study reports FastConformer on Common Voice but does not report FastConformer on medical interviews, while concluding more broadly that purpose-built, domain-tuned systems outperform generic open models in noisy clinical dialogue (Pietroń et al., 25 Feb 2026).

Future directions in the cited work follow from these constraints. The Arabic report recommends self-supervised pre-training on large unlabeled Arabic speech pools, adapter-based fine-tuning for additional dialects, and extensions to relative positional encoding windows and convolution kernel sizes for ultra-long recitations (Grigoryan et al., 18 Jul 2025). The Romanian report identifies streaming and on-device variants as a next step (Pirlogeanu et al., 5 Nov 2025). Taken together, these directions indicate that FastConformer remains an active architectural substrate for scaling, streaming, multilingual transfer, and domain specialization rather than a closed design point.