DF-Conformer: Efficient Speech Model

• DF-Conformer architecture is a hybrid model that marries convolution, self-attention, and factorization to efficiently capture both local and global speech features.
• It employs dual decoders, low-rank modules, deformable convolutions, and FAVOR+ linear attention to optimize both accuracy and computational efficiency.
• This model is effectively applied in multilingual ASR, speech enhancement, and mask prediction tasks, achieving significant performance improvements across benchmarks.

DF-Conformer architectures refer to a family of speech representation models—often based on the Conformer block design—that integrate advanced convolutional, self-attention, and low-rank/factorized or linearized components to achieve efficient and effective sequence modeling. Multiple research threads have used the “DF-Conformer” terminology, with overlapping but distinct meanings: (1) Dual-Decoder Conformer for multilingual ASR, (2) Dilated FAVOR Conformer for speech enhancement and rapid self-attention, (3) Deformer Conformer using deformable depthwise convolutions to enhance local feature extraction, and (4) DF-Conformer incorporating low-rank factorization for parameter and computational efficiency. All variants build upon the structural core of the Conformer architecture and are designed for speech recognition, enhancement, or related speech-processing domains.

1. Foundational Conformer Architectures

The original Conformer block merges macaron-style feed-forward layers, multi-head self-attention, and a convolutional module to jointly model local and global dependencies in sequence data. The typical block structure is as follows:

1. First feed-forward module: Macaron-style, with 0.5× residual scaling, expansion ×4, Swish (or GELU) nonlinearity, pre-layer normalization.
2. Multi-head self-attention (MHSA): Projects queries, keys, values; applies scaled dot product with either sinusoidal or relative positional encoding.
3. Convolution module: Point-wise 1×1 convolution expands channels, gated linear unit (GLU) activation, 1-D (depthwise) convolution, batch normalization, Swish activation, pointwise projection back to model dimension.
4. Second feed-forward module: Also macaron-style.
5. Final layer normalization.

The general block output is:

$$\mathrm{Output} = \mathrm{LayerNorm}\left(x + 0.5\,\mathrm{FFN}_1(x) + \mathrm{MHSA}(\cdot) + \mathrm{Conv}(\cdot) + 0.5\,\mathrm{FFN}_2(\cdot)\right)$$

This recipe—macaron FFN sandwiching, multi-head self-attention, and a convolution module—forms the basis for all DF-Conformer variants. The original Conformer achieves state-of-the-art WERs on major ASR benchmarks and sets a baseline for the more advanced variants (Gulati et al., 2020).

2. Dual-Decoder DF-Conformer for Multilingual Speech Recognition

The “DF-Conformer” introduced by Kakwani et al. (N, 2021) consists of:

• 12-layer Conformer Encoder: Processes 40-dimensional mel-filterbank input features, applies two-stage convolutional subsampling (stride 2×2), sinusoidal positional embeddings, and standard conformer blocks with model dimension 4096 and eight attention heads.
• Dual Transformer Decoders:
  • Phoneme Decoder (PHN-DEC): 3-layer Transformer decoder performing auxiliary phoneme recognition, using masked self-attention and encoder cross-attention.
  • Grapheme Decoder (GRP-DEC): 6-layer Transformer decoder predicting grapheme sequences, which are prefixed during decoding by an explicit language identification tag.
• Language-ID Classifier: Takes the final encoder output, aggregates via average pooling, and passes through two dense layers to produce a 6-way language prediction.
• Joint Multi-Task Training: The model is trained end to end, optimizing the sum:

$$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$

with $\alpha=0.3$, $\beta=0.5$, $\gamma=0.5$, $\pi=10.0$ for, respectively, encoder CTC, phoneme decoder, grapheme decoder (including language-ID token), and language-ID classifier losses.

• Performance: On six low-resource Indian languages, the full DF-Conformer reduces WER relative to GMM-HMM by 41% and TDNN-HMM by 11%, outperforming single decoder baselines and comparable approaches.

A key architectural element is “conditional decoding,” where the grapheme decoder must first predict a language-ID tag, enforcing language-conditional output throughout the sequence (N, 2021).

3. Efficiency Enhancements: FAVOR+ Linear Attention and Dilated Convolution

To address the quadratic complexity of standard self-attention in Conformer blocks, the “Dilated FAVOR Conformer (DF-Conformer)” integrates two key modules:

• FAVOR+ Attention: Replaces softmax with a random-feature approximation that reduces the cost from $O(T^2 d)$ to $O(T d r)$, where $r \ll T$. Given queries and keys $Q, K \in \mathbb{R}^{T \times d}$, FAVOR+ computes:
  • $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$0: positive orthogonal random feature maps.
  • $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$1; then output $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$2 where $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$3.
• Dilated Depthwise Convolutions: The convolutional submodules expand local receptive fields via exponential dilation scheduling (e.g., $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$4), without large increases in computation (Seki et al., 4 Nov 2025, Koizumi et al., 2021).

This combination allows DF-Conformer blocks to attain efficient, scalable sequential modeling, which is especially critical for long speech inputs. In speech enhancement, these architectural choices yield substantive improvements in SI-SNRi and ESTOI at minimal additional compute cost (Koizumi et al., 2021). Table summarizing efficiency and effectiveness:

Model	Complexity per block	SI-SNRi (dB)	RTF
Conformer (vanilla)	$$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$5	13.91	0.31
DF-Conformer (FAVOR+ plus dilated conv)	$$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$6	14.43	0.13

4. Advanced Local Modeling: Deformable Depthwise Convolution

The “Deformer” block *Editor's term replaces static depthwise convolution within the Conformer block with a deformable variant:

• Deformable Convolution: For each timestep $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$7 and kernel index $$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$8, the convolution computes:

$$L = \alpha L_\mathrm{ctc} + \beta L_\mathrm{pr} + \gamma L_\mathrm{gr} + \pi L_\mathrm{lid}$$9

where $\alpha=0.3$0 are scalar filter weights, $\alpha=0.3$1 are integer kernel offsets, and $\alpha=0.3$2 are learned, fractional offsets produced dynamically for each input. Linear interpolation (in 1-D) enables application to non-integer locations.

• Offset Prediction: A separate 1-D convolution (offset-CNN) outputs offsets for all positions and kernel indices; weight initialization of this CNN to zero enhances stability.
• Empirical Results: Insertion of deformable convolution in half the encoder layers yields $\alpha=0.3$3 relative WER improvement (no LM) and $\alpha=0.3$4 (with LM) over baseline Conformer on WSJ eval92.
• Receptive Field Dynamics: Lower layers learn small, symmetric offsets (high localization), while higher layers exploit broader, more adaptive contextual windows—validated by offset distribution statistics and visualization of attention patterns.

The deformable variant strengthens local feature extraction and enhances coupling to global context as processed by attention modules in deeper layers.

5. Parameter-Efficient DF-Conformer: Low-Rank Modular Factorization

A separate DF-Conformer variant replaces the largest parameter blocks in the Conformer with low-rank (LR) versions (Gulati et al., 2020):

• Low-Rank Feed-Forward Networks: Instead of standard matrices, the FFN uses rank-$\alpha=0.3$5 factorization, e.g., $\alpha=0.3$6, with $\alpha=0.3$7.
• Low-Rank Depthwise Convolution: The per-channel conv kernels $\alpha=0.3$8 are factorized as $\alpha=0.3$9, with $\beta=0.5$0.
• Computational Savings: If $\beta=0.5$1, FFN params decrease by 50%; $\beta=0.5$2 yields 75% fewer convolutional params. The result is a faster, smaller model maintaining recognition accuracy (WER increase $\beta=0.5$30.1% absolute for sensible choices of $\beta=0.5$4).
• Implementation: The full block and parameter schedule closely matches the original Conformer (ordering, normalization, gating), but replaces the heaviest linear modules with their factorized counterparts.

6. Speech Enhancement Applications and Integration with Mask Prediction

The DF-Conformer block serves as the mask prediction network within speech enhancement architectures such as Conv-TasNet (Koizumi et al., 2021). The improvement over TDCN++ and vanilla Conformer is attributable to:

• Linear FAVOR+ self-attention and dilated depthwise convolutions within the mask prediction stack.
• Integration Strategy: Each TDCN block in Conv-TasNet is replaced by a full DF-Conformer block, taking “bottleneck” dense projections as input, stacking $\beta=0.5$5 (e.g., 8) layers, and using cyclically increasing dilations.
• Training regimen: Models are trained on millions of noisy speech examples, using negative SNR-based losses and aggressive data and weight averaging strategies.

Experimentally, DF-Conformer achieves higher SI-SNRi and similar computational cost compared to equivalently sized TDCN++ and Conformer baselines.

7. Further Directions: Hybridization and Structured State-Space

Recent work proposes replacing FAVOR+ in the DF-Conformer block with structured state-space models (SSM), notably the bidirectional Hydra module (Seki et al., 4 Nov 2025). The block-wise structure (FFN → SSM → dilated conv → FFN) is maintained, with the claimed benefits of:

• Elimination of kernel approximation error from FAVOR+,
• Retention of linear complexity in sequence length,
• Improved global sequence modeling and focus due to exact, injective SSMs.

In sum, DF-Conformer denotes a spectrum of Conformer-based, hybrid sequence architectures that layer efficient self-attention mechanisms, enhanced local convolutions (static, low-rank, or deformable), dual-path decoders (for ASR), or acts as a drop-in module for mask prediction in speech enhancement and generative speech models. Each variant matches task requirements via carefully chosen attention/convolutional schemes, with rigorous empirical evidence supporting efficiency and performance gains (N, 2021, Koizumi et al., 2021, Xie et al., 2022, Gulati et al., 2020, Seki et al., 4 Nov 2025).