Explaining Conformer-1’s underperformance on the Numbers domain

Determine the factors responsible for Conformer-1’s inferior performance relative to other providers on the in-house Numbers domain and ascertain whether the use of pseudo-labels and the applied filtering strategy causally contribute to this discrepancy.

Background

In head-to-head comparisons against commercial ASR providers on multiple in-house domains, Conformer-1 wins most domains but lags on the Numbers domain. The authors hypothesize that pseudo-labeling choices and filtering strategies may be affecting numerical transcription quality.

They explicitly state that understanding this discrepancy remains unresolved and identify it as an area for future research, indicating the need for targeted analysis of dataset composition, labeling strategies, and model behavior on numerical content.