Is 2B parameters the effective scaling limit for self-supervised speech encoders?

Determine whether a parameter scale of approximately 2 billion constitutes an effective upper limit for scaling self-supervised speech representation learning models (such as wav2vec 2.0) for speech tasks, by ascertaining if additional capacity beyond 2B yields diminishing returns or if 2B parameters are already sufficient for most downstream speech applications.

Background

Prior public speech SSL encoders (e.g., XLS-R and USM) have reached roughly 2B parameters. Whether further increases in model size would provide meaningful gains remained unsettled.

Omnilingual ASR scales wav2vec 2.0 encoders up to 7B parameters across 4.3M hours of unlabeled speech from 1,600+ languages, positioning the work to empirically probe scaling behavior. The open question asks if 2B marks a practical ceiling or if sizable benefits persist with larger models.