Feasibility of SSL-based Speech Foundation Models

Determine whether self-supervised learning techniques for speech can yield a single foundation model for speech processing whose frozen representations transfer effectively across diverse downstream tasks, including content recognition, speaker-related tasks, prosody, semantics, and generation, without extensive task-specific fine-tuning.

Background

The paper observes that prior evaluations of speech self-supervised learning (SSL) models have largely focused on narrow task sets, most commonly automatic speech recognition, unlike the multi-task benchmarks prevalent in NLP. This leaves uncertainty around whether SSL can support a general-purpose foundation model for speech that performs well across heterogeneous tasks.

To address this, the authors introduce SUPERB, a broad benchmark spanning 15 tasks across content, speaker, prosody, semantics, and generation, and propose a unified evaluation framework using frozen encoders and lightweight task-specific heads. The open question explicitly raised in the introduction motivates the creation of SUPERB to test the paradigm systematically.