SSA-HuBERT-XL: African Speech Encoder

• SSA-HuBERT-XL is a scalable self-supervised speech encoder designed exclusively for diverse Sub-Saharan African languages, addressing under-representation in speech technology.
• It extends the HuBERT architecture with a 7-layer CNN front-end and a 48-layer Transformer backbone, scaling up to 964 million parameters for enhanced ASR and LID performance.
• Empirical evaluations show reduced word error rates (up to 1.6% lower than prior large models) and improved benchmarks on seen and unseen languages using a corpus of 60,000 hours.

SSA-HuBERT-XL is a large-scale self-supervised speech encoder tailored exclusively for Sub-Saharan African languages, extending the HuBERT architecture to new capacity and data scale regimes. This model is designed to address under-representation of African languages in multilingual speech processing research and deployed systems by leveraging substantial model capacity and Africa-centric acoustic corpora. With 964 million parameters, SSA-HuBERT-XL outscales prior African speech encoders by over an order of magnitude, and is trained solely on approximately 60,000 hours of raw African audio data. The model sets new benchmarks for automatic speech recognition (ASR) and language identification (LID) in the low-resource, Africa-centric speech domain (Caubrière et al., 28 Nov 2025).

1. Architecture of SSA-HuBERT-XL

SSA-HuBERT-XL retains the convolutional feature encoder and Transformer backbone of the original HuBERT design, with modifications primarily in scale:

• Convolutional Front-End: A 7-layer 1D CNN processes raw audio with strides of [5, 2, 2, 2, 2, 2, 2] and kernel sizes [10, 3, 3, 3, 3, 2, 2], each followed by layer normalization and GELU non-linearity. The output feature dimension is 1280.
• Transformer Encoder: 48 layers with 1280-dimensional hidden states ($d_{\text{model}}=1280$), 16 self-attention heads per layer (each head 80-dimensional), and 5120-dimensional position-wise feed-forward sub-networks. Dropout of 0.1 is applied to both attention and feed-forward operations. Absolute (learned) positional embeddings are injected at each layer input.
• Parameterization: The model comprises 964 million parameters, nearly 10$\times$ the Base variant.
• Non-modified Design Choices: No introduction of adapters, hierarchical layering, or architectural deviations beyond scaling channel counts and transformer dimensionality from published HuBERT Base/Large versions.

2. Pre-Training Objective and Loss

The objective of SSA-HuBERT-XL is inherited from HuBERT’s masked prediction strategy:

• Pseudo-Label Extraction: Discrete "pseudo-labels" are computed via offline k-means clustering on MFCC features, or on intermediate HuBERT representations.
• Masking Strategy: Given input audio $X$, approximately 15% of frames (selected as contiguous spans of roughly 10 frames per span, spaced by $\sim$25 frames) are masked.
• Prediction Target: For each masked timestep $t \in M$, the model outputs a softmax probability $p(c_t| X_{\neg M})$ over $K=500$ cluster IDs.
• Loss Function: Cross-entropy over masked frame positions:

$$\mathcal{L} = -\sum_{t \in M} \log p(c_t|X_{\neg M})$$

where $M$ is the set of masked indices, $c_t$ the pseudo-label at time $t$, and $X_{\neg M}$ the masked input sequence.

All clustering, masking, and modeling hyperparameters directly follow those of HuBERT [5], ensuring controlled analysis of scaling behaviors.

3. Training Data Composition and Experimental Configuration

SSA-HuBERT-XL is pre-trained exclusively on African speech, comprising a highly diverse set of data sources:

• Corpus: Approximately 60,000 hours of raw, 16 kHz audio.
• Language Coverage: 18 Sub-Saharan languages, prioritized by data hours: Swahili, Hausa, Kinyarwanda, African-accented French, Bambara, Lingala, Sango, Tamasheq, Maninkakan, Songhai, Fula, Luba-Lulua, Kituba, Zarma, Wolof, Dyula, Mossi, Gulmancema.
• Optimization Settings:
  • Framework: Fairseq.
  • Hardware: 8×NVIDIA H100 GPUs.
  • Total Updates: 450,000 steps.
  • Batch Size: 56.25 seconds of audio per GPU (utilizing gradient accumulation over 32 steps).
  • Optimizer: Adam, with parameters $\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-6}$.
  • Learning Rate: Peak at $10^{-3}$, linear warm-up for 25,000 steps, then polynomial decay to zero.
  • Regularization: Weight decay 0.01, dropout 0.1.

4. Comparative Evaluation on Downstream Tasks

Evaluation on the FLEURS-SSA 20-language subset quantifies performance in ASR (with a greedy CTC decoder) and LID:

Model	#Params	CER (%)	WER (%)
SSA-HuBERT-Base-v2	95M	13.0	45.1
AfriHuBERT-n [7]	95M	11.5	40.6
SSA-HuBERT-Large	317M	10.5	37.8
XLS-R 128 [1]	317M	10.8	39.2
SSA-HuBERT-XL	964M	10.1	37.2

SSA-HuBERT-XL reduces WER by 1.6% (relative, vs. Large) and 17.5% (relative, vs. Base). Improvements are most pronounced for languages present in the pre-training corpus (“seen”), but gains extend to “unseen” languages, though with diminished magnitude.

LID evaluation presents the following results:

Model	LID (%)	LID_smooth (%)
AfriHuBERT-n [7]	93.3	93.5
SSA-HuBERT-Base-v2	78.3	85.5
SSA-HuBERT-Large	89.9	93.7
SSA-HuBERT-XL	92.7	93.1

AfriHuBERT-n retains a slight lead in LID, attributed to broader language coverage during pre-training. SSA-HuBERT-XL, however, closes most of the performance gap with increased encoder capacity and corpus scale (Caubrière et al., 28 Nov 2025).

5. Scaling Behaviors and Empirical Trends

Experimental results indicate:

• Capacity-Data Interplay: Both the Large (317M) and XL (964M) encoders yield lower CER/WER than Base, verifying that expanded model size enables better utilization of large-scale African speech corpora.
• Pre-training Language Coverage: Downstream ASR gains are maximized for languages included in pre-training (“seen”). Unseen languages benefit, but the incremental error reduction is smaller (e.g., XL delivers a 1–2 percentage point WER drop vs. Large on unseen languages, versus 2–3 on seen).
• Saturation Dynamics: No evidence of strict performance saturation up to 964M parameters—WER improvements from Large to XL remain positive but are diminishing (approx. 1.6% relative). Additional scaling or increased dialectal coverage will be required to investigate potential limits.

A plausible implication is that both parameter scaling and diversified corpus coverage are necessary to further reduce error rates in low-resource language speech recognition.

6. Significance and Implications for African Speech Technology

SSA-HuBERT-XL establishes a new benchmark for Africa-centric self-supervised speech encoding. Its design—massive parameterization, exclusive focus on Sub-Saharan African languages, large unlabeled corpus, and HuBERT-style pre-training—demonstrates that substantial architectural and data scaling can yield state-of-the-art results for ASR and LID in settings with substantial linguistic and resource diversity. The model’s open-weight release enables further downstream work, and its empirical trends provide guidance for future research into scaling, data composition, and transfer for low-resource multilingual speech processing (Caubrière et al., 28 Nov 2025).