---
title: 'HuBERT-Base: Speech & Hungarian Models'
url: https://www.emergentmind.com/topics/hubert-base
type: topic
---

# HuBERT-Base: Speech & Hungarian Models

HuBERT-Base refers to two distinct models in the literature: (1) the Hidden-Unit BERT (HuBERT) model for self-supervised speech representation learning, as introduced in "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units" [2106.07447], and (2) the Hungarian BERT model ("huBERT"), a monolingual contextual language model for Hungarian, examined in "Evaluating Contextualized Language Models for Hungarian" [2102.10848]. This entry disambiguates and details both models according to their respective research contexts, architectures, objectives, and empirical performance.

## 1. Hidden-Unit BERT (HuBERT-Base) for Speech Representation

HuBERT-Base [2106.07447] is a self-supervised speech representation model built to address three challenges in unsupervised speech pretraining: the absence of a lexicon, multiple variable-length sound units per utterance, and unsegmented continuous input. It utilizes offline clustering and masked prediction for learning acoustic-linguistic representations. The base configuration contains approximately 95 million parameters.

### Architectural Details

- **Convolutional waveform encoder**: 7 layers, each with 512 channels; striding pattern [5, 2, 2, 2, 2, 2, 2] (total 320× down-sampling); kernel widths [10, 3, 3, 3, 3, 2, 2].
- **Transformer encoder**: 12 layers; hidden size 768; feed-forward inner dimension 3072; 8 attention heads; LayerDrop probability 0.05.
- **Positional encoding**: Fixed sinusoidal scheme as in BERT/Transformer architectures.
- **Output projection**: 256-d embedding, softmax over $C$ clusters ($C=100$ at iteration 1).

## 2. Self-Supervised Objective and Offline Clustering

HuBERT-Base employs a masked hidden-unit prediction objective, predicting aligned cluster labels over masked spans in the input. For input sequence $X=[x_1,\dots,x_T]$, masked spans $M\subset\{1,\dots,T\}$ are replaced by a learned embedding $\tilde{x}$. The Transformer $f$ computes $o_t$ and outputs the probability distribution:

$$
p_f(c | \tilde{X}, t) = \frac{\exp(\mathrm{sim}(A o_t, e_c)/\tau)}{\sum_{c'=1}^C \exp(\mathrm{sim}(A o_t, e_{c'})/\tau)}
$$

where $A$ is a learned projection, $e_c$ is the cluster embedding, and $\tau=0.1$. The masked loss is defined as:

$$
L_\mathrm{m}(f; X, M, Z) = -\sum_{t\in M} \log p_f(z_t | \tilde{X}, t)
$$

with $Z=[z_1, ..., z_T]$ the offline-computed cluster assignments. Only masked frames are included in the loss ($\alpha=1$).

Offline clustering is performed in two iterations:
- **Iteration 1**: MFCC+$\Delta$+$\Delta\Delta$ features from LibriSpeech-960h; $k$-means with $100$ clusters.
- **Iteration 2**: Layer 6 output from Base-it1; $k$-means with $500$ clusters, using 10% subsampled data.
The intuition is that as HuBERT’s representations improve, clusterings align more closely with true phonetic units.

## 3. Pre-training and Fine-tuning Regimen

Pre-training uses LibriSpeech-960h, 32 GPUs, an effective batch size of ≈2800 seconds of audio per step, Adam optimizer ($\beta_1=0.9$, $\beta_2=0.98$), and linear learning rate warm-up (peak $\mathrm{lr}=5\times10^{-4}$). Masking uses 10-frame spans ($\approx$200 ms) with 8% of frames as span starts (SpanBERT-style). Iteration 1 employs 250,000 steps; Iteration 2 uses 400,000 steps.

Fine-tuning freezes the convolutional encoder and optimizes the Transformer and CTC head over a 29-token vocabulary (26 English letters, space, apostrophe, blank) using CTC loss. A "freeze-step" hyperparameter determines how long to train only the new softmax layer, before unfreezing the Transformer. Decoding leverages wav2letter++ beam search with LM score fusion.

## 4. Empirical Results and Ablations

HuBERT-Base demonstrates competitive or superior word error rate (WER) to wav2vec 2.0-Base across all labeled data regimes on LibriSpeech:

| Labeled Hours | wav2vec 2.0-Base (test-other WER) | HuBERT-Base (test-other WER) |
|:-------------:|:-------------------------------:|:----------------------------:|
| 10 min        | 15.6                            | 15.3                         |
| 1 h           | 11.3                            | 11.3                         |
| 10 h          | 9.5                             | 9.4                          |
| 100 h         | 8.0                             | 8.1                          |

Ablation studies indicate:
- Best results are achieved with masked-only loss ($\alpha=1$), not including unmasked frames.
- Teacher cluster quality (evaluated by phone-normalized MI) improves over iterations: 0.25 (MFCC), 0.68 (Base-it1 L6), 0.70 (Base-it2).
- Increasing mask probability or batch size improves performance (batch size scaling 32→128 GPUs yields ≈20% WER reduction).
- Cluster ensembles and longer pre-training further reduce WER.
- Clustering on middle Transformer layers (6–9) yields highest phone purity; iterative pseudo-label refinement improves phonetic alignment.

## 5. Hungarian BERT (huBERT-Base) as Monolingual Language Model

huBERT-Base [2102.10848] is a monolingual contextualized language model for Hungarian, architecturally mirroring BERT-Base: 12 Transformer layers, hidden size 768, 12 attention heads, feedforward size 3072, and ≈110M parameters. No modifications are introduced versus BERT-Base aside from trained embeddings and vocabulary.

### Tokenization and Pretraining

- **Vocabulary**: ≈32,000 WordPiece subwords, trained on Hungarian Webcorpus 2.0 (~9B tokens).
- **Pretraining objectives**: Masked language modeling (15% tokens masked) and next-sentence prediction, replicating Devlin et al. (2018).
- **Training regimen**: 256 sequences/batch (up to 512 length), learning-rate warm-up and linear decay, ≈1M steps.

### Empirical Performance

Freezing model weights, performance on three Hungarian NLP tasks is reported:
- **Morphological probing**: Last subword representations from middle layers (4–7) achieve >90% accuracy on noun/adjective probes; overall probes ~85–90% (vs. ~80–85% XLM-RoBERTa; ~75–80% mBERT).
- **POS tagging (Szeged UD)**: Layer 6 achieves ~95% accuracy, with final layer at ~93%; weighted layer combination yields ~96%.
- **NER (Szeged corpus)**: Layer 6 F1 ≈92%, layer 12 ≈90%; surpasses multilingual and distilled baselines by 1–4 points.

huBERT-Base consistently outperforms mBERT, XLM-RoBERTa, and other multilingual models in token-level tasks, with the greatest margin in lower–middle layers, reflecting alignment of subword segmentation with Hungarian morphology.

## 6. Layerwise Insights and Best Practices

Empirical studies show linguistic task optima in layers 4–8: lower layers encode local morphological cues, middle layers capture part-of-speech and phrase structure, higher layers emphasize semantics. For rich morphology languages like Hungarian, last-subword representations capture discriminative suffix information.

Recommended practices:
- For token-level tasks, use last subword representation.
- Prefer layers 4–7 for morphology and POS tagging, layers 6–9 for NER and semantics. Weighted sums aid at the cost of model complexity.
- For fine-tuning, favor higher learning rates for middle layers or gradual unfreezing.

## 7. Comparative Discussion and Significance

Both HuBERT and huBERT-Base (Hungarian) demonstrate the efficacy of domain-adapted pretraining—either aligning acoustic units for self-supervised speech, or leveraging rich monolingual corpora and tailored subword segmentation for morphologically complex languages. HuBERT-Base for speech reveals that iterative offline clustering and masked prediction converge to high-quality phonetic and word-level representations with a lightweight architecture and high data efficiency, matching or exceeding previous state-of-the-art models. huBERT-Base, as a monolingual BERT-Base trained on a large Hungarian corpus, achieves state-of-the-art performance for Hungarian morphology, POS, and NER, substantiating the value of language-specific vocabulary and curated subwords for inflectionally rich languages [2106.07447, 2102.10848].

Source: https://www.emergentmind.com/topics/hubert-base