HuBERT-Base: Speech & Hungarian Models

• HuBERT-Base is a term referring to both a self-supervised speech representation model and a Hungarian contextual language model.
• The speech model employs iterative offline clustering, masked prediction, and transformer encoders to achieve competitive performance on LibriSpeech benchmarks.
• The Hungarian BERT leverages language-specific tokenization and tailored architecture to excel in morphology, POS tagging, and NER tasks.

HuBERT-Base refers to two distinct models in the literature: (1) the Hidden-Unit BERT (HuBERT) model for self-supervised speech representation learning, as introduced in "HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units" (Hsu et al., 2021), and (2) the Hungarian BERT model ("huBERT"), a monolingual contextual LLM for Hungarian, examined in "Evaluating Contextualized LLMs for Hungarian" (Ács et al., 2021). This entry disambiguates and details both models according to their respective research contexts, architectures, objectives, and empirical performance.

1. Hidden-Unit BERT (HuBERT-Base) for Speech Representation

HuBERT-Base (Hsu et al., 2021) is a self-supervised speech representation model built to address three challenges in unsupervised speech pretraining: the absence of a lexicon, multiple variable-length sound units per utterance, and unsegmented continuous input. It utilizes offline clustering and masked prediction for learning acoustic-linguistic representations. The base configuration contains approximately 95 million parameters.

Architectural Details

• Convolutional waveform encoder: 7 layers, each with 512 channels; striding pattern 5, 2, 2, 2, 2, 2, 2; kernel widths [10, 3, 3, 3, 3, 2, 2].
• Transformer encoder: 12 layers; hidden size 768; feed-forward inner dimension 3072; 8 attention heads; LayerDrop probability 0.05.
• Positional encoding: Fixed sinusoidal scheme as in BERT/Transformer architectures.
• Output projection: 256-d embedding, softmax over $C$ clusters ($C=100$ at iteration 1).

2. Self-Supervised Objective and Offline Clustering

HuBERT-Base employs a masked hidden-unit prediction objective, predicting aligned cluster labels over masked spans in the input. For input sequence $X=[x_1,\dots,x_T]$, masked spans $M\subset\{1,\dots,T\}$ are replaced by a learned embedding $\tilde{x}$. The Transformer $f$ computes $o_t$ and outputs the probability distribution:

$$p_f(c | \tilde{X}, t) = \frac{\exp(\mathrm{sim}(A o_t, e_c)/\tau)}{\sum_{c'=1}^C \exp(\mathrm{sim}(A o_t, e_{c'})/\tau)}$$

where $A$ is a learned projection, $e_c$ is the cluster embedding, and $C=100$0. The masked loss is defined as:

$C=100$1

with $C=100$2 the offline-computed cluster assignments. Only masked frames are included in the loss ($C=100$3).

Offline clustering is performed in two iterations:

• Iteration 1: MFCC+$C=100$4+$C=100$5 features from LibriSpeech-960h; $C=100$6-means with $C=100$7 clusters.
• Iteration 2: Layer 6 output from Base-it1; $C=100$8-means with $C=100$9 clusters, using 10% subsampled data. The intuition is that as HuBERT’s representations improve, clusterings align more closely with true phonetic units.

3. Pre-training and Fine-tuning Regimen

Pre-training uses LibriSpeech-960h, 32 GPUs, an effective batch size of ≈2800 seconds of audio per step, Adam optimizer ($X=[x_1,\dots,x_T]$0, $X=[x_1,\dots,x_T]$1), and linear learning rate warm-up (peak $X=[x_1,\dots,x_T]$2). Masking uses 10-frame spans ($X=[x_1,\dots,x_T]$3200 ms) with 8% of frames as span starts (SpanBERT-style). Iteration 1 employs 250,000 steps; Iteration 2 uses 400,000 steps.

Fine-tuning freezes the convolutional encoder and optimizes the Transformer and CTC head over a 29-token vocabulary (26 English letters, space, apostrophe, blank) using CTC loss. A "freeze-step" hyperparameter determines how long to train only the new softmax layer, before unfreezing the Transformer. Decoding leverages wav2letter++ beam search with LM score fusion.

4. Empirical Results and Ablations

HuBERT-Base demonstrates competitive or superior word error rate (WER) to wav2vec 2.0-Base across all labeled data regimes on LibriSpeech:

Labeled Hours	wav2vec 2.0-Base (test-other WER)	HuBERT-Base (test-other WER)
10 min	15.6	15.3
1 h	11.3	11.3
10 h	9.5	9.4
100 h	8.0	8.1

Ablation studies indicate:

• Best results are achieved with masked-only loss ($X=[x_1,\dots,x_T]$4), not including unmasked frames.
• Teacher cluster quality (evaluated by phone-normalized MI) improves over iterations: 0.25 (MFCC), 0.68 (Base-it1 L6), 0.70 (Base-it2).
• Increasing mask probability or batch size improves performance (batch size scaling 32→128 GPUs yields ≈20% WER reduction).
• Cluster ensembles and longer pre-training further reduce WER.
• Clustering on middle Transformer layers (6–9) yields highest phone purity; iterative pseudo-label refinement improves phonetic alignment.

5. Hungarian BERT (huBERT-Base) as Monolingual LLM

huBERT-Base (Ács et al., 2021) is a monolingual contextualized LLM for Hungarian, architecturally mirroring BERT-Base: 12 Transformer layers, hidden size 768, 12 attention heads, feedforward size 3072, and ≈110M parameters. No modifications are introduced versus BERT-Base aside from trained embeddings and vocabulary.

Tokenization and Pretraining

• Vocabulary: ≈32,000 WordPiece subwords, trained on Hungarian Webcorpus 2.0 (~9B tokens).
• Pretraining objectives: Masked language modeling (15% tokens masked) and next-sentence prediction, replicating Devlin et al. (2018).
• Training regimen: 256 sequences/batch (up to 512 length), learning-rate warm-up and linear decay, ≈1M steps.

Empirical Performance

Freezing model weights, performance on three Hungarian NLP tasks is reported:

• Morphological probing: Last subword representations from middle layers (4–7) achieve >90% accuracy on noun/adjective probes; overall probes ~85–90% (vs. ~80–85% XLM-RoBERTa; ~75–80% mBERT).
• POS tagging (Szeged UD): Layer 6 achieves ~95% accuracy, with final layer at ~93%; weighted layer combination yields ~96%.
• NER (Szeged corpus): Layer 6 F1 ≈92%, layer 12 ≈90%; surpasses multilingual and distilled baselines by 1–4 points.

huBERT-Base consistently outperforms mBERT, XLM-RoBERTa, and other multilingual models in token-level tasks, with the greatest margin in lower–middle layers, reflecting alignment of subword segmentation with Hungarian morphology.

6. Layerwise Insights and Best Practices

Empirical studies show linguistic task optima in layers 4–8: lower layers encode local morphological cues, middle layers capture part-of-speech and phrase structure, higher layers emphasize semantics. For rich morphology languages like Hungarian, last-subword representations capture discriminative suffix information.

Recommended practices:

• For token-level tasks, use last subword representation.
• Prefer layers 4–7 for morphology and POS tagging, layers 6–9 for NER and semantics. Weighted sums aid at the cost of model complexity.
• For fine-tuning, favor higher learning rates for middle layers or gradual unfreezing.

7. Comparative Discussion and Significance

Both HuBERT and huBERT-Base (Hungarian) demonstrate the efficacy of domain-adapted pretraining—either aligning acoustic units for self-supervised speech, or leveraging rich monolingual corpora and tailored subword segmentation for morphologically complex languages. HuBERT-Base for speech reveals that iterative offline clustering and masked prediction converge to high-quality phonetic and word-level representations with a lightweight architecture and high data efficiency, matching or exceeding previous state-of-the-art models. huBERT-Base, as a monolingual BERT-Base trained on a large Hungarian corpus, achieves state-of-the-art performance for Hungarian morphology, POS, and NER, substantiating the value of language-specific vocabulary and curated subwords for inflectionally rich languages (Hsu et al., 2021, Ács et al., 2021).