Masked Cluster Prediction: HuBERT & WavLM

Updated 14 May 2026

Masked Cluster Prediction is a self-supervised objective using unsupervised clustering and selective masking to learn high-level representations from raw audio and multimodal inputs.
It leverages iterative clustering and Transformer architectures to refine pseudo-label targets, enhancing performance in ASR and paralinguistic tasks.
Models like HuBERT, WavLM, and AV-HuBERT demonstrate its versatility across audio, visual, and sign language applications.

Masked Cluster Prediction (HuBERT, WavLM)

Masked cluster prediction is a foundational self-supervised pre-training objective used in HuBERT, WavLM, and their multimodal or multi-stream extensions. It enables neural networks to learn contextualized, high-level representations of speech or other modalities by predicting quantized or clustered units (pseudo-labels) for masked input segments, without requiring manual annotations. This approach has been widely adopted for speech, audio-visual, and sign language modeling, providing a flexible, generalizable pretext task that supports both content-based and paralinguistic downstream applications.

1. Principle and Workflow of Masked Cluster Prediction

Masked cluster prediction is defined by two core components: the generation of discrete target units (typically via unsupervised clustering) and a masked prediction objective applied only to selected regions of a continuous input sequence.

Discrete unit generation:

Given a sequence of raw acoustic features (e.g., MFCCs, mel-filterbanks, or learned neural embeddings), k-means clustering is performed to generate a finite vocabulary of cluster centroids. Each frame $t$ is assigned to its nearest centroid, forming a pseudo-label $z_t \in \{1, \ldots, K\}$ , with $K$ as the cluster count. For example, HuBERT employs 100 clusters on MFCCs in the first iteration, and 500 clusters on intermediate Transformer features in subsequent iterations, refining targets as the model improves (Hsu et al., 2021). WavLM, AV-HuBERT, and u-HuBERT follow analogous iterative, staged target refinement (Chen et al., 2021, Shi et al., 2022, Hsu et al., 2022).

Masking strategy:

A fraction of time indices in the input sequence is selected for masking (e.g., 8–25% of frames), typically in contiguous spans of fixed length (e.g., 10 frames, or 400 ms). The corresponding input representations are replaced by a trainable [MASK] token or zero vector. In multimodal settings (AV-HuBERT, u-HuBERT), masking can be applied to each modality independently or jointly.

Prediction head and loss:

The model processes the corrupted input sequence through a Transformer stack (or equivalent), then predicts the discrete cluster assignment for each masked time step via a softmax-linear head. The training loss is the sum (or average) of cross-entropy errors over the masked positions:

$\mathcal{L} = -\sum_{t \in M} \log p_t(y_t) \,,$

where $M$ indexes masked frames and $y_t$ is the cluster label.

This iterative, BERT-inspired procedure repeatedly alternates between cluster assignment (target refinement) and masked prediction, progressively improving the semantic quality of the learned clusters and thus the representations (Hsu et al., 2021, Chen et al., 2021, Shi et al., 2022).

HuBERT and WavLM:

The canonical architecture combines a convolutional frame encoder, a multi-layer Transformer, and a softmax cluster prediction head. WavLM enhances this baseline with denoising objectives (training on corrupted/noisy inputs while still predicting clean targets) and a gated relative position bias in its Transformer layers, improving both content and paralinguistic representation (Chen et al., 2021).

AV-HuBERT and u-HuBERT:

Audio-visual and multimodal masked cluster prediction models process parallel streams—audio and video (lip images)—pre-processing each with appropriate encoders (e.g., mel-filterbanks for audio, ResNet-18 for video), fusing them at the feature level, and passing through a shared Transformer. Cluster targets are derived from concatenated, fused embeddings, and the loss enforces consistent cluster prediction even under modality dropout, enabling robust zero-shot modality transfer (Shi et al., 2022, Hsu et al., 2022).

SHuBERT:

In sign language, SHuBERT extends masked cluster prediction to four parallel visual streams (face, left hand, right hand, body pose). Each stream is clustered independently, and masking is performed per stream, with cross-entropy loss summed over all masked positions and streams (Gueuwou et al., 2024).

Target design:

Initial cluster assignments can be generated from raw features (MFCCs, mel-filterbanks, learned CNN outputs), or from pre-trained model hidden states (deeper Transformer layers for more abstract units). Key design variables include:

Number of clusters $K$ : Large $K$ (up to 25,000) encodes finer-grained phonetic or acoustic information; small $K$ captures high-level abstractions (Chen et al., 2024).
Clustering layer: Lower layers carry more prosodic or speaker information, higher layers offer more phonetic abstraction (Chen et al., 2024).
Hierarchical quantization: Residual Vector Quantization (RVQ) allows compositional, multi-level discrete targets without requiring excessively large $K$ per level (Chen et al., 2024, Yadav et al., 2024).
Phonetic-grounded or context-dependent units: Alternative strategies, including phoneme tokens (MonoBERT), context-dependent tied-state assignments (PolyBERT), or triphones, improve performance especially for ASR (Ma et al., 2023).

Iterative clustering:

A recurring motif is iterative reclustering: after each round of masked prediction training, new targets are generated by clustering hidden representations from an intermediate Transformer layer of the current model. This tightens the alignment of learned units with phonetic and linguistic content, as evidenced by sharp reductions in word/phoneme error rates (PER, WER) across multiple iterations (Hsu et al., 2021, Chen et al., 2021, Shi et al., 2022).

Unit quality and ablation:

Higher fidelity in cluster assignment (either by using low-noise phoneme units or context-dependent states) consistently improves downstream ASR and other content-based tasks. Overly fine clusterings, or clustering on non-semantic (random) features, can degrade certain paralinguistic tasks (Ma et al., 2023, Chen et al., 2024).

4. Multimodal, Multi-Stream, and Hierarchical Extensions

Audio-visual (AV-HuBERT, u-HuBERT):

In these frameworks, masked cluster prediction is unified across audio and video modalities. Targets are derived from fused feature representations, and the shared Transformer backbone is required to predict cluster assignments irrespective of which modalities are present. Modality dropout, a core innovation, forces the network to learn cluster-predictive mappings for any combination of modalities, enabling robust zero-shot transfer across input domains (audio-only, visual-only, audio-visual) (Hsu et al., 2022, Shi et al., 2022).

Sign language (SHuBERT):

SHuBERT adapts masked cluster prediction to heterogeneous visual input: each channel (face, left hand, right hand, body pose) is clustered and masked independently, and the network is trained to recover cluster labels per stream. The design allows the model to capture both spatial and temporal dependencies relevant for sign language recognition and translation (Gueuwou et al., 2024).

Multi-resolution (MR-HuBERT):

MR-HuBERT predicts masked units at multiple temporal granularities simultaneously (e.g., 20 ms and 40 ms frame rates) by embedding a hierarchical Transformer structure and dual prediction heads. This architecture enhances robustness and efficiency by capturing both fine and coarse semantic patterns (Shi et al., 2023).

5. Variants, Improvements, and Comparative Analysis

WavLM innovations:

Incorporates an implicit denoising objective by requiring the model to predict cluster targets generated from "clean" utterances while operating on corrupted input (simulating noise/overlap). Gated relative position bias is used to enhance self-attention, adapting the receptive field dynamically based on content, which benefits both ASR and non-ASR tasks (Chen et al., 2021).

MS-HuBERT:

Addresses pre-training/inference distribution mismatch via a "Swap" method: at every Transformer layer, masked and clean feature streams exchange representations at masked positions. A multi-cluster masked prediction loss enforces accurate prediction over several concurrent clusterings at different granularities, leading to consistent improvements in WER and content encoding (Yadav et al., 2024).

Alternative cluster anchoring (GMM-JEPA):

GMM-Anchored JEPA diverges from hard, iterative k-means by fitting a single Gaussian Mixture Model (GMM) on log-mel features, using soft posteriors as auxiliary targets throughout training with a decaying KL-regularization. This yields more uniform cluster utilization and improved performance on ASR and paralinguistic tasks compared to WavLM-style hard-cluster baselines (Ioannides et al., 30 Jan 2026).

Online clustering and self-distillation (SpidR):

SpidR eliminates offline k-means iterations, instead adopting an online codebook maintained per layer and learning via a student–teacher self-distillation regime. This approach accelerates pre-training and improves codebook stability and linguistic fidelity relative to HuBERT and WavLM (Poli et al., 23 Dec 2025).

Model/Method	Target Generation	Cluster Assignment	Multimodal/Multistream
HuBERT	Offline iterative k-means	Hard, per-frame	Audio only
WavLM	Same as HuBERT	Hard, per-frame	Audio only, with denoising
AV-HuBERT	Iterative, fused features	Hard, fused	Audio + video
u-HuBERT	Unified k-means, fused	Hard, shared space	Audio, video, mixed, dropout
SHuBERT	Per-stream k-means	Hard, per stream	4 visual channels for sign
GMM-JEPA	1-time GMM	Soft, probabilistic	Audio only
SpidR	Online codebook (per layer)	Hard, per-frame	Audio only

6. Zero-Shot Generalization and Downstream Impact

The unified masked cluster prediction framework, especially as realized in u-HuBERT and AV-HuBERT, enables a single model to accept audio, video, or audio-visual input and to generalize zero-shot to modalities omitted during fine-tuning. For instance, a u-HuBERT model fine-tuned on audio-only transcripts achieves 31.6% WER on previously unseen visual-only input, compared to 97% for a model without modality dropout (Hsu et al., 2022). These models match or exceed the performance of specialized, modality-specific baselines, establishing masked cluster prediction as the method of choice for universal speech foundation models capable of cross-modal and content-agnostic representation transfer.

Empirical results across ASR, keyword spotting, semantic tasks, enhancement, and speaker identification show that appropriate choice of cluster granularity, multitarget heads, and denoising augmentation allows these models to excel across content and paralinguistic benchmarks (Chen et al., 2024, Shi et al., 2023, Lin et al., 2023). Fine-grained clusters drive gains in phone/word error rate, but may affect speaker recognition if too detailed; multitarget clustering across layers or RVQ levels is a practical solution for capturing both content and paralinguistic cues (Chen et al., 2024, Yadav et al., 2024).

7. Limitations, Best Practices, and Future Directions

Despite their successes, masked cluster prediction models have intrinsic limitations:

The quality of discovered discrete units is dependent on feature choices and clustering granularity. Poor initialization or excessive cluster fineness can constrain generalization or compressivity (Chen et al., 2024, Ma et al., 2023).
Speaker and paralinguistic features are not directly modeled by the masked cluster prediction loss; auxiliary loss terms or data augmentation (as in WavLM) are needed to robustly encode these attributes (Yadav et al., 2023).
Distribution mismatch between pre-training (frequent masking) and inference (no masking) can degrade performance; mechanisms like "Swap" or multi-view consistency help bridge this gap (Yadav et al., 2024).
Best practice recommendations include tuning the number of clusters for task focus (content, denoising, speaker), choosing clustering layers appropriate for target abstraction, and using multitarget or hierarchical cluster prediction when computationally feasible (Chen et al., 2024).

Emerging approaches leverage soft, probabilistic cluster assignments to mitigate mode-collapse and improve cluster entropy (Ioannides et al., 30 Jan 2026), and online codebook adaptation for efficient, stable unit discovery (Poli et al., 23 Dec 2025). Unsupervised cluster correction (e.g., to normalize accent) or multi-resolution clustering further expand the applicability of masked cluster prediction frameworks (Poncelet et al., 2023, Shi et al., 2023).

Masked cluster prediction—and its advances across model architectures and modalities—continues to underpin state-of-the-art self-supervised learning in speech, audio-visual, and sign language processing, supporting robust, versatile, and transferable representation learning (Hsu et al., 2021, Chen et al., 2021, Shi et al., 2022, Hsu et al., 2022, Gueuwou et al., 2024, Shi et al., 2023, Ioannides et al., 30 Jan 2026, Poli et al., 23 Dec 2025, Yadav et al., 2024).