---
title: 'AV-HuBERT: Audio-Visual Speech Pretraining'
url: https://www.emergentmind.com/topics/audio-visual-hubert-av-hubert
type: topic
---

# AV-HuBERT: Audio-Visual Speech Pretraining

Audio-Visual HuBERT (AV-HuBERT) is a self-supervised pretraining framework for audio-visual speech representation learning that extends HuBERT from audio-only speech to synchronized mouth video and speech audio, with the goal of learning a shared latent representation without transcripts during pretraining and transferring it to lip reading, audio-visual speech recognition (AVSR), and automatic speech recognition (ASR) [2201.02184]. Its defining ingredients are masked multimodal cluster prediction, iterative pseudo-label refinement, modality dropout, and a shared Transformer encoder over temporally aligned audio and visual streams [2201.02184]. Subsequent work has used AV-HuBERT both as a standalone AVSR encoder and as a reusable audiovisual representation model in multilingual recognition, speech enhancement and separation, dysarthric speech reconstruction, speaker verification, target speech extraction, talking-face generation, and multimodal deepfake detection [2201.01763][2502.01547].

## 1. Origins and conceptual role

AV-HuBERT was introduced from the observation that speech is naturally bimodal: the audio signal carries rich phonetic information when conditions are clean, while the visual modality is noise-invariant and can stabilize recognition when audio is degraded by background noise, overlap, channel distortion, or low SNR [2201.02184][2303.12187]. In this formulation, synchronized lip motion and speech acoustics are treated as a strong self-supervised signal for speech representation learning, rather than merely as inputs to a supervised AVSR system [2201.02184].

The framework extends HuBERT by replacing audio-only masked hidden-unit prediction with masked multimodal cluster prediction over synchronized audio and video [2201.02184]. Instead of learning from raw reconstruction or contrastive matching, the model predicts discrete cluster assignments for masked regions, where those cluster targets are discovered automatically and refined iteratively [2201.02184]. The original work argues that audio-visual clusters are better than audio-only or visual-only clusters, and reports cluster-quality gains under phonetic-label, purity, and normalized mutual information analyses, with iterative refinement improving both cluster quality and downstream WER [2201.02184].

This positioning is historically important because earlier AVSR systems were predominantly supervised and therefore constrained by the availability of labeled audiovisual corpora [2201.01763]. AV-HuBERT shifted the emphasis toward large-scale unlabeled synchronized video and audio, making label-efficient audiovisual speech learning feasible on benchmarks such as LRS3 [2201.02184][2201.01763].

## 2. Core architecture and pretraining mechanics

In its original form, AV-HuBERT uses a visual encoder, an audio encoder, a shared Transformer backbone, and a projection head for cluster prediction [2201.02184]. The visual stream is a modified ResNet-18 operating on mouth ROI frames, the audio stream is a simple linear projection of aligned acoustic features, and the two streams are fused by channel-wise concatenation before contextual modeling by the shared Transformer [2201.02184]. The standard temporal alignment uses video sampled at \(25\) Hz and audio represented with \(26\)-dim log filterbank energies, with audio frames stacked to match the video time step [2201.02184].

The multimodal fusion with modality dropout is defined as

\[
\mathbf f^{av}_{t} = \begin{cases}
concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \\
concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \\
concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a)
\end{cases}
\]

where \(p_m\) is the probability that both modalities are used, and \(p_a\) is the probability of retaining audio when only one modality is kept [2201.02184]. This mechanism was introduced to reduce the mismatch between multimodal pretraining and unimodal downstream settings such as lip reading, where only video is available at inference time [2201.02184].

A distinctive component of the masking procedure is visual masking by substitution. Rather than replacing masked visual spans with noise or a learned mask embedding, AV-HuBERT substitutes them with real segments from the same utterance:

\[
\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}
\]

for each masked interval \((s_i,t_i)\), where \(\mathbf I^{v,f}\) is an imposter segment sampled from the same utterance [2201.02184]. The reported motivation is that real same-utterance substitution preserves temporal smoothness and creates a harder, more realistic corruption than simple feature masking or noise injection [2201.02184].

Pseudo-label generation follows the HuBERT-style iterative clustering pipeline. The first iteration clusters MFCC features, while later iterations cluster learned hidden representations from a previous model [2201.02184]. The number of clusters increases over training as

\[
100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000
\]

across five iterations [2201.02184]. The multimodal masked prediction loss is

\[
L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),
\]

with \(\alpha = 0\) in the main experiments, so the loss is evaluated only on masked positions [2201.02184].

The broader encoder template remained influential in later work. Base and Large variants are described elsewhere as using, respectively, 12 Transformer blocks with 768-dimensional embeddings and 24 blocks with 1024-dimensional embeddings [2604.00982]. This continuity made AV-HuBERT a stable backbone for subsequent encoder-side and system-level modifications.

## 3. Fine-tuning, label efficiency, and benchmark performance

AV-HuBERT was introduced as a transferable pretraining method for lip reading and ASR, and its most visible early results came on LRS3 [2201.02184]. On that benchmark, the original paper reports \(32.5\%\) WER for lip reading with only \(30\) hours of labeled data, outperforming a former state-of-the-art system at \(33.6\%\) WER that had been trained with \(31\)K hours of transcribed video data [2201.02184]. With all \(433\) hours of labeled LRS3 data and self-training, the lip-reading WER is further reduced to \(26.9\%\) [2201.02184]. For audio-only ASR on the same benchmark, using AV-HuBERT-derived targets yields \(1.3\%\) WER versus \(2.3\%\) for the previous state of the art, a reported \(40\%\) relative WER reduction [2201.02184].

The low-resource effect is especially pronounced. Using only labeled data, pretraining reduces lip-reading WER from \(98.6\) to \(68.8\) with \(1\) hour of labels, from \(90.8\) to \(57.6\) with \(10\) hours, and from \(77.8\) to \(54.2\) with \(100\) hours [2201.02184]. These results establish AV-HuBERT as a label-efficient audiovisual representation learner rather than merely a strong high-resource recognizer.

The companion robust AVSR study built directly on AV-HuBERT and showed that self-supervised audiovisual pretraining materially changes behavior under acoustic corruption [2201.01763]. On clean LRS3, the best AV-HuBERT model achieves \(1.4\%\) WER, compared with \(2.3\%\) for the prior best model by Ma et al. [2201.01763]. Under babble noise, the gains are much larger. In the \(30\)-hour labeled setting, prior state-of-the-art AVSR by Xu et al. reports \(28.0\%\) average WER, while AV-HuBERT reports \(14.1\%\), a \(49.6\%\) relative reduction achieved with \(30\) hours rather than \(433\) hours of labeled data [2201.01763]. At \(0\) dB babble noise, the reported comparison is \(42.5\%\) for Afouras et al., \(25.5\%\) for Xu et al., and \(8.3\%\) for AV-HuBERT in the \(30\)-hour setup [2201.01763].

The same study reports that adding visual input reduces noisy WER dramatically relative to audio-only systems. Averaged across noise types and pretraining setups, the noisy WER drops from \(42.6\) to \(20.0\) with \(30\) labeled hours and from \(31.3\) to \(9.2\) with \(433\) labeled hours [2201.01763]. The abstract summarizes this as reducing the WER of an audio-based model by over \(75\%\) on average, from \(25.8\%\) to \(5.8\%\) [2201.01763]. Noise-augmented pretraining further improves noisy-test performance, from \(9.3\%\) to \(7.8\%\) in the \(30\)-hour setting and from \(6.9\%\) to \(5.8\%\) in the \(433\)-hour setting [2201.01763].

These results made AV-HuBERT a reference point for two distinct but related reasons: it improved low-resource visual speech learning, and it established a particularly strong robustness profile in babble and overlapping-speech conditions where lip motion helps the system identify the target speaker [2201.02184][2201.01763].

## 4. Unified, multilingual, and encoder-side extensions

Several later systems preserve AV-HuBERT’s encoder role while modifying either the pretraining regime or the downstream fusion strategy. u-HuBERT generalizes AV-HuBERT from audio-visual-only pretraining to unified pretraining on audio-visual, audio-only, and visual-only speech with a shared target space and stronger modality dropout [2207.07036]. In its strongest reported LRS3 setting, a single fine-tuned model yields \(1.2\%\) WER for audio-visual input, \(1.4\%\) for audio input, and \(27.2\%\) for visual input, while supporting zero-shot modality generalization [2207.07036]. AV2vec, by contrast, replaces AV-HuBERT’s offline iterative clustering-and-retraining loop with online multimodal self-distillation from a momentum teacher, and reports that total training time is reduced to less than one-fifth of AV-HuBERT while remaining broadly comparable, with AV2vec-MLM surpassing AV-HuBERT on several downstream settings [2212.02782].

Architectural modifications to the encoder have also been explored. “Conformer enhanced AV-HuBERT” replaces the original Transformer-style backbone with a conformer encoder, uses relative sinusoidal positional encoding, changes the audio front end from \(26\)-dim to \(80\)-dim filterbanks and from a \(25\) ms to a \(15\) ms window, replaces the visual front end with MobileNetV2, and introduces gated fusion instead of simple concatenation [2303.12187]. Relative to baseline AV-HuBERT, the abstract reports \(7\%\) and \(16\%\) relative WER reduction on LRS3 in clean and noisy one-phase evaluation, and the Mandarin configuration reports reductions from \(18.93\) CER to \(14.26\) CER under the fully enhanced conformer backbone [2303.12187].

Encoder-side viseme supervision has produced another line of extension. VisG AV-HuBERT adds a lightweight viseme prediction sub-network during fine-tuning, with a linear projection layer, layer normalization, GELU activation, dropout \(p=0.3\), and a final classification layer predicting \(15\) classes in total: \(14\) viseme categories plus a blank token used by CTC [2604.00982]. The multi-task objective is

\[
L = L_{CE} + \alpha\, L_{CTC},
\]

with \(\alpha = 0.2\) for Base and \(\alpha = 0.15\) for Large [2604.00982]. On LRS3 Speech noise at \(-10\) dB SNR, AV-HuBERT Large reports \(13.59\%\) WER, while VisG AV-HuBERT Large reports \(6.60\%\), a \(51.4\%\) relative improvement; CER under the same condition drops from \(12.14\%\) to \(5.10\%\) [2604.00982]. The paper interprets the largest gains as reductions in substitution errors, suggesting improved speech-unit discrimination under heavy noise [2604.00982].

A separate multilingual development appears in mWhisper-Flamingo, where a multilingual AV-HuBERT pretrained on multilingual videos serves as the visual or lip-reading encoder within a Whisper-based encoder-decoder AVSR system [2502.01547]. In that architecture, AV-HuBERT is not a new algorithm but the video branch supplying visual speech representations to gated cross-attention layers inserted in Whisper’s decoder [2502.01547]. The system introduces decoder modality dropout, with training modes sampled using probabilities \(p_{AV}\), \(p_A\), and \(p_V\), and the best reported setting is

\[
p_{AV} = 0.5,\quad p_A = 0,\quad p_V = 0.5.
\]

Under \(0\) dB babble noise on MuAViC, Whisper Medium fine-tuned reports \(48.0\%\) average non-English WER, whereas mWhisper-Flamingo Medium reports \(43.7\%\), a \(10.6\%\) relative improvement; in the small-model noisy ablation, multilingual AV-HuBERT with decoder modality dropout and fine-tuning of the visual encoder yields \(36.6\%\) average non-English WER, compared with \(37.4\%\) when using English AV-HuBERT [2502.01547]. This supports the claim that multilingual visual pretraining is beneficial for multilingual AVSR [2502.01547].

Taken together, these extensions show that AV-HuBERT has functioned less as a fixed endpoint than as a reusable encoder family. Some work modifies the pretraining target space, some modifies the backbone, and some preserves the encoder while relocating multimodal fusion elsewhere in the system. A plausible implication is that AV-HuBERT’s durability comes from the stability of its audiovisual latent space rather than from any single downstream decoder design.

## 5. Speech enhancement, separation, and reconstruction

AV-HuBERT has also been repurposed for regression-style speech processing tasks. In audio-visual speech enhancement (AVSE) and audio-visual speech separation (AVSS), one influential pattern is to use AV-HuBERT as a pretrained multimodal front end whose layerwise hidden states are aggregated with a trainable weighted sum and then passed to a masking network [2210.17456]. In that formulation, Base AV-HuBERT with \(12\) Transformer layers is pretrained on LRS3 for five iterations, and the downstream regression head uses fully connected layers plus a \(2\)-layer BLSTM [2210.17456]. For AVSE on TSMV, the best result comes from partial fine-tuning, with PESQ \(1.40\) and STOI \(0.68\), compared with \(1.18\) and \(0.60\) for the noisy baseline; for AVSS, partial fine-tuning yields \(3.59\) dB SI-SNR and \(4.59\) dB SDR, while training from scratch yields the best SDR at \(4.63\) dB [2210.17456]. The paper’s principal conclusion is that multimodal self-supervised embeddings from AV-HuBERT generalize beyond categorical recognition tasks to continuous regression problems [2210.17456].

A related CI-oriented AVSE system, SSL-AVSE, uses AV-HuBERT as a Transformer-based SSL audiovisual front end for cochlear implant simulation under limited task-specific data [2307.07748]. Hidden states \(H^l\) from AV-HuBERT are fused by a trainable weighted sum,

\[
H_{WS}=\sum_{l=1}^L w^l H^l,\qquad w^l\ge 0,\qquad \sum_l w^l=1,
\]

then concatenated with noisy log1p spectral features and fed to a BLSTM enhancement model [2307.07748]. Fine-tuning the AV-HuBERT transformer encoder improves PESQ from \(1.434\) to \(1.665\) and STOI from \(0.695\) to \(0.738\); in CI-vocoded speech, NCM improves by \(87.2\%\) relative to the noisy baseline at \(-7\) dB SNR [2307.07748].

AV2Wav moves AV-HuBERT into waveform re-synthesis. It conditions a WaveGrad-style diffusion vocoder on continuous hidden representations from the last layer of a frozen, noise-robust AV-HuBERT Large checkpoint rather than predicting a mask directly [2309.08030]. The paper argues for continuous rather than discrete AV-HuBERT representations because continuous features retain speaker identity and prosody [2309.08030]. In the main AVSE comparison, the masking baseline reports WER \(26.40\) and P-SI-SDR \(13.79\), whereas AV2Wav-23-long-avse reports WER \(16.76\) and P-SI-SDR \(21.21\) [2309.08030].

AV-HuBERT has also been used in dysarthric speech reconstruction. In AVHuBERT-DSR, a pretrained AV-HuBERT Base model replaces a conventional VGG-based audio-visual encoder and provides phoneme embeddings for a variance adaptor, speaker encoder, mel-decoder, and Parallel WaveGAN vocoder [2401.17796]. On UASpeech, the AV-HuBERT-based system reports an average WER reduction of \(8.2\%\) over the audio-only system, an average reduction of \(32.3\%\) versus original dysarthric speech, and a \(42.1\%\) absolute WER reduction for the most severe speaker M12 [2401.17796]. The gains are reported as increasing with dysarthria severity, which suggests that the visual modality becomes progressively more valuable as acoustic content extraction becomes less reliable [2401.17796].

In target speech extraction, AVHuMAR-TSE integrates pretrained AV-HuBERT into the cue encoder of a time-domain extraction system and augments it with a Mask-And-Recover self-supervised auxiliary objective [2403.16078]. The final model reports SI-SDR \(12.331\), SI-SDRi \(12.379\), SDR \(12.726\), PESQ \(2.922\), and STOI \(0.887\) on VoxCeleb2, improving over MuSE and over the version without MAR [2403.16078]. Here AV-HuBERT functions as a visual-cue refiner that improves synchronization between lip motion and the extracted speech representation [2403.16078].

## 6. Generative, biometric, and forensic uses

Outside core recognition and enhancement, AV-HuBERT has been adopted as a stable audiovisual expert for generation and evaluation. In talking-face generation, a finetuned lip-reading AV-HuBERT model is used both to define a lip synchronization loss and to derive three evaluation metrics, \(\text{AVS}_u\), \(\text{AVS}_m\), and \(\text{AVS}_v\) [2405.04327]. The training loss uses final-layer audio and visual features \(F^A_{AVH}, F^V_{AVH} \in \mathbb{R}^{T \times 768}\) and computes synchronization only on the generated interval:

\[
L_{sync} = -log(CS(F^{A_{t:t+k}_{AVH}, F^{V_{t:t+k}_{AVH})).
\]

On LRS2, the unsupervised AV-HuBERT-based supervision variant reports SSIM \(0.947\), PSNR \(31.273\), FID \(4.51\), LMD \(1.188\), \(\text{AVS}_u = 0.508\), \(\text{AVS}_m = 0.939\), and \(\text{AVS}_v = 0.879\), outperforming the SyncNet-based baseline [2405.04327]. The paper attributes this to AV-HuBERT features being more stable, less fluctuation-prone on ground-truth audio-lip pairs, and more robust to spatial shifts and affine transformations than SyncNet [2405.04327].

For speaker representation learning, AV-HuBERT has been used as a self-supervised backbone for lip-based audio-visual speaker embeddings [2205.07180]. With roughly \(2{,}800\) hours of unlabeled pretraining data from LRS3 and VoxCeleb2, the reported conclusion is roughly ten-fold label-efficiency improvement for both audio-only and audio-visual speaker verification [2205.07180]. In clean conditions, incorporating visual information reduces EER by \(38\%\), and in noisy conditions by \(75\%\); for example, under full VC2 fine-tuning the audio-only AV-HuBERT-B model reports \(5.1\%\) EER on VC2 clean while the audio-visual version reports \(2.4\%\), and on noisy VC1 average EER the AV-HuBERT-L comparison is \(27.8\%\) audio-only versus \(4.7\%\) audio-visual [2205.07180].

In multimodal deepfake detection, AV-Lip-Sync+ uses AV-HuBERT as a transformer-based self-supervised audio-visual feature extractor for lip, audio, and joint audiovisual embeddings, then combines those with a synchronization-difference representation and a multi-scale temporal convolutional network [2311.02733]. The synchronization cue is constructed as

\[
\vec F_{sync} = \{|{\vec F_{vi} - {\vec F_{ai}|\}_{i=1}^T,
\]

and the fused representation is classified with binary cross-entropy [2311.02733]. On FakeAVCeleb, AV-Lip-Sync+ reports accuracies of \(0.91\) on Faceswap, \(0.99\) on Faceswap\_wav2lip, \(0.97\) on RTVC, \(0.99\) on Wav2lip, and \(0.99\) on Test-set-2, with AUC values all above \(0.96\); on DeepfakeTIMIT, AV-Lip-Sync+ with a face encoder reports \(0.9996\) AUC on LQ and \(0.9998\) on HQ [2311.02733]. In this setting, AV-HuBERT is valued less as a recognizer than as a detector of natural audio-visual consistency.

These applications illustrate a broad shift in how AV-HuBERT is used. Rather than being confined to transcript prediction, it increasingly functions as an audiovisual prior: a learned representation space in which synchronization, speaker identity, speech content, and noise robustness can be exploited by downstream systems.

## 7. Internal representations, bio-fidelity, and limitations

Recent interpretability work has examined what AV-HuBERT actually encodes. One study of viseme representations uses t-SNE and probing on a base AV-HuBERT model fine-tuned on the \(433\)-hour split of LRS3 and reports that final viseme classification accuracy rises from \(69.45\%\) for video-only input to \(84.82\%\) for noisy audio-visual input and \(93.3\%\) for clean audio-visual input [2509.16023]. The same analysis reports that visually distinct consonant visemes such as **F**, **W**, **CH**, and **P** form clear clusters even in video-only conditions, whereas ambiguous or under-represented visemes such as **ER** and **K** benefit strongly from audio [2509.16023]. The authors’ interpretation is that vision supplies coarse viseme grouping while audio refines those groups with phonemic detail [2509.16023]. This suggests that AV-HuBERT’s multimodal hierarchy is not a simple averaging of modalities, but an asymmetric refinement process in which the utility of audio depends strongly on visual salience and class frequency.

A different line of work evaluates AV-HuBERT as a model of human multisensory speech perception. In a McGurk-style comparison between AV-HuBERT Large and \(N=44\) naïve adult Spanish speakers, the model shows a McGurk-like effect under incongruent auditory \(/ba/\) and visual \(/ga/\) input, producing fused \(/da/\) responses on \(17/25\) trials, or \(68.0\%\), and auditory-consistent \(/ba/\) responses on \(8/25\) trials, or \(32.0\%\) [2601.15869]. Humans report fused \(/da/\) in \(47.7\%\) of cases and auditory-consistent \(/ba/\) in \(31.8\%\) [2601.15869]. The paper characterizes the near-match in auditory dominance (\(32.0\%\) versus \(31.8\%\)) as a “striking quantitative isomorphism,” but emphasizes that AV-HuBERT over-fuses relative to humans and lacks their perceptual stochasticity and response diversity [2601.15869].

Temporal analysis raises a different limitation. A study comparing AV-HuBERT with audio-only HuBERT tracks the time course of phonetic decodability and finds that phoneme information becomes available in AV-HuBERT embeddings only about \(20\) ms before HuBERT, despite natural audiovisual speech often exhibiting a lip lead of \(100\)–\(300\) ms [2506.20361]. The proposed explanation is architectural: AV-HuBERT stacks every four \(100\) Hz audio frames and concatenates them with one \(25\) fps visual frame, effectively reducing the temporal resolution to \(40\) ms and yielding an expected shift of about \(20\) ms [2506.20361]. The same work concludes that AV-HuBERT does not adequately capture the temporal dynamics of multimodal speech perception and remains largely audio-dominated for timing-sensitive questions [2506.20361].

These interpretability results complicate a common misconception. AV-HuBERT is often taken to be a general model of human audiovisual speech perception because it performs well on AVSR and can reproduce some qualitative multisensory phenomena. The evidence is narrower. It encodes meaningful visual speech structure, benefits substantially from audiovisual integration, and can mimic some behavioral outcomes, but it also exhibits deterministic categorical biases and limited sensitivity to the natural temporal asynchrony between lip movements and voicing [2509.16023][2601.15869][2506.20361]. A plausible implication is that AV-HuBERT is a strong engineering model of audiovisual speech representation, but an incomplete mechanistic model of biological multisensory speech perception.

Source: https://www.emergentmind.com/topics/audio-visual-hubert-av-hubert