AV-HuBERT: Audio-Visual Speech Pretraining

• AV-HuBERT is a self-supervised pretraining framework that learns shared audio-visual speech representations through masked multimodal cluster prediction and iterative pseudo-label refinement.
• It employs a shared Transformer encoder along with dedicated visual and audio encoders and modality dropout to effectively merge lip motion and acoustic cues, enhancing performance in low-resource and noisy settings.
• AV-HuBERT’s flexible architecture has led to diverse applications including lip reading, automatic speech recognition, speech enhancement, speaker verification, and deepfake detection.

Audio-Visual HuBERT (AV-HuBERT) is a self-supervised pretraining framework for audio-visual speech representation learning that extends HuBERT from audio-only speech to synchronized mouth video and speech audio, with the goal of learning a shared latent representation without transcripts during pretraining and transferring it to lip reading, audio-visual speech recognition (AVSR), and automatic speech recognition (ASR) (Shi et al., 2022). Its defining ingredients are masked multimodal cluster prediction, iterative pseudo-label refinement, modality dropout, and a shared Transformer encoder over temporally aligned audio and visual streams (Shi et al., 2022). Subsequent work has used AV-HuBERT both as a standalone AVSR encoder and as a reusable audiovisual representation model in multilingual recognition, speech enhancement and separation, dysarthric speech reconstruction, speaker verification, target speech extraction, talking-face generation, and multimodal deepfake detection (Shi et al., 2022, Rouditchenko et al., 3 Feb 2025).

1. Origins and conceptual role

AV-HuBERT was introduced from the observation that speech is naturally bimodal: the audio signal carries rich phonetic information when conditions are clean, while the visual modality is noise-invariant and can stabilize recognition when audio is degraded by background noise, overlap, channel distortion, or low SNR (Shi et al., 2022, Ren et al., 2023). In this formulation, synchronized lip motion and speech acoustics are treated as a strong self-supervised signal for speech representation learning, rather than merely as inputs to a supervised AVSR system (Shi et al., 2022).

The framework extends HuBERT by replacing audio-only masked hidden-unit prediction with masked multimodal cluster prediction over synchronized audio and video (Shi et al., 2022). Instead of learning from raw reconstruction or contrastive matching, the model predicts discrete cluster assignments for masked regions, where those cluster targets are discovered automatically and refined iteratively (Shi et al., 2022). The original work argues that audio-visual clusters are better than audio-only or visual-only clusters, and reports cluster-quality gains under phonetic-label, purity, and normalized mutual information analyses, with iterative refinement improving both cluster quality and downstream WER (Shi et al., 2022).

This positioning is historically important because earlier AVSR systems were predominantly supervised and therefore constrained by the availability of labeled audiovisual corpora (Shi et al., 2022). AV-HuBERT shifted the emphasis toward large-scale unlabeled synchronized video and audio, making label-efficient audiovisual speech learning feasible on benchmarks such as LRS3 (Shi et al., 2022, Shi et al., 2022).

2. Core architecture and pretraining mechanics

In its original form, AV-HuBERT uses a visual encoder, an audio encoder, a shared Transformer backbone, and a projection head for cluster prediction (Shi et al., 2022). The visual stream is a modified ResNet-18 operating on mouth ROI frames, the audio stream is a simple linear projection of aligned acoustic features, and the two streams are fused by channel-wise concatenation before contextual modeling by the shared Transformer (Shi et al., 2022). The standard temporal alignment uses video sampled at $25$ Hz and audio represented with $26$-dim log filterbank energies, with audio frames stacked to match the video time step (Shi et al., 2022).

The multimodal fusion with modality dropout is defined as

$$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$

where $p_m$ is the probability that both modalities are used, and $p_a$ is the probability of retaining audio when only one modality is kept (Shi et al., 2022). This mechanism was introduced to reduce the mismatch between multimodal pretraining and unimodal downstream settings such as lip reading, where only video is available at inference time (Shi et al., 2022).

A distinctive component of the masking procedure is visual masking by substitution. Rather than replacing masked visual spans with noise or a learned mask embedding, AV-HuBERT substitutes them with real segments from the same utterance:

$$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$

for each masked interval $(s_i,t_i)$, where $\mathbf I^{v,f}$ is an imposter segment sampled from the same utterance (Shi et al., 2022). The reported motivation is that real same-utterance substitution preserves temporal smoothness and creates a harder, more realistic corruption than simple feature masking or noise injection (Shi et al., 2022).

Pseudo-label generation follows the HuBERT-style iterative clustering pipeline. The first iteration clusters MFCC features, while later iterations cluster learned hidden representations from a previous model (Shi et al., 2022). The number of clusters increases over training as

$$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$

across five iterations (Shi et al., 2022). The multimodal masked prediction loss is

$$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$

with $26$0 in the main experiments, so the loss is evaluated only on masked positions (Shi et al., 2022).

The broader encoder template remained influential in later work. Base and Large variants are described elsewhere as using, respectively, 12 Transformer blocks with 768-dimensional embeddings and 24 blocks with 1024-dimensional embeddings (Papadopoulos et al., 1 Apr 2026). This continuity made AV-HuBERT a stable backbone for subsequent encoder-side and system-level modifications.

3. Fine-tuning, label efficiency, and benchmark performance

AV-HuBERT was introduced as a transferable pretraining method for lip reading and ASR, and its most visible early results came on LRS3 (Shi et al., 2022). On that benchmark, the original paper reports $26$1 WER for lip reading with only $26$2 hours of labeled data, outperforming a former state-of-the-art system at $26$3 WER that had been trained with $26$4K hours of transcribed video data (Shi et al., 2022). With all $26$5 hours of labeled LRS3 data and self-training, the lip-reading WER is further reduced to $26$6 (Shi et al., 2022). For audio-only ASR on the same benchmark, using AV-HuBERT-derived targets yields $26$7 WER versus $26$8 for the previous state of the art, a reported $26$9 relative WER reduction (Shi et al., 2022).

The low-resource effect is especially pronounced. Using only labeled data, pretraining reduces lip-reading WER from $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$0 to $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$1 with $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$2 hour of labels, from $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$3 to $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$4 with $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$5 hours, and from $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$6 to $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$7 with $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$8 hours (Shi et al., 2022). These results establish AV-HuBERT as a label-efficient audiovisual representation learner rather than merely a strong high-resource recognizer.

The companion robust AVSR study built directly on AV-HuBERT and showed that self-supervised audiovisual pretraining materially changes behavior under acoustic corruption (Shi et al., 2022). On clean LRS3, the best AV-HuBERT model achieves $$\mathbf f^{av}_{t} = \begin{cases} concat(\mathbf f^a_t, \mathbf f^v_t) & \text{with } p_m \ concat(\mathbf f^a_t, \mathbf 0) & \text{with } (1-p_m)p_a \ concat(\mathbf 0, \mathbf f^v_t) & \text{with } (1-p_m)(1-p_a) \end{cases}$$9 WER, compared with $p_m$0 for the prior best model by Ma et al. (Shi et al., 2022). Under babble noise, the gains are much larger. In the $p_m$1-hour labeled setting, prior state-of-the-art AVSR by Xu et al. reports $p_m$2 average WER, while AV-HuBERT reports $p_m$3, a $p_m$4 relative reduction achieved with $p_m$5 hours rather than $p_m$6 hours of labeled data (Shi et al., 2022). At $p_m$7 dB babble noise, the reported comparison is $p_m$8 for Afouras et al., $p_m$9 for Xu et al., and $p_a$0 for AV-HuBERT in the $p_a$1-hour setup (Shi et al., 2022).

The same study reports that adding visual input reduces noisy WER dramatically relative to audio-only systems. Averaged across noise types and pretraining setups, the noisy WER drops from $p_a$2 to $p_a$3 with $p_a$4 labeled hours and from $p_a$5 to $p_a$6 with $p_a$7 labeled hours (Shi et al., 2022). The abstract summarizes this as reducing the WER of an audio-based model by over $p_a$8 on average, from $p_a$9 to $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$0 (Shi et al., 2022). Noise-augmented pretraining further improves noisy-test performance, from $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$1 to $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$2 in the $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$3-hour setting and from $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$4 to $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$5 in the $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$6-hour setting (Shi et al., 2022).

These results made AV-HuBERT a reference point for two distinct but related reasons: it improved low-resource visual speech learning, and it established a particularly strong robustness profile in babble and overlapping-speech conditions where lip motion helps the system identify the target speaker (Shi et al., 2022, Shi et al., 2022).

4. Unified, multilingual, and encoder-side extensions

Several later systems preserve AV-HuBERT’s encoder role while modifying either the pretraining regime or the downstream fusion strategy. u-HuBERT generalizes AV-HuBERT from audio-visual-only pretraining to unified pretraining on audio-visual, audio-only, and visual-only speech with a shared target space and stronger modality dropout (Hsu et al., 2022). In its strongest reported LRS3 setting, a single fine-tuned model yields $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$7 WER for audio-visual input, $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$8 for audio input, and $$\tilde{\mathbf I}^{v}_{s_i:t_i} = \mathbf I^{v,f}_{p_i : p_i+t_i-s_i}$$9 for visual input, while supporting zero-shot modality generalization (Hsu et al., 2022). AV2vec, by contrast, replaces AV-HuBERT’s offline iterative clustering-and-retraining loop with online multimodal self-distillation from a momentum teacher, and reports that total training time is reduced to less than one-fifth of AV-HuBERT while remaining broadly comparable, with AV2vec-MLM surpassing AV-HuBERT on several downstream settings (Zhang et al., 2022).

Architectural modifications to the encoder have also been explored. “Conformer enhanced AV-HuBERT” replaces the original Transformer-style backbone with a conformer encoder, uses relative sinusoidal positional encoding, changes the audio front end from $(s_i,t_i)$0-dim to $(s_i,t_i)$1-dim filterbanks and from a $(s_i,t_i)$2 ms to a $(s_i,t_i)$3 ms window, replaces the visual front end with MobileNetV2, and introduces gated fusion instead of simple concatenation (Ren et al., 2023). Relative to baseline AV-HuBERT, the abstract reports $(s_i,t_i)$4 and $(s_i,t_i)$5 relative WER reduction on LRS3 in clean and noisy one-phase evaluation, and the Mandarin configuration reports reductions from $(s_i,t_i)$6 CER to $(s_i,t_i)$7 CER under the fully enhanced conformer backbone (Ren et al., 2023).

Encoder-side viseme supervision has produced another line of extension. VisG AV-HuBERT adds a lightweight viseme prediction sub-network during fine-tuning, with a linear projection layer, layer normalization, GELU activation, dropout $(s_i,t_i)$8, and a final classification layer predicting $(s_i,t_i)$9 classes in total: $\mathbf I^{v,f}$0 viseme categories plus a blank token used by CTC (Papadopoulos et al., 1 Apr 2026). The multi-task objective is

$\mathbf I^{v,f}$1

with $\mathbf I^{v,f}$2 for Base and $\mathbf I^{v,f}$3 for Large (Papadopoulos et al., 1 Apr 2026). On LRS3 Speech noise at $\mathbf I^{v,f}$4 dB SNR, AV-HuBERT Large reports $\mathbf I^{v,f}$5 WER, while VisG AV-HuBERT Large reports $\mathbf I^{v,f}$6, a $\mathbf I^{v,f}$7 relative improvement; CER under the same condition drops from $\mathbf I^{v,f}$8 to $\mathbf I^{v,f}$9 (Papadopoulos et al., 1 Apr 2026). The paper interprets the largest gains as reductions in substitution errors, suggesting improved speech-unit discrimination under heavy noise (Papadopoulos et al., 1 Apr 2026).

A separate multilingual development appears in mWhisper-Flamingo, where a multilingual AV-HuBERT pretrained on multilingual videos serves as the visual or lip-reading encoder within a Whisper-based encoder-decoder AVSR system (Rouditchenko et al., 3 Feb 2025). In that architecture, AV-HuBERT is not a new algorithm but the video branch supplying visual speech representations to gated cross-attention layers inserted in Whisper’s decoder (Rouditchenko et al., 3 Feb 2025). The system introduces decoder modality dropout, with training modes sampled using probabilities $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$0, $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$1, and $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$2, and the best reported setting is

$$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$3

Under $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$4 dB babble noise on MuAViC, Whisper Medium fine-tuned reports $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$5 average non-English WER, whereas mWhisper-Flamingo Medium reports $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$6, a $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$7 relative improvement; in the small-model noisy ablation, multilingual AV-HuBERT with decoder modality dropout and fine-tuning of the visual encoder yields $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$8 average non-English WER, compared with $$100 \rightarrow 100 \rightarrow 500 \rightarrow 1000 \rightarrow 2000$$9 when using English AV-HuBERT (Rouditchenko et al., 3 Feb 2025). This supports the claim that multilingual visual pretraining is beneficial for multilingual AVSR (Rouditchenko et al., 3 Feb 2025).

Taken together, these extensions show that AV-HuBERT has functioned less as a fixed endpoint than as a reusable encoder family. Some work modifies the pretraining target space, some modifies the backbone, and some preserves the encoder while relocating multimodal fusion elsewhere in the system. A plausible implication is that AV-HuBERT’s durability comes from the stability of its audiovisual latent space rather than from any single downstream decoder design.

5. Speech enhancement, separation, and reconstruction

AV-HuBERT has also been repurposed for regression-style speech processing tasks. In audio-visual speech enhancement (AVSE) and audio-visual speech separation (AVSS), one influential pattern is to use AV-HuBERT as a pretrained multimodal front end whose layerwise hidden states are aggregated with a trainable weighted sum and then passed to a masking network (Chern et al., 2022). In that formulation, Base AV-HuBERT with $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$0 Transformer layers is pretrained on LRS3 for five iterations, and the downstream regression head uses fully connected layers plus a $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$1-layer BLSTM (Chern et al., 2022). For AVSE on TSMV, the best result comes from partial fine-tuning, with PESQ $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$2 and STOI $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$3, compared with $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$4 and $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$5 for the noisy baseline; for AVSS, partial fine-tuning yields $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$6 dB SI-SNR and $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$7 dB SDR, while training from scratch yields the best SDR at $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$8 dB (Chern et al., 2022). The paper’s principal conclusion is that multimodal self-supervised embeddings from AV-HuBERT generalize beyond categorical recognition tasks to continuous regression problems (Chern et al., 2022).

A related CI-oriented AVSE system, SSL-AVSE, uses AV-HuBERT as a Transformer-based SSL audiovisual front end for cochlear implant simulation under limited task-specific data (Lai et al., 2023). Hidden states $$L = -\sum_{t \in M^a \cup M^v} \log p_t(z_t) - \alpha \sum_{t \notin M^a \cup M^v} \log p_t(z_t),$$9 from AV-HuBERT are fused by a trainable weighted sum,

$26$00

then concatenated with noisy log1p spectral features and fed to a BLSTM enhancement model (Lai et al., 2023). Fine-tuning the AV-HuBERT transformer encoder improves PESQ from $26$01 to $26$02 and STOI from $26$03 to $26$04; in CI-vocoded speech, NCM improves by $26$05 relative to the noisy baseline at $26$06 dB SNR (Lai et al., 2023).

AV2Wav moves AV-HuBERT into waveform re-synthesis. It conditions a WaveGrad-style diffusion vocoder on continuous hidden representations from the last layer of a frozen, noise-robust AV-HuBERT Large checkpoint rather than predicting a mask directly (Chou et al., 2023). The paper argues for continuous rather than discrete AV-HuBERT representations because continuous features retain speaker identity and prosody (Chou et al., 2023). In the main AVSE comparison, the masking baseline reports WER $26$07 and P-SI-SDR $26$08, whereas AV2Wav-23-long-avse reports WER $26$09 and P-SI-SDR $26$10 (Chou et al., 2023).

AV-HuBERT has also been used in dysarthric speech reconstruction. In AVHuBERT-DSR, a pretrained AV-HuBERT Base model replaces a conventional VGG-based audio-visual encoder and provides phoneme embeddings for a variance adaptor, speaker encoder, mel-decoder, and Parallel WaveGAN vocoder (Chen et al., 2024). On UASpeech, the AV-HuBERT-based system reports an average WER reduction of $26$11 over the audio-only system, an average reduction of $26$12 versus original dysarthric speech, and a $26$13 absolute WER reduction for the most severe speaker M12 (Chen et al., 2024). The gains are reported as increasing with dysarthria severity, which suggests that the visual modality becomes progressively more valuable as acoustic content extraction becomes less reliable (Chen et al., 2024).

In target speech extraction, AVHuMAR-TSE integrates pretrained AV-HuBERT into the cue encoder of a time-domain extraction system and augments it with a Mask-And-Recover self-supervised auxiliary objective (Wu et al., 2024). The final model reports SI-SDR $26$14, SI-SDRi $26$15, SDR $26$16, PESQ $26$17, and STOI $26$18 on VoxCeleb2, improving over MuSE and over the version without MAR (Wu et al., 2024). Here AV-HuBERT functions as a visual-cue refiner that improves synchronization between lip motion and the extracted speech representation (Wu et al., 2024).

6. Generative, biometric, and forensic uses

Outside core recognition and enhancement, AV-HuBERT has been adopted as a stable audiovisual expert for generation and evaluation. In talking-face generation, a finetuned lip-reading AV-HuBERT model is used both to define a lip synchronization loss and to derive three evaluation metrics, $26$19, $26$20, and $26$21 (Yaman et al., 2024). The training loss uses final-layer audio and visual features $26$22 and computes synchronization only on the generated interval:

$26$23

On LRS2, the unsupervised AV-HuBERT-based supervision variant reports SSIM $26$24, PSNR $26$25, FID $26$26, LMD $26$27, $26$28, $26$29, and $26$30, outperforming the SyncNet-based baseline (Yaman et al., 2024). The paper attributes this to AV-HuBERT features being more stable, less fluctuation-prone on ground-truth audio-lip pairs, and more robust to spatial shifts and affine transformations than SyncNet (Yaman et al., 2024).

For speaker representation learning, AV-HuBERT has been used as a self-supervised backbone for lip-based audio-visual speaker embeddings (Shi et al., 2022). With roughly $26$31 hours of unlabeled pretraining data from LRS3 and VoxCeleb2, the reported conclusion is roughly ten-fold label-efficiency improvement for both audio-only and audio-visual speaker verification (Shi et al., 2022). In clean conditions, incorporating visual information reduces EER by $26$32, and in noisy conditions by $26$33; for example, under full VC2 fine-tuning the audio-only AV-HuBERT-B model reports $26$34 EER on VC2 clean while the audio-visual version reports $26$35, and on noisy VC1 average EER the AV-HuBERT-L comparison is $26$36 audio-only versus $26$37 audio-visual (Shi et al., 2022).

In multimodal deepfake detection, AV-Lip-Sync+ uses AV-HuBERT as a transformer-based self-supervised audio-visual feature extractor for lip, audio, and joint audiovisual embeddings, then combines those with a synchronization-difference representation and a multi-scale temporal convolutional network (Shahzad et al., 2023). The synchronization cue is constructed as

$26$38

and the fused representation is classified with binary cross-entropy (Shahzad et al., 2023). On FakeAVCeleb, AV-Lip-Sync+ reports accuracies of $26$39 on Faceswap, $26$40 on Faceswap_wav2lip, $26$41 on RTVC, $26$42 on Wav2lip, and $26$43 on Test-set-2, with AUC values all above $26$44; on DeepfakeTIMIT, AV-Lip-Sync+ with a face encoder reports $26$45 AUC on LQ and $26$46 on HQ (Shahzad et al., 2023). In this setting, AV-HuBERT is valued less as a recognizer than as a detector of natural audio-visual consistency.

These applications illustrate a broad shift in how AV-HuBERT is used. Rather than being confined to transcript prediction, it increasingly functions as an audiovisual prior: a learned representation space in which synchronization, speaker identity, speech content, and noise robustness can be exploited by downstream systems.

7. Internal representations, bio-fidelity, and limitations

Recent interpretability work has examined what AV-HuBERT actually encodes. One study of viseme representations uses t-SNE and probing on a base AV-HuBERT model fine-tuned on the $26$47-hour split of LRS3 and reports that final viseme classification accuracy rises from $26$48 for video-only input to $26$49 for noisy audio-visual input and $26$50 for clean audio-visual input (Papadopoulos et al., 19 Sep 2025). The same analysis reports that visually distinct consonant visemes such as F, W, CH, and P form clear clusters even in video-only conditions, whereas ambiguous or under-represented visemes such as ER and K benefit strongly from audio (Papadopoulos et al., 19 Sep 2025). The authors’ interpretation is that vision supplies coarse viseme grouping while audio refines those groups with phonemic detail (Papadopoulos et al., 19 Sep 2025). This suggests that AV-HuBERT’s multimodal hierarchy is not a simple averaging of modalities, but an asymmetric refinement process in which the utility of audio depends strongly on visual salience and class frequency.

A different line of work evaluates AV-HuBERT as a model of human multisensory speech perception. In a McGurk-style comparison between AV-HuBERT Large and $26$51 naïve adult Spanish speakers, the model shows a McGurk-like effect under incongruent auditory $26$52 and visual $26$53 input, producing fused $26$54 responses on $26$55 trials, or $26$56, and auditory-consistent $26$57 responses on $26$58 trials, or $26$59 (López, 22 Jan 2026). Humans report fused $26$60 in $26$61 of cases and auditory-consistent $26$62 in $26$63 (López, 22 Jan 2026). The paper characterizes the near-match in auditory dominance ($26$64 versus $26$65) as a “striking quantitative isomorphism,” but emphasizes that AV-HuBERT over-fuses relative to humans and lacks their perceptual stochasticity and response diversity (López, 22 Jan 2026).

Temporal analysis raises a different limitation. A study comparing AV-HuBERT with audio-only HuBERT tracks the time course of phonetic decodability and finds that phoneme information becomes available in AV-HuBERT embeddings only about $26$66 ms before HuBERT, despite natural audiovisual speech often exhibiting a lip lead of $26$67–$26$68 ms (Wang et al., 25 Jun 2025). The proposed explanation is architectural: AV-HuBERT stacks every four $26$69 Hz audio frames and concatenates them with one $26$70 fps visual frame, effectively reducing the temporal resolution to $26$71 ms and yielding an expected shift of about $26$72 ms (Wang et al., 25 Jun 2025). The same work concludes that AV-HuBERT does not adequately capture the temporal dynamics of multimodal speech perception and remains largely audio-dominated for timing-sensitive questions (Wang et al., 25 Jun 2025).

These interpretability results complicate a common misconception. AV-HuBERT is often taken to be a general model of human audiovisual speech perception because it performs well on AVSR and can reproduce some qualitative multisensory phenomena. The evidence is narrower. It encodes meaningful visual speech structure, benefits substantially from audiovisual integration, and can mimic some behavioral outcomes, but it also exhibits deterministic categorical biases and limited sensitivity to the natural temporal asynchrony between lip movements and voicing (Papadopoulos et al., 19 Sep 2025, López, 22 Jan 2026, Wang et al., 25 Jun 2025). A plausible implication is that AV-HuBERT is a strong engineering model of audiovisual speech representation, but an incomplete mechanistic model of biological multisensory speech perception.