HuBERT-VIC: Robust Speech Recognition

• The paper demonstrates that HuBERT-VIC, by integrating VICReg and a viseme-guided auxiliary loss, significantly improves speech recognition robustness under noisy conditions.
• It employs a dual teacher/student framework with weighted invariance, variance, and covariance losses to reduce the impact of acoustic degradation while preserving clean-speech accuracy.
• Empirical results on LibriSpeech and LRS datasets show notable reductions in word error rates, highlighting the model’s effectiveness in both speech-only and audio-visual settings.

HuBERT-VIC refers to two distinct but related research directions addressing robust speech recognition: (1) a variant of the HuBERT speech foundation model enhanced with variance-invariance-covariance regularization (VICReg) for improved noise robustness in automatic speech recognition (ASR) (Ahn et al., 17 Aug 2025), and (2) viseme-guided AV-HuBERT, in which HuBERT-style audio-visual encoding is coupled with a viseme auxiliary loss to strengthen encoder reliance on visual articulatory cues under noisy conditions (Papadopoulos et al., 1 Apr 2026). Both leverage advanced regularization and multitask learning principles to address deficiencies in speech foundation models when confronted by acoustic degradation or audio-visual domain ambiguity.

1. Core Concepts and Motivation

Conventional speech foundation models (SFMs) such as HuBERT demonstrate considerable performance degradation when exposed to additive noise due to a training bias toward clean-speech corpora. Similarly, in audio-visual speech recognition (AVSR), there remains an open question regarding the balance of gains from language modeling versus encoder-level robustness, particularly under high-noise or speech-on-speech conditions. HuBERT-VIC in its speech-only form addresses this by regularizing learned representations through variance, invariance, and covariance (VIC) constraints, promoting greater generalization to out-of-distribution noise. The viseme-guided AV-HuBERT adaptation introduces visual speech unit prediction as an auxiliary target, directly instilling a visual-articulatory inductive bias intended to counteract acoustic masking or channel corruption (Ahn et al., 17 Aug 2025, Papadopoulos et al., 1 Apr 2026).

2. HuBERT-VIC: Speech Foundation Model with VICReg

The HuBERT-VIC framework builds on the standard HuBERT Base model (7 CNN layers, 12 transformer layers, embedding dimension $d=768$), augmenting masked prediction pretraining with VICReg. The pretraining pipeline implements a clean “teacher” (frozen, clean waveform, no masking/dropout), and a noise-augmented “student” (noisy waveforms, SNR $\sim$ Uniform[5,10] dB, noise from MUSAN babble/music/natural subsets), both initialized from a HuBERT-Base checkpoint trained on 960 h LibriSpeech. For each mini-batch, $n=512$ frame embeddings are sampled from both teacher $\mathbf{Z}$ and student $\mathbf{Z}' \in \mathbb{R}^{n \times d}$; the loss consists of the standard HuBERT masked-prediction objective and a weighted sum of VICReg terms:

$$\mathcal{L}_{tot} = \mathcal{L}_m + \alpha\, \mathcal{L}_{VIC}$$

with

$$\mathcal{L}_{VIC} = \lambda\, s(\mathbf{Z}, \mathbf{Z}') + \mu\, v(\mathbf{Z}') + \nu\, c(\mathbf{Z}')$$

where

• $s(\mathbf{Z}, \mathbf{Z}')$: invariance (MSE between teacher/student embeddings),
• $v(\mathbf{Z}')$: variance regularization (ensures per-channel std $\geq \gamma$),
• $\sim$0: covariance penalty (discourages inter-channel redundancy).

Default hyperparameters: $\sim$1, $\sim$2, $\sim$3, $\sim$4, $\sim$5, $\sim$6; $\sim$7 samples per batch; pretraining for 50k Adam steps with precomputed $\sim$8 k-means pseudo-labels. Fine-tuning uses 100 h labeled LibriSpeech (train-clean-100) with CTC loss and 4-gram LM (beam size 1500).

3. Formal VICReg Objectives

The variance-invariance-covariance regularization decomposes as follows, where $\sim$9 (teacher) and $n=512$0 (student) are $n=512$1 matrices:

• Invariance: $n=512$2
• Variance: $n=512$3
• Covariance: $n=512$4, $n=512$5

Each addresses a distinct collapse mode: invariance aligns noisy and clean, variance prevents channel collapse, and covariance avoids trivial redundancy (Ahn et al., 17 Aug 2025).

4. Viseme-Guided AV-HuBERT: Audio-Visual Extension

The viseme-guided AV-HuBERT adaptation integrates a lightweight viseme prediction head (“VIC head”) atop the fused audio-visual encoder output, designed as $n=512$6 repetitions of Linear(d$n=512$7d), LayerNorm, GELU, and Dropout($n=512$8), followed by a Linear(d$n=512$915) layer for viseme CTC over 14 viseme classes plus blank. $\mathbf{Z}$0 is optimal for Base/Large encoders, respectively. The multi-task loss is:

$\mathbf{Z}$1

where $\mathbf{Z}$2 is the standard autoregressive cross-entropy (text sequence), and $\mathbf{Z}$3 is the CTC loss over viseme classes. $\mathbf{Z}$4 (Base), $\mathbf{Z}$5 (Large) balances the two tasks. Viseme ground truth derives from Montreal Forced Aligner with Lee’s 14-class ARPABET viseme mapping. This protocol is applied to the LRS3 and LRS2 datasets, with audio augmented by MUSAN and speech noise (mix $\mathbf{Z}$6, SNR $\mathbf{Z}$7 dB). Optimization employs Adam with AV-HuBERT’s learning rate schedule (Papadopoulos et al., 1 Apr 2026).

5. Empirical Performance

For HuBERT-VIC (speech-only), evaluation on LibriSpeech test-clean/test-other with babble/music/natural noise and SNRs in $\mathbf{Z}$8 dB demonstrates superior robustness:

Model	test-clean N-WER	test-other N-WER
HuBERT-noisy-PT	7.3	9.5
HuBERT-AGG (distill)	6.3	8.6
HuBERT-VIC	5.6 (-23.3%)	15.1 (-13.2%)

In the 5–10 dB regime, HuBERT-VIC achieves 4.2%/11.3% (test-clean/test-other), marginally exceeding HuBERT-AGG, while maintaining clean-speech accuracy (3.4% WER). Ablations indicate that the invariance loss provides the largest improvement, with variance and covariance regularization producing additional incremental gains (Ahn et al., 17 Aug 2025).

AV-HuBERT with viseme guidance (HuBERT-VIC, Editor's term for this context) yields the following on LRS3/LRS2:

Model	LRS3 Clean WER	LRS2 Clean WER	–10 dB Speech WER
AV-HuBERT Base	4.10%	10.09%	15.06%
HuBERT-VIC Base	3.78%	10.58%	13.55%
AV-HuBERT Large	1.40%	10.30%	13.59%
HuBERT-VIC Large	1.38%	9.93%	6.60%

At –10 dB speech noise, the Large model reduces WER from 13.59% to 6.60% (51.4% relative); similar gains occur at –5 dB and 0 dB. Error analysis shows substantial decreases in substitutions, insertions, and deletions under speech noise.

6. Component Analysis and Representation Learning Impact

Ablation studies for HuBERT-VIC reveal:

• Adding only the invariance term reduces WER substantially compared to $\mathbf{Z}$9 alone,
• Adding variance regularization gives a further slight improvement,
• The full VICReg (invariance + variance + covariance) gives the best results.

The invariance loss forces noisy student representations toward the clean teacher, reducing clean-noisy mismatch. Variance regularization eliminates channel collapse, increasing channel-wise diversity—a property correlated with SNR and ASR accuracy. Covariance minimization promotes factorized, specialized channel embeddings. In the AV context, explicit viseme supervision in the encoder enhances the discrimination of visually similar speech units under noise, notably reducing substitution errors on difficult sentences and under challenging speech-on-speech conditions (Ahn et al., 17 Aug 2025, Papadopoulos et al., 1 Apr 2026).

7. Significance and Outlook

HuBERT-VIC establishes the utility of robust self-supervised regularization and explicit multimodal supervision within SFMs and AVSR systems. By ensuring representation diversity, clean-noisy alignment, and decorrelation, these frameworks simultaneously sustain clean performance and dramatically boost resilience to nonstationary and adversarial acoustic conditions. The general principle—statistical regularization of internal representations and multitask visual-articulatory guidance—constitutes a methodological advance applicable to a broad range of noisy channel and speech-in-the-wild scenarios (Ahn et al., 17 Aug 2025, Papadopoulos et al., 1 Apr 2026).