Hierarchical Codec Diffusion for Video-to-Speech Generation

Abstract: Video-to-Speech (VTS) generation aims to synthesize speech from a silent video without auditory signals. However, existing VTS methods disregard the hierarchical nature of speech, which spans coarse speaker-aware semantics to fine-grained prosodic details. This oversight hinders direct alignment between visual and speech features at specific hierarchical levels during property matching. In this paper, leveraging the hierarchical structure of Residual Vector Quantization (RVQ)-based codec, we propose HiCoDiT, a novel Hierarchical Codec Diffusion Transformer that exploits the inherent hierarchy of discrete speech tokens to achieve strong audio-visual alignment. Specifically, since lower-level tokens encode coarse speaker-aware semantics and higher-level tokens capture fine-grained prosody, HiCoDiT employs low-level and high-level blocks to generate tokens at different levels. The low-level blocks condition on lip-synchronized motion and facial identity to capture speaker-aware content, while the high-level blocks use facial expression to modulate prosodic dynamics. Finally, to enable more effective coarse-to-fine conditioning, we propose a dual-scale adaptive instance layer normalization that jointly captures global vocal style through channel-wise normalization and local prosody dynamics through temporal-wise normalization. Extensive experiments demonstrate that HiCoDiT outperforms baselines in fidelity and expressiveness, highlighting the potential of discrete modelling for VTS. The code and speech demo are both available at https://github.com/Jiaxin-Ye/HiCoDiT.

• The paper introduces HiCoDiT, a hierarchical codec diffusion transformer that aligns discrete low- and high-level speech tokens with corresponding visual cues.
• It employs residual vector quantization and a dual-block transformer to disentangle speaker semantics from prosody, achieving state-of-the-art audio-visual alignment.
• Experimental results demonstrate improved intelligibility, naturalness, and robust out-of-domain performance on LRS datasets compared to prior video-only models.

Hierarchical Codec Diffusion for Video-to-Speech Generation

Introduction

The paper "Hierarchical Codec Diffusion for Video-to-Speech Generation" (2604.15923) introduces HiCoDiT, a hierarchical codec diffusion transformer designed for high-fidelity video-to-speech (VTS) generation. HiCoDiT leverages the intrinsic hierarchical structure of discrete speech token spaces, harnessing residual vector quantization (RVQ) to tokenize speech into low-level (speaker- and semantics-aware) and high-level (prosody-centric) representations. Through a dual-block transformer architecture and a disentangled visual conditioning pipeline, HiCoDiT achieves superior audio-visual alignment and expressiveness, mitigating the information asymmetry between visual and acoustic modalities that plagues prior art.

Figure 1: The HiCoDiT framework formulates video-to-speech as hierarchical masked token prediction, aligning visual features with hierarchical speech tokens for improved semantic and prosody generation.

Speech Token Hierarchy and Visual Conditioning

The core premise is that speech signals exhibit a hierarchical composition, wherein lower-level codebook tokens encode coarse-grained semantics and speaker identity, while higher-level tokens capture fine-grained prosody. HiCoDiT utilizes an RVQ codec with twelve quantization layers, partitioning the token sequence into $\bm{x}^{r_1:r_2}$ (low-level) and $\bm{x}^{r_3:r_{12}}$ (high-level) components.

Figure 2: RVQ hierarchy analysis reveals that semantic and timbre fidelity primarily improve at lower levels, while prosodic quality gains are pronounced in higher layers.

Visual information is disentangled into three key modalities for hierarchical conditioning:

• Lip Motion ($\bm{c}_\text{lip}$): Extracted via AV-HuBERT, providing temporally-aligned content cues injected into low-level blocks.
• Identity ($\bm{c}_\text{id}$): Derived from ArcFace embeddings, regularized toward GE2E audio embeddings and infused through dual-scale AdaLN in low-level blocks for timbre consistency.
• Emotion ($\bm{c}_\text{emo}$): Obtained from smoothed Poster2 expression sequences, injected into high-level blocks via a novel temporal-channel dual-scale AdaLN, modulating prosodic expressiveness dynamically.

This architectural separation aligns visual attributes with their corresponding speech representations, enabling effective cross-modal mapping and speech synthesis.

Hierarchical Masked Token Diffusion

On the modeling side, HiCoDiT formulates speech generation as a hierarchical masked token prediction problem, implemented through a discrete diffusion process. Speech tokens are masked according to a noise schedule and reconstructed via a masked token diffusion transformer. The denoising score entropy (DSE) objective is used to parameterize the reverse process, where score networks are trained to predict the data distribution ratios for unmasking.

This framework enables scalable, efficient, and high-quality generation by exploiting the discrete nature of RVQ tokens and explicitly modeling hierarchical dependencies. The transformer backbone further integrates both frame-synchronized concatenative and global/temporal normalization-based conditioning to maximize responsiveness to visual cues.

Experimental Evaluation

HiCoDiT is trained on a significantly preprocessed VoxCeleb2 dataset (261.5 hours, 3,438 speakers, 7 emotions) and evaluated on in-the-wild LRS2/LRS3 datasets, targeting out-of-domain (OOD) generalization and real-world applicability.

Objective Results:

• On LRS3, HiCoDiT achieves state-of-the-art WER (29.41), UTMOS (3.84), and DNSMOS (3.50) among video-only models, surpassing prior methods such as FTV and AlignDiT in intelligibility, perceptual quality, and synchronization accuracy.
• On LRS2, it maintains competitive WER (39.99), UTMOS (3.68), and DNSMOS (3.35), robustly outperforming other approaches across expressiveness and lip sync metrics.

Subjective Evaluation:

• HiCoDiT leads in naturalness ($\text{MOS}_\text{nat}$ = 3.17 vs. 2.80/2.47 for FTV/AlignDiT) and synchronization ($\text{MOS}_\text{syn}$ = 3.50). Expressiveness is slightly below FTV, attributed to speaker diversity constraints in training data.
• In A/B comparisons, human raters prefer HiCoDiT over AlignDiT (57.0%) and FTV (52.1%), and slightly over ground truth (53.9%), highlighting perceptual closeness to real speech.

  Figure 3: Mel-spectrograms visualize clarity and signal-to-noise improvements from HiCoDiT compared to baseline models.

OOD Generalization:

• HiCoDiT demonstrates robust generalization on real film data, outperforming EmoDubber and AlignDiT in WER, MCD, DNSMOS, emotion, speaker similarity, and lip sync distance.

  Figure 4: Generated mel-spectrograms on real-world film data emphasize strong OOD generalization and robustness.

Ablations:

• Hierarchical modeling is critical: removing it significantly degrades all metrics, especially expressiveness and synchronization.
• Dual-scale AdaLN is essential for dynamic emotion/prosody control, outperforming single-scale alternatives.
• Visual conditioning strategies (GE2E loss for identity, Poster2 for emotion) are indispensable for preserving speaker timbre and affective accuracy.

Implications and Future Directions

HiCoDiT establishes a path toward more coherent and expressive cross-modal generation by imposing architectural and representational priors mirroring the intrinsic hierarchy of speech. The model's discrete, interpretable latent space opens doors for fine-grained editing, controllable generation, and efficient adaptation for related tasks such as codec-based editing, in-context video dubbing, or voice conversion.

Key theoretical implications include the demonstration that explicit token-level alignment enables superior modality bridging and disentanglement, establishing a principled foundation for future VTS and multimodal generation paradigms. HiCoDiT's robustness on OOD data also signals readiness for real-world, variable environments.

Scalability to broader emotional, linguistic, and speaker distributions remains an open direction. Further research could explore more granular hierarchy divisions, multimodal supervision signals, or end-to-end training with text and other context modalities. Integrating advanced diffusion sampling or expressive LLMs could further elevate generation fidelity and controllability.

Conclusion

HiCoDiT introduces a hierarchical discrete token diffusion paradigm for video-to-speech generation, aligning visual content, identity, and emotional expression to speech tokens at the appropriate semantic levels. Empirical results demonstrate strong objective and subjective gains across intelligibility, fidelity, synchronization, and expressiveness, while architectural ablations validate the necessity of hierarchical and disentangled conditioning. HiCoDiT marks a significant step toward controllable, high-fidelity, and robust audio-visual generation, with practical implications for silent video dubbing, assistive communication, and generalized multimodal synthesis.