Audio-Video Joint Denoising

• Audio-video joint denoising is a multimodal approach that exploits temporal and causal relationships to reduce noise in both audio and video streams.
• It employs fusion strategies such as cross-modal attention and connector-based adaptation to align features and enhance signal fidelity.
• Empirical studies demonstrate improved video quality metrics and enhanced synchronization, underscoring its value in robust multimedia processing.

Audio-video joint denoising refers to a class of methodologies that leverage both audio and video signals to enhance denoising, alignment, or fusion in multimodal data streams. Unlike unimodal denoising, these approaches explicitly address redundancy, noise, synchronization, and causality across modalities, aiming to yield more robust representations and higher fidelity generation or understanding. Recent research demonstrates that joint denoising not only improves audio-video synchrony but can also enhance single-modality performance, especially for video, by inducing models to internalize physical or causal relationships between modalities.

1. Joint Denoising Principles and Motivations

Audio-video joint denoising builds on the observation that multimodal signals contain both complementary and redundant information, and natural temporal correlations (e.g., object contact and impact sound) enable more informed predictions than treating each modality independently. Two key motivations underlie joint denoising frameworks:

• Cross-modal regularization: Conditioning the denoising of one modality on the other exploits shared structure and resolves ambiguities (such as muted visuals or non-lip-synced audio).
• Causal grounding: Training models to predict both audio and video simultaneously encourages the internalization of physically plausible cross-modal relationships, improving generalization in video dynamics and event detection.

Recent controlled studies, such as "Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation" (Wu et al., 2 Dec 2025), have shown that audio supervision can systematically improve video fidelity and physical plausibility, even when only video is evaluated.

2. Architectural Strategies for Joint Denoising

A range of architectures has been developed for audio-video joint denoising. Salient examples include:

• Fusion of discrete pretrained backbones: AVFullDiT (Wu et al., 2 Dec 2025) uses early unimodal DiT towers for each modality, merging them in top layers via symmetric "AVFull-Attention," which allows bidirectional conditioning between video and audio with minimal new parameters.
• Cross-modal guidance modules: MMDisCo (Hayakawa et al., 28 May 2024) employs a joint discriminator to guide and correlate single-modality diffusion models, adjusting scores to match the joint distribution via a lightweight, trainable module.
• Connector-based adaptation and positional alignment: Integrating latent-diffusion U-Nets for video (AnimateDiff) and audio (AudioLDM), a baseline design (Ishii et al., 26 Sep 2024) injects only connector modules (transformer-based self-attention) between fixed U-Nets, with Cross-Modal Conditioning as Positional Encoding (CMC-PE) enforcing strict temporal alignment.

These strategies exploit either parameter sharing, flexible adapters, or explicit temporal encoding to unify audio and video feature spaces, yielding robust fusion points for denoising.

3. Formal Modeling and Loss Functions

Contemporary joint denoising models generalize the standard diffusion or score-matching framework across modalities:

• Diffusion formulation: For each modality $s \in \{v, a\}$, the forward process adds Gaussian noise in a modality-specific schedule, while the reverse process is parameterized using joint denoising networks that condition on both noisy video and audio latents at appropriately aligned time steps (Ishii et al., 26 Sep 2024).
• Joint loss objectives: The typical loss combines audio and video denoising errors, potentially with weighted terms:

$$L_{joint} = \lambda_v E[||\epsilon^v - \hat{v}_v||^2] + \lambda_a E[||\epsilon^a - \hat{v}_a||^2]$$

where $\hat{v}_v$ and $\hat{v}_a$ are predicted velocities/noises for video and audio (Wu et al., 2 Dec 2025).

• Adversarial or discriminator-based guidance: In MMDisCo (Hayakawa et al., 28 May 2024), the joint score is decomposed as:

$$\nabla_{x_t} \log q(x_t, y_t) \approx -\frac{1}{\sqrt{1-\bar{\alpha}_t}} \epsilon^x_\phi(x_t, t) + \nabla_{x_t} \log \frac{D_\theta}{1 - D_\theta}$$

where $D_\theta$ estimates the density ratio between true and fake audio-video pairs.

An essential practical innovation is the adjustment of local timesteps to ensure that, at each global diffusion step, the noise level is comparable across modalities, as in the timestep adjustment mechanism of (Ishii et al., 26 Sep 2024).

4. Synchronization, Alignment, and Information Routing Mechanisms

Ensuring precise temporal alignment and information flow is critical in audio-video joint denoising. Techniques include:

• Timestep adjustment: The mechanism resynchronizes noise schedules by mapping the global step to the correct local time indices for each modality, aligning the effective SNR between video and audio at every step (Ishii et al., 26 Sep 2024).
• Cross-Modal Conditioning as Positional Encoding (CMC-PE): Instead of generic cross-attention, CMC-PE treats cross-modal information as a positional bias, strictly tied to frame or timestep indices, enforcing frame-to-frame conditioning and temporal coherence (Ishii et al., 26 Sep 2024).
• Full multi-head attention over concatenated audio-video tokens: In AVFullDiT (Wu et al., 2 Dec 2025), joint blocks apply attention across the concatenated representation of both modalities, reusing pretrained projections with small adapters, maximizing parameter efficiency while allowing deep cross-modal fusion.
• Bottleneck constraints: DBF (2305.14652) forces cross-modal signals through a small set of bottleneck tokens, suppressing redundant or noisy information and promoting the passage of only salient features.

Alignment modules such as AVSyncRoPE further ensure that positional encodings in audio and video are temporally coherent, which is essential for synchronization and physically plausible outputs (Wu et al., 2 Dec 2025).

5. Evaluation Protocols and Empirical Evidence

Evaluation of audio-video joint denoising encompasses unimodal quality, cross-modal alignment, and causal realism:

• Video quality: Assessed by Fréchet Video Distance (FVD), background/subject consistency, dynamic degree, and image quality (Ishii et al., 26 Sep 2024, Wu et al., 2 Dec 2025).
• Audio quality: Measured by Fréchet Audio Distance (FAD), and CLAP similarity (Hayakawa et al., 28 May 2024, Ishii et al., 26 Sep 2024).
• Cross-modal alignment: Metrics include AV-align (onset–optical flow IoU), IB-AV (ImageBind audio-video similarity), AV-Align, and Synchformer (Hayakawa et al., 28 May 2024, Ishii et al., 26 Sep 2024).
• Physical commonsense: Evaluated using Videophy-2 physics plausibility metrics (Wu et al., 2 Dec 2025).

Empirical results demonstrate that joint denoising systematically improves both single-modality fidelity and cross-modal alignment. For instance, in (Wu et al., 2 Dec 2025), joint training with AVFullDiT yields improvements in video Background Consistency (97.44→97.93%), Image Quality (59.04→59.87%), and physics scores on contact-intensive events (+3.14%), even when the video is evaluated in isolation. Substantial alignment and perceptual gains are observed in comparative and ablation studies across models and datasets (Hayakawa et al., 28 May 2024, Ishii et al., 26 Sep 2024).

6. Applications and Broader Impact

Audio-video joint denoising models have demonstrated utility in:

• Sounding video generation: Creation of temporally coherent, naturally synchronized audio-visual outputs from latent diffusion models, outperforming individual or sequential pipelines (Ishii et al., 26 Sep 2024, Wu et al., 2 Dec 2025).
• Multimodal parsing and event detection: Dynamic denoising and refinement of label assignments in weakly-supervised setups, with application to parsing and localizing events that may only manifest in one modality (Cheng et al., 2022).
• Denoising and enhancement in real-world data: Fine-tuning joint denoisers for restoration tasks (e.g., video deblocking guided by clean audio) is feasible within the same frameworks (Ishii et al., 26 Sep 2024).
• Physical world modeling: Cross-modal co-training improves the capacity of generative models to capture physically grounded and causal interactions, suggesting potential for enhanced video understanding, robotic perception, and virtual simulation (Wu et al., 2 Dec 2025).

A plausible implication is that audio, as a privileged modality, regularizes video generation tasks by providing discriminative cues for underdetermined motion events, facilitating robust scene understanding and more stable generative outputs.

7. Extensions, Limitations, and Research Directions

Several research directions and limitations are evident in current audio-video joint denoising literature:

• Generalization to additional modalities: The frameworks support the integration of further privileged or correlated channels (e.g., depth, event streams) for richer world modeling (Wu et al., 2 Dec 2025).
• Label noise mitigation: Joint-modal label denoising (JoMoLD) strategies, leveraging cross-modal loss discrepancies and per-class noise estimates, can be extended to other multi-sensor contexts and weak supervision scenarios (Cheng et al., 2022).
• Scalability and efficiency: Parameter-efficient fusion and adapter strategies (AVFull-Attention, CMC-PE) mitigate compute overhead, but further reduction in computational cost and memory requirements remains an open challenge (Hayakawa et al., 28 May 2024, Ishii et al., 26 Sep 2024).
• Robustness to misalignment and annotation noise: Mechanisms such as timestep adjustment and bottleneck attention improve robustness, but severe temporal misalignment and out-of-domain generalization demand more sophisticated intervention (2305.14652).
• Domain-specific enhancements: Domain adaptation (e.g., domain-specific noise schedules, connector tuning) and restoration by unequal noise allocation per modality are promising areas for future research (Ishii et al., 26 Sep 2024).

This body of evidence demonstrates that principled joint denoising architectures, leveraging cross-modal correspondence and synchronization, constitute a powerful methodology for advancing both generation and understanding in audio-visual machine learning.