Learning to Highlight Audio by Watching Movies

Abstract: Recent years have seen a significant increase in video content creation and consumption. Crafting engaging content requires the careful curation of both visual and audio elements. While visual cue curation, through techniques like optimal viewpoint selection or post-editing, has been central to media production, its natural counterpart, audio, has not undergone equivalent advancements. This often results in a disconnect between visual and acoustic saliency. To bridge this gap, we introduce a novel task: visually-guided acoustic highlighting, which aims to transform audio to deliver appropriate highlighting effects guided by the accompanying video, ultimately creating a more harmonious audio-visual experience. We propose a flexible, transformer-based multimodal framework to solve this task. To train our model, we also introduce a new dataset -- the muddy mix dataset, leveraging the meticulous audio and video crafting found in movies, which provides a form of free supervision. We develop a pseudo-data generation process to simulate poorly mixed audio, mimicking real-world scenarios through a three-step process -- separation, adjustment, and remixing. Our approach consistently outperforms several baselines in both quantitative and subjective evaluation. We also systematically study the impact of different types of contextual guidance and difficulty levels of the dataset. Our project page is here: https://wikichao.github.io/VisAH/.

• The paper proposes visually-guided acoustic highlighting, a new task and framework (VisAH) that uses visual cues to automatically adjust and improve poorly mixed audio in videos for better audio-visual alignment.
• It introduces a transformer-based multimodal framework integrating audio and visual signals, trained on a novel 'muddy mix' dataset derived from movie clips to learn from expertly mixed audio-visual examples.
• Evaluation shows VisAH outperforms existing methods in correcting audio saliency guided by video and has implications for enhancing video editing tools and improving audio quality in various video contexts.

Visually-Guided Acoustic Highlighting in Videos

The paper "Learning to Highlight Audio by Watching Movies" addresses a significant shortcoming in modern media production—the discordance between visual and acoustic elements in video content. While considerable advances have been made in optimizing visual cues, manipulations and balancing of audio elements have lagged, leading to audio-visual experiences that are often incongruent. This research proposes a novel task: visually-guided acoustic highlighting, which seeks to automatically transform poorly mixed audio to match the saliency of its visual counterpart in videos.

The authors introduce a multimodal framework powered by transformers, dubbed VisAH, which integrates both audio and visual signals to achieve balanced acoustic output guided by video content. For model training, they leverage a new dataset—the muddy mix dataset—crafted from movie clips to provide free supervision where audio is expertly mixed to highlight pertinent elements alongside visual cues.

Technical Framework:

The approach is structured around:

• Audio Backbone: Utilizes a dual U-Net architecture for spectrogram and waveform inputs, extracting latent audio features that encapsulate poorly mixed sounds.
• Latent Highlighting Module: Employs transformer encoding with video or caption-based context, facilitating audio transformation aligned with the visual narrative.

This framework is trained to map poorly mixed audio and video sequences to an enhanced audio signal that mirrors intentional audio-visual alignment found in expertly mixed movie scenes.

Dataset Creation:

To simulate real-world poorly mixed scenarios, a pseudo-data generation process is developed, entailing:

• Separation: Decomposition of high-quality movie audio into speech, music, and sound effects.
• Adjustment: Alteration of the loudness of these components based on predefined levels of suppression or emphasis.
• Remixing: Linear recombination to create input signals with intentional misalignment from their original movie highlight effects.

Evaluation:

The model's efficacy is assessed through a suite of metrics examining waveform similarity, semantic alignment, and time alignment relative to movie-based ground truths. The results demonstrate VisAH’s superiority over existing methods, including Learn2Remix and Listen, Chat, and Edit (LCE), particularly in correcting audio saliency guided by video input.

Implications and Future Directions:

The findings highlight potential applications in refining video-to-audio generation models and enhancing audio quality in less controlled recording environments like web videos. The approach could redefine audio editing practices, offering tools to achieve cinematic-level audio-visual synchronization by leveraging the free supervision available in movie datasets.

Future research could expand upon multimodal condition fusion to enhance audio synthesis further and refine dataset generation strategies, such as employing more diverse audio source separation methods or introducing temporal loudness variability. Such work could extend the applicability of visually-guided acoustic highlighting beyond movie settings to other contexts requiring nuanced audio balancing.

Overall, this paper establishes a foundation for integrating deep learning and media production techniques, promising advancements in creating more coherent and immersive audio-visual experiences.