OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup (2410.21269v1)

Abstract: The scaling up has brought tremendous success in the fields of vision and language in recent years. When it comes to audio, however, researchers encounter a major challenge in scaling up the training data, as most natural audio contains diverse interfering signals. To address this limitation, we introduce Omni-modal Sound Separation (OmniSep), a novel framework capable of isolating clean soundtracks based on omni-modal queries, encompassing both single-modal and multi-modal composed queries. Specifically, we introduce the Query-Mixup strategy, which blends query features from different modalities during training. This enables OmniSep to optimize multiple modalities concurrently, effectively bringing all modalities under a unified framework for sound separation. We further enhance this flexibility by allowing queries to influence sound separation positively or negatively, facilitating the retention or removal of specific sounds as desired. Finally, OmniSep employs a retrieval-augmented approach known as Query-Aug, which enables open-vocabulary sound separation. Experimental evaluations on MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ datasets demonstrate effectiveness of OmniSep, achieving state-of-the-art performance in text-, image-, and audio-queried sound separation tasks. For samples and further information, please visit the demo page at \url{https://omnisep.github.io/}.

• The paper presents the Query-Mixup strategy to merge multi-modal queries for enhanced sound separation.
• It employs negative query handling to identify and remove undesired audio components effectively.
• The framework achieves open-vocabulary separation with state-of-the-art performance across benchmark datasets.

OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup

The paper presents a novel framework, OmniSep, aimed at advancing the domain of sound separation through the integration of omni-modal queries. By employing a Query-Mixup technique, the authors propose an efficient method for isolating clean soundtracks from environments rich in audio interference. This approach accommodates both single-modal and multi-modal queries, effectively converging various modalities into a unified system that optimizes sound separation tasks.

Core Contributions

OmniSep introduces several key innovations:

1. Query-Mixup Strategy: This approach strategically merges query features from different modalities during the training phase, allowing the model to optimize across modalities concurrently. By doing so, OmniSep provides a cohesive solution that bridges the gap across text, image, and audio queries.
2. Negative Query Handling: The model introduces a mechanism to influence sound separation targets by identifying and eliminating sounds associated with undesirable queries. This enhances the flexibility of the model by allowing both the retention and removal of specific audio components.
3. Open-Vocabulary Sound Separation: Using the Query-Aug method, OmniSep utilizes a retrieval-augmented approach that supports open-vocabulary queries. This is particularly pertinent given the limitations of predefined class labels in existing datasets, as it allows for dynamic, unrestricted natural language descriptions.
4. State-of-the-Art Performance: Experiments conducted on datasets such as MUSIC, VGGSOUND-CLEAN+, and MUSIC-CLEAN+ exhibit OmniSep’s superior performance in various sound separation contexts, establishing it as a cutting-edge solution in this space.

Numerical Results

The experimental evaluations clearly demonstrate OmniSep’s capacity to outperform existing models across a range of metrics. Specifically, OmniSep achieves notable gains in mean SDR across the tested datasets, showcasing its efficacy in both text-, image-, and audio-queried sound separation tasks. Notably, the model’s performance is further amplified by the inclusion of negative queries, providing significant enhancements in sound elimination capabilities.

Theoretical and Practical Implications

OmniSep’s framework not only contributes to practical applications in audio processing but also prompts theoretical advancements in multi-modal machine learning. Practically, the ability to handle complex audio environments with varied queries has substantial implications in media processing, surveillance, and environmental sound analysis. Theoretically, the integration of Query-Mixup opens new avenues for research into cross-modal interactions and joint optimization techniques in machine learning.

Future Directions

The research lays a foundation for multiple potential directions in future AI and audio processing research. Expanding the dataset to encompass a broader range of natural audio events, incorporating more sophisticated neural architectures for query processing, and exploring further enhancements to the Query-Aug mechanism could improve the robustness and adaptability of sound separation models. Additionally, insights from this work could inform applications beyond sound separation, such as in fields where multi-sensory data integration is crucial.

In conclusion, the OmniSep framework represents a significant stride forward in sound separation technology, emphasizing the potential of unified, omni-modal approaches for comprehensive audio analysis and processing. The model’s robust performance across diverse scenarios underscores its potential to redefine standards in both academic and applied aspects of sound separation.