ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction (2012.06170v3)

Abstract: We propose the ViNet architecture for audio-visual saliency prediction. ViNet is a fully convolutional encoder-decoder architecture. The encoder uses visual features from a network trained for action recognition, and the decoder infers a saliency map via trilinear interpolation and 3D convolutions, combining features from multiple hierarchies. The overall architecture of ViNet is conceptually simple; it is causal and runs in real-time (60 fps). ViNet does not use audio as input and still outperforms the state-of-the-art audio-visual saliency prediction models on nine different datasets (three visual-only and six audio-visual datasets). ViNet also surpasses human performance on the CC, SIM and AUC metrics for the AVE dataset, and to our knowledge, it is the first network to do so. We also explore a variation of ViNet architecture by augmenting audio features into the decoder. To our surprise, upon sufficient training, the network becomes agnostic to the input audio and provides the same output irrespective of the input. Interestingly, we also observe similar behaviour in the previous state-of-the-art models \cite{tsiami2020stavis} for audio-visual saliency prediction. Our findings contrast with previous works on deep learning-based audio-visual saliency prediction, suggesting a clear avenue for future explorations incorporating audio in a more effective manner. The code and pre-trained models are available at https://github.com/samyak0210/ViNet.

• The paper presents a fully convolutional encoder-decoder architecture that extracts and fuses hierarchical spatio-temporal features for high-fidelity saliency mapping.
• The model operates in real time at 60 fps and outperforms state-of-the-art benchmarks, even surpassing human performance on the AVE dataset.
• The study reveals that integrating audio features does not enhance performance, emphasizing the robust capabilities of visual-only saliency prediction.

ViNet: Advancements in Visual Modality for Saliency Prediction

The paper presents ViNet, a cutting-edge architecture designed specifically for visual saliency prediction in videos, focusing on outperforming existing audio-visual models by leveraging purely visual information. This approach challenges the prevailing notion that audio cues are crucial for enhancing video saliency prediction tasks.

Architectural Overview

ViNet introduces a fully convolutional encoder-decoder architecture, underpinned by convolutional operations that extend across both spatial and temporal dimensions. The encoder component of ViNet employs pretrained features from networks like S3D, traditionally used for action recognition, hence benefiting from the extensive representational capabilities developed on dynamic spatio-temporal datasets. The decoder is responsible for constructing a high-fidelity saliency map, utilizing trilinear interpolation and 3D convolutions to integrate features from various hierarchical levels effectively.

Key highlights of the ViNet model include:

• Causal Processing: The model can operate in real time (at 60 fps), making it well-suited for applications requiring low latency, such as real-time video processing.
• Hierarchical Feature Fusion: Employing multi-level features allows ViNet to diversify its representational capacity, blending fine-grained and abstract features to predict saliency more accurately.

Unprecedented Performance

ViNet surpasses the current state-of-the-art across a variety of benchmarks, achieving top ranks on dominant video saliency datasets such as DHF1K without depending on audio inputs. More impressively, it eclipses human performance metrics on the AVE dataset, highlighting its robust predictive capabilities. Despite its simplicity, ViNet excels in predicting visual saliency through strategic architectural choices and efficient feature amalgamation.

Audio Features and Multi-Modal Models

The paper also examines the potential augmentation of audio inputs into ViNet, yielding a variant called AViNet. Initial findings where audio features were integrated into the visual processing pipeline demonstrated that after sufficient training, the model inherently disregarded audio inputs, producing identical salience maps irrespective of the audio data provided. This revelation extends back to prior architectures, implying that audio information might not be leveraged as robustly by the model as previously thought.

Implications and Future Directions

The results presented in this paper indicate a compelling direction for future research into harnessing audio-visual data in a manner that genuinely contributes to saliency prediction. A critical exploration is necessary to understand why existing methods fail to capitalize on audio cues effectively, potentially requiring new datasets and refined architectures that capture the multimodal interdependencies more intimately.

The broader implications of ViNet extend into fields like human-computer interaction, robotic vision systems, and adaptive video compression strategies. The emphasis on visual modality presents cost-effective deployments of saliency detection systems where auditory data is sparse or redundant.

Conclusion

This research showcases the power of visual features, articulated skillfully through a concise yet potent architecture. ViNet epitomizes the strides in video saliency detection, setting a benchmark for visual-only prediction models. The opportunity to refine and revamp audio-visual fusion techniques also poses an exciting frontier for academic inquiry and technological advancement. As the field evolves, ViNet will likely stand as a pivotal reference point for embracing minimalistic yet effective design principles in deep learning models for video saliency prediction.