CLIPRerank: An Extremely Simple Method for Improving Ad-hoc Video Search (2401.08449v1)

Abstract: Ad-hoc Video Search (AVS) enables users to search for unlabeled video content using on-the-fly textual queries. Current deep learning-based models for AVS are trained to optimize holistic similarity between short videos and their associated descriptions. However, due to the diversity of ad-hoc queries, even for a short video, its truly relevant part w.r.t. a given query can be of shorter duration. In such a scenario, the holistic similarity becomes suboptimal. To remedy the issue, we propose in this paper CLIPRerank, a fine-grained re-scoring method. We compute cross-modal similarities between query and video frames using a pre-trained CLIP model, with multi-frame scores aggregated by max pooling. The fine-grained score is weightedly added to the initial score for search result reranking. As such, CLIPRerank is agnostic to the underlying video retrieval models and extremely simple, making it a handy plug-in for boosting AVS. Experiments on the challenging TRECVID AVS benchmarks (from 2016 to 2021) justify the effectiveness of the proposed strategy. CLIPRerank consistently improves the TRECVID top performers and multiple existing models including SEA, W2VV++, Dual Encoding, Dual Task, LAFF, CLIP2Video, TS2-Net and X-CLIP. Our method also works when substituting BLIP-2 for CLIP.

• The paper introduces CLIPRerank, a method that refines video search by aggregating frame-level text-video similarity scores using a pre-trained CLIP model.
• It demonstrates notable improvements, with performance gains exceeding 10% on TRECVID benchmarks across various retrieval models.
• The approach is model-agnostic and computationally efficient, enabling easy integration with existing ad-hoc video search systems.

The paper presents "CLIPRerank," a methodology aimed at enhancing Ad-hoc Video Search (AVS) by incorporating frame-level similarity scores between textual queries and video frames, derived using a pre-trained CLIP model. The AVS task involves querying unlabeled video content with text input, necessitating a fine-grained approach due to the variable relevance of video segments in response to diverse queries.

The CLIPRerank methodology addresses the limitations of traditional holistic similarity measures, which may fail when only segments of a video are relevant to the query. By leveraging cross-modal similarities computed with CLIP for individual video frames, and aggregating these using max pooling, CLIPRerank refines and reranks the initial search results from various baseline video retrieval models. This approach is model-agnostic, providing an easy-to-implement augmentation to existing systems.

The evaluation conducted on the TRECVID benchmark datasets from 2016 to 2021 showcases the efficacy of CLIPRerank in improving performance across a variety of retrieval models, including top-performing systems and baseline models such as SEA, W2VV++, Dual Encoding, and LAFF. The paper demonstrates consistent improvement across all models, with some instances showing over 10% enhancement in performance metrics. Additionally, the approach shows robustness when substituting other vision-LLMs, such as BLIP-2, for CLIP.

The authors highlight the computational efficiency of CLIPRerank, asserting that re-ranking via this method incurs minimal overhead in terms of processing time. Furthermore, the experimental results underscore the significant benefits of employing frame-level text-video similarity over holistic matching strategies, advocating for fine-grained scoring enhancements in AVS. The methodology not only augments existing models but also provides insight into the potential application of large vision-LLMs like CLIP in improving search result relevance in a straightforward manner.