Lighthouse: A User-Friendly Library for Reproducible Video Moment Retrieval and Highlight Detection

Abstract: We propose Lighthouse, a user-friendly library for reproducible video moment retrieval and highlight detection (MR-HD). Although researchers proposed various MR-HD approaches, the research community holds two main issues. The first is a lack of comprehensive and reproducible experiments across various methods, datasets, and video-text features. This is because no unified training and evaluation codebase covers multiple settings. The second is user-unfriendly design. Because previous works use different libraries, researchers set up individual environments. In addition, most works release only the training codes, requiring users to implement the whole inference process of MR-HD. Lighthouse addresses these issues by implementing a unified reproducible codebase that includes six models, three features, and five datasets. In addition, it provides an inference API and web demo to make these methods easily accessible for researchers and developers. Our experiments demonstrate that Lighthouse generally reproduces the reported scores in the reference papers. The code is available at https://github.com/line/lighthouse.

• The paper demonstrates that Lighthouse integrates six models and five datasets to reproduce video moment retrieval and highlight detection results.
• It streamlines experimental reproducibility with a YAML configuration system and an accessible inference API, addressing fragmented setups.
• The framework features a web demo for visual validation, encouraging broader adoption and innovative multimodal research.

Understanding Lighthouse: A Library for Video Moment Retrieval and Highlight Detection

The paper introduces Lighthouse, an integrative library designed to address persistent challenges in the field of video moment retrieval and highlight detection (MR-HD). This research provides a new synthesis of multiple approaches, features, and datasets, delivering a unified platform for both reproducible experiments and easy deployment. The authors identify two primary issues hindering advancement in MR-HD: a lack of comprehensive experimental comparability across methods and datasets, and a user experience fragmented by disparate development environments. Lighthouse is positioned as a remedy, offering an all-encompassing codebase that integrates six models, three features, and five datasets. The implementation includes an inference API and a web demo, broadening accessibility for researchers and developers.

Key Achievements of Lighthouse

Lighthouse is a comprehensive framework supporting MR, HD, and the combined task of MR-HD. The library's key achievements fall under two main aspects: reproducibility and user-friendly design.

1. Reproducibility:
   • Lighthouse integrates multiple existing MR-HD methodologies, harmonizing video and text encodings—primarily using CLIP, SlowFast, ResNet152, and GloVe—as part of its feature extractors.
   • The framework provides an architecture for reproducible training and testing, utilizing a configuration system in YAML format, allowing experiments to be replicated or altered by specifying different settings.
   • Comparing Lighthouse scores with those reported in original papers indicates that the library successfully reproduces stated results. It suggests that using sequential motion information (as with CLIP+Slowfast) yields better outcomes compared to static frame analysis, affirming the benefit of CLIP in MR-HD tasks.
2. User-friendly Design:
   • The library can be easily installed without cumbersome setup procedures, contrasting with the fragmented approach of previous works.
   • An inference API provides higher-level access to MR-HD processes, enabling researchers to handle complex video-text interaction tasks without deep expertise in these specific areas.
   • Lighthouse includes a web demo enabling visual inspection of MR-HD outcomes, promoting interaction with and validation of model outputs.

Experimental Evaluation

The effectiveness of Lighthouse is demonstrated through empirical evaluations. The study includes comprehensive experiments showing that recent MR-HD methods do not consistently surpass earlier approaches across various datasets and feature configurations. This underscores the importance of Lighthouse's all-inclusive testing framework, which simplifies the explorative process of determining optimal methodologies.

On diverse datasets such as QVHighlights, ActivityNet Captions, Charades-STA, TaCoS, and TVSum, Lighthouse illustrates both individual task applicability for MR and HD, and the combined efficacy in joint MR-HD tasks. The paper presents a detailed performance comparison of supported methods, reflecting robust model performances and confirming that Lighthouse achieves competitive outcomes aligned with previous leading MR-HD methods.

Implications and Future Directions

The open-source availability of Lighthouse under the Apache 2.0 license signifies an effort to propel further research by offering a shared platform for experimental reproducibility and expansion. By supporting an extensive combination of settings with simplified evaluation and comparison capabilities, Lighthouse paves the way for enhanced experimental transparency, potentially catalyzing new advancements in MR-HD fields.

The research community is encouraged to leverage Lighthouse to drive explorative analyses across multiple tasks, which may lead to innovative MR-HD applications across various domains where content retrieval speed and accuracy are crucial. Researchers might explore integrating Lighthouse in multimodal AI systems, examining synergy between language and vision under dynamic interaction conditions, potentially crafting more nuanced video content retrieval applications.

In summary, Lighthouse addresses critical reproducibility and usability concerns within MR-HD research, enhancing the robustness of model comparisons and facilitating wider developer access to cutting-edge video analysis tools.