The Solution for the ICCV 2023 Perception Test Challenge 2023 -- Task 6 -- Grounded videoQA
Abstract: In this paper, we introduce a grounded video question-answering solution. Our research reveals that the fixed official baseline method for video question answering involves two main steps: visual grounding and object tracking. However, a significant challenge emerges during the initial step, where selected frames may lack clearly identifiable target objects. Furthermore, single images cannot address questions like "Track the container from which the person pours the first time." To tackle this issue, we propose an alternative two-stage approach:(1) First, we leverage the VALOR model to answer questions based on video information.(2) concatenate the answered questions with their respective answers. Finally, we employ TubeDETR to generate bounding boxes for the targets.
- Valor: Vision-audio-language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345, 2023.
- Mdetr–modulated detection for end-to-end multi-modal understanding, 2021. https://arxiv.org/pdf/2104.12763.pdf.
- Violet: End-to-end video-language transformers with masked visual-token modeling.
- Roberta: A robustly optimized bert pretraining approach. Cornell University - arXiv,Cornell University - arXiv, Jul 2019.
- Tubedetr: Spatio-temporal video grounding with transformers, 2022. https://arxiv.org/pdf/2203.16434.pdf.
- Semi-supervised multi-modal multi-instance multi-label deep network with optimal transport. IEEE Transactions on Knowledge and Data Engineering, 33(2):696–709, 2019.
- Towards global video scene segmentation with context-aware transformer. Proceedings of the AAAI conference on artificial intelligence, 37(3):3206–3213, 2023.
- Comprehensive semi-supervised multi-modal learning. In IJCAI, pages 4092–4098, 2019.
- Complex object classification: A multi-modal multi-instance multi-label deep network with optimal transport. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2594–2603, 2018.
- Corporate relative valuation using heterogeneous multi-modal graph neural network. IEEE Transactions on Knowledge and Data Engineering, 35(1):211–224, 2021.
- Domfn: A divergence-orientated multi-modal fusion network for resume assessment. Proceedings of the 30th ACM International Conference on Multimedia, pages 1612–1620, 2022.
- Adaptive deep models for incremental learning: Considering capacity scalability and sustainability. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 74–82, 2019.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.