Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
80 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Multi-Modal Domain Adaptation Across Video Scenes for Temporal Video Grounding (2312.13633v1)

Published 21 Dec 2023 in cs.CV

Abstract: Temporal Video Grounding (TVG) aims to localize the temporal boundary of a specific segment in an untrimmed video based on a given language query. Since datasets in this domain are often gathered from limited video scenes, models tend to overfit to scene-specific factors, which leads to suboptimal performance when encountering new scenes in real-world applications. In a new scene, the fine-grained annotations are often insufficient due to the expensive labor cost, while the coarse-grained video-query pairs are easier to obtain. Thus, to address this issue and enhance model performance on new scenes, we explore the TVG task in an unsupervised domain adaptation (UDA) setting across scenes for the first time, where the video-query pairs in the source scene (domain) are labeled with temporal boundaries, while those in the target scene are not. Under the UDA setting, we introduce a novel Adversarial Multi-modal Domain Adaptation (AMDA) method to adaptively adjust the model's scene-related knowledge by incorporating insights from the target data. Specifically, we tackle the domain gap by utilizing domain discriminators, which help identify valuable scene-related features effective across both domains. Concurrently, we mitigate the semantic gap between different modalities by aligning video-query pairs with related semantics. Furthermore, we employ a mask-reconstruction approach to enhance the understanding of temporal semantics within a scene. Extensive experiments on Charades-STA, ActivityNet Captions, and YouCook2 demonstrate the effectiveness of our proposed method.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (53)
  1. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, 961–970.
  2. Rethinking the bottom-up framework for query-based video localization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 10551–10558.
  3. Temporal attentive alignment for large-scale video domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 6321–6330.
  4. Multi-level attentive adversarial learning with temporal dilation for unsupervised video domain adaptation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1259–1268.
  5. Mind-the-Gap! Unsupervised Domain Adaptation for Text-Video Retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, 1072–1080.
  6. Look closer to ground better: Weakly-supervised temporal grounding of sentence in video. arXiv preprint arXiv:2001.09308.
  7. Multi-modal cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia.
  8. Domain-adversarial training of neural networks. The journal of machine learning research, 17(1): 2096–2030.
  9. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision, 5267–5275.
  10. Mac: Mining activity concepts for language-based temporal localization. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), 245–253. IEEE.
  11. Deep Domain Adaptation in Action Space. In BMVC, volume 2, 5.
  12. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, 706–715.
  13. Weakly-supervised video moment retrieval via semantic completion network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 11539–11546.
  14. Context-aware biaffine localizing network for temporal sentence grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11235–11244.
  15. Learning transferable features with deep adaptation networks. In International conference on machine learning, 97–105. PMLR.
  16. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31.
  17. Deep transfer learning with joint adaptation networks. In International conference on machine learning, 2208–2217. PMLR.
  18. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983.
  19. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  20. Unsupervised domain adaptation via discriminative manifold propagation. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  21. Weakly supervised video moment retrieval from text queries. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 11592–11601.
  22. Local-global video-text interactions for temporal grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10810–10819.
  23. Multi-modal domain adaptation for fine-grained action recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 122–132.
  24. Domain Adaptation in Multi-View Embedding for Cross-Modal Video Retrieval. arXiv preprint arXiv:2110.12812.
  25. Interventional video grounding with dual contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2765–2775.
  26. Domain adaptation via transfer component analysis. IEEE transactions on neural networks, 22(2): 199–210.
  27. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), 1532–1543.
  28. Grounding action descriptions in videos. Transactions of the Association for Computational Linguistics, 1: 25–36.
  29. Generalized conditional domain adaptation: A causal perspective with low-rank translators. IEEE transactions on cybernetics, 50(2): 821–834.
  30. Proposal-free temporal moment localization of a natural-language query in video using guided attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2464–2473.
  31. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, 510–526. Springer.
  32. Correlation alignment for unsupervised domain adaptation. In Domain Adaptation in Computer Vision Applications, 153–171. Springer.
  33. Deep coral: Correlation alignment for deep domain adaptation. In European conference on computer vision, 443–450. Springer.
  34. Regularized two granularity loss function for weakly supervised video moment retrieval. IEEE Transactions on Multimedia.
  35. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, 4489–4497.
  36. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, 7167–7176.
  37. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474.
  38. Temporally grounding language queries in videos by contextual boundary-aware prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 12168–12175.
  39. Scene-robust natural language video localization via learning domain-invariant representations. In Findings of the Association for Computational Linguistics: ACL 2023, 144–160.
  40. Multilevel language and vision integration for text-to-clip retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 9062–9069.
  41. Larger norm more transferable: An adaptive feature norm approach for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 1426–1435.
  42. Aligning correlation information for domain adaptation in action recognition. IEEE Transactions on Neural Networks and Learning Systems.
  43. HiSA: Hierarchically semantic associating for video temporal grounding. IEEE Transactions on Image Processing, 31: 5178–5188.
  44. Heterogeneous graph attention network for unsupervised multiple-target domain adaptation. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  45. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541.
  46. Dense regression network for video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 10287–10296.
  47. Span-based localizing network for natural language video localization. arXiv preprint arXiv:2004.13931.
  48. Learning 2d temporal adjacent networks for moment localization with natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, 12870–12877.
  49. Cross-modal interaction networks for query-based moment retrieval in videos. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, 655–664.
  50. Cascaded prediction network via segment tree for temporal video grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 4197–4206.
  51. Weakly supervised temporal sentence grounding with gaussian-based contrastive proposal learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 15555–15564.
  52. Towards automatic learning of procedures from web instructional videos. In Thirty-Second AAAI Conference on Artificial Intelligence.
  53. Deep subdomain adaptation network for image classification. IEEE transactions on neural networks and learning systems, 32(4): 1713–1722.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Haifeng Huang (20 papers)
  2. Yang Zhao (382 papers)
  3. Zehan Wang (37 papers)
  4. Yan Xia (169 papers)
  5. Zhou Zhao (218 papers)
Citations (1)
View on Semantic Scholar