Papers
Topics
Authors
Recent
Search
2000 character limit reached

Open-Ended Multi-Modal Relational Reasoning for Video Question Answering

Published 1 Dec 2020 in cs.AI, cs.HC, and cs.RO | (2012.00822v4)

Abstract: In this paper, we introduce a robotic agent specifically designed to analyze external environments and address participants' questions. The primary focus of this agent is to assist individuals using language-based interactions within video-based scenes. Our proposed method integrates video recognition technology and natural language processing models within the robotic agent. We investigate the crucial factors affecting human-robot interactions by examining pertinent issues arising between participants and robot agents. Methodologically, our experimental findings reveal a positive relationship between trust and interaction efficiency. Furthermore, our model demonstrates a 2\% to 3\% performance enhancement in comparison to other benchmark methods.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. L. Jin, H. Zhang, Y. Shen, and C. Ye, “Human-robot interaction for assisted object grasping by a wearable robotic object manipulation aid for the blind,” in 2020 IEEE International Conference on Human-Machine Systems (ICHMS), 2020, pp. 1–6.
  2. N. Robinson, B. Tidd, D. Campbell, D. Kulić, and P. Corke, “Robotic vision for human-robot interaction and collaboration: A survey and systematic review,” J. Hum.-Robot Interact., vol. 12, no. 1, feb 2023.
  3. C. Ye, S. Hong, X. Qian, and W. Wu, “Co-robotic cane: A new robotic navigation aid for the visually impaired,” IEEE Systems, Man, and Cybernetics Magazine, vol. 2, no. 2, pp. 33–42, 2016.
  4. L. O. Russo, G. A. Farulla, D. Pianu, A. R. Salgarella, M. Controzzi, C. Cipriani, C. M. Oddo, C. Geraci, S. Rosa, and M. Indaco, “Parloma – a novel human-robot interaction system for deaf-blind remote communication,” International Journal of Advanced Robotic Systems, vol. 12, no. 5, p. 57, 2015.
  5. B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, “Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers,” 2021.
  6. M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng, et al., “Ros: an open-source robot operating system,” in ICRA workshop on open source software, vol. 3, no. 3.2.   Kobe, Japan, 2009, p. 5.
  7. D. Tran, H. Wang, L. Torresani, J. Ray, Y. LeCun, and M. Paluri, “A closer look at spatiotemporal convolutions for action recognition,” 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6450–6459, 2018.
  8. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016, pp. 770–778.
  9. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018.
  10. Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. R. Salakhutdinov, and Q. V. Le, “Xlnet: Generalized autoregressive pretraining for language understanding,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32.   Curran Associates, Inc., 2019, pp. 5753–5763.
  11. K. Yi, J. Wu, C. Gan, A. Torralba, P. Kohli, and J. B. Tenenbaum, “Neural-symbolic vqa: Disentangling reasoning from vision and language understanding,” in Proceedings of the 32nd International Conference on Neural Information Processing Systems, ser. NIPS’18.   Red Hook, NY, USA: Curran Associates Inc., 2018, p. 1039–1050.
  12. H. Luo, N. Liu, and C. Feng, “Question classification with deep contextualized transformer,” arXiv preprint arXiv:1910.10492, 2019.
  13. Y. Hou, W. Che, Y. Lai, Z. Zhou, Y. Liu, H. Liu, and T. Liu, “Few-shot slot tagging with collapsed dependency transfer and label-enhanced task-adaptive projection network,” 2020.
  14. Z. Zhou, D. Zhang, W. Xiao, N. Dingwall, X. Ma, A. O. Arnold, and B. Xiang, “Learning dialogue representations from consecutive utterances,” 2022.
  15. T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean, “Distributed representations of words and phrases and their compositionality,” in Advances in Neural Information Processing Systems, C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds., vol. 26.   Curran Associates, Inc., 2013, pp. 3111–3119.
  16. M. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).   New Orleans, Louisiana: Association for Computational Linguistics, June 2018, pp. 2227–2237.
  17. K. Samel, Z. Zhao, B. Chen, K. Wang, R. Luo, and L. Song, “How to design sample and computationally efficient vqa models,” 2021.
  18. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, u. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems, ser. NIPS’17.   Red Hook, NY, USA: Curran Associates Inc., 2017, p. 6000–6010.
  19. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe, “Training language models to follow instructions with human feedback,” in Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35.   Curran Associates, Inc., 2022, pp. 27 730–27 744.
  20. J. Lu, D. Batra, D. Parikh, and S. Lee, “Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks,” in Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, Eds., vol. 32.   Curran Associates, Inc., 2019, pp. 13–23.
  21. R. Qin, H. Luo, Z. Fan, and Z. Ren, “Ibert: Idiom cloze-style reading comprehension with attention,” 2021.
  22. M. Liu, H. Luo, L. Thong, Y. Li, C. Zhang, and L. Song, “Sciannotate: A tool for integrating weak labeling sources for sequence labeling,” 2022.
  23. Y. Ji, Z. Zhou, H. Liu, and R. V. Davuluri, “DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome,” Bioinformatics, vol. 37, no. 15, pp. 2112–2120, 02 2021.
  24. R. Girdhar and D. Ramanan, “Cater: A diagnostic dataset for compositional actions and temporal reasoning,” ArXiv, vol. abs/1910.04744, 2020.
  25. K. He, G. Gkioxari, P. Dollár, and R. B. Girshick, “Mask r-cnn,” 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2980–2988, 2017.
  26. S. Li, “3d-detnet: a single stage video-based vehicle detector,” in International Workshop on Pattern Recognition, 2018.
  27. J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017, pp. 4724–4733.
  28. A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox, “Flownet: Learning optical flow with convolutional networks,” in 2015 IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2758–2766.
  29. K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” 2015.
  30. K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, “Lstm: A search space odyssey,” IEEE transactions on neural networks and learning systems, vol. 28, no. 10, pp. 2222–2232, 2016.
  31. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervision,” 2022.
  32. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” 2017.
  33. J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” 2016.
  34. P. Shaw, J. Uszkoreit, and A. Vaswani, “Self-attention with relative position representations,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers).   New Orleans, Louisiana: Association for Computational Linguistics, June 2018, pp. 464–468.
  35. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019.
  36. J. J. Garrett et al., “Ajax: A new approach to web applications,” 2005.
  37. J. Lei, L. Yu, M. Bansal, and T. Berg, “TVQA: Localized, compositional video question answering,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.   Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 1369–1379.
  38. J. Kim, M. Ma, K. Kim, S. Kim, and C. Yoo, “Progressive attention memory network for movie story question answering,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8329–8338, 2019.
  39. J. Kim, M. Ma, K. Kim, S. Kim, and C. D. Yoo, “Gaining extra supervision via multi-task learning for multi-modal video question answering,” in 2019 International Joint Conference on Neural Networks (IJCNN).   IEEE, 2019, pp. 1–8.
  40. Z. Yang, N. Garcia, C. Chu, M. Otani, Y. Nakashima, and H. Takemura, “Bert representations for video question answering,” in 2020 IEEE Winter Conference on Applications of Computer Vision (WACV), March 2020, pp. 1545–1554.
  41. A. Chadha, G. Arora, and N. Kaloty, “iperceive: Applying common-sense reasoning to multi-modal dense video captioning and video question answering,” arXiv preprint arXiv:2011.07735, 2020.
  42. Z. Yang, D. Yang, C. Dyer, X. He, A. Smola, and E. Hovy, “Hierarchical attention networks for document classification,” in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.   San Diego, California: Association for Computational Linguistics, June 2016, pp. 1480–1489.
Citations (3)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up