MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks (2310.04965v2)
Abstract: Automatically generating scripts (i.e. sequences of key steps described in text) from video demonstrations and reasoning about the subsequent steps are crucial to the modern AI virtual assistants to guide humans to complete everyday tasks, especially unfamiliar ones. However, current methods for generative script learning rely heavily on well-structured preceding steps described in text and/or images or are limited to a certain domain, resulting in a disparity with real-world user scenarios. To address these limitations, we present a new benchmark challenge -- MultiScript, with two new tasks on task-oriented multimodal script learning: (1) multimodal script generation, and (2) subsequent step prediction. For both tasks, the input consists of a target task name and a video illustrating what has been done to complete the target task, and the expected output is (1) a sequence of structured step descriptions in text based on the demonstration video, and (2) a single text description for the subsequent step, respectively. Built from WikiHow, MultiScript covers multimodal scripts in videos and text descriptions for over 6,655 human everyday tasks across 19 diverse domains. To establish baseline performance on MultiScript, we propose two knowledge-guided multimodal generative frameworks that incorporate the task-related knowledge prompted from LLMs such as Vicuna. Experimental results show that our proposed approaches significantly improve over the competitive baselines.
- Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
- Scaling Egocentric Vision: The EPIC-KITCHENS Dataset. In European Conference on Computer Vision (ECCV).
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100. International Journal of Computer Vision (IJCV), 130: 33–55.
- Meteor Universal: Language Specific Translation Evaluation for Any Target Language. 376–380.
- Anticipative Video Transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 13505–13515.
- Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
- Improved Instruction Ordering in Recipe-Grounded Conversation. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 10086–10104. Toronto, Canada: Association for Computational Linguistics.
- In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video. In Computer Vision – ECCV 2018: 15th European Conference, Munich, Germany, September 8–14, 2018, Proceedings, Part V, 639–655. Berlin, Heidelberg: Springer-Verlag. ISBN 978-3-030-01227-4.
- Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. 74–81.
- Learning To Recognize Procedural Activities With Distant Supervision. 13853–13863.
- UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation.
- Reasoning about Goals, Steps, and Temporal Ordering with WikiHow. EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 4630–4639.
- Goal-oriented script construction. arXiv preprint arXiv:2107.13189.
- Katna: Tool for automating video keyframe extraction, video compression, Image Autocrop and Smart image resize tasks.
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips. In ICCV.
- Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2630–2640.
- TL;DW? Summarizing Instructional Videos with Task Relevance and Cross-Modal Saliency. LNCS, 13694: 540–557.
- Bleu: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02, 311–318.
- Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 8494–8502.
- The Art of SOCRATIC QUESTIONING: Recursive Thinking with Large Language Models. In Bouamor, H.; Pino, J.; and Bali, K., eds., Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 4177–4199. Singapore: Association for Computational Linguistics.
- Predicting the Next Action by Modeling the Abstract Goal.
- Nits-vc system for vatex video captioning challenge 2020. arXiv preprint arXiv:2006.04058.
- Coin: A large-scale dataset for comprehensive instructional video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1207–1216.
- Augmented SBERT: Data Augmentation Method for Improving Bi-Encoders for Pairwise Sentence Scoring Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 296–310. Online: Association for Computational Linguistics.
- Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework. arXiv preprint arXiv:2202.03052.
- Multimedia Generative Script Learning for Task Planning.
- Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European conference on computer vision (ECCV), 305–321.
- ExpertPrompting: Instructing Large Language Models to be Distinguished Experts. arXiv preprint arXiv:2305.14688.
- Vision-Flan:Scaling Visual Instruction Tuning.
- MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11445–11465. Toronto, Canada: Association for Computational Linguistics.
- Visual Goal-Step Inference using wikiHow. EMNLP 2021 - 2021 Conference on Empirical Methods in Natural Language Processing, Proceedings, 2167–2179.
- Visual Goal-Step Inference using wikiHow. arXiv preprint arXiv:2104.05845.
- Reasoning about Goals, Steps, and Temporal Ordering with WikiHow. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 4630–4639.
- BERTScore: Evaluating Text Generation with BERT.
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685.
- Anticipative Feature Fusion Transformer for Multi-Modal Action Anticipation. arXiv preprint arXiv:2210.12649.
- Jingyuan Qi (8 papers)
- Minqian Liu (15 papers)
- Ying Shen (76 papers)
- Zhiyang Xu (29 papers)
- Lifu Huang (92 papers)