2000 character limit reached
MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning (2402.13625v2)
Published 21 Feb 2024 in cs.CL
Abstract: Since commonsense information has been recorded significantly less frequently than its existence, LLMs pre-trained by text generation have difficulty to learn sufficient commonsense knowledge. Several studies have leveraged text retrieval to augment the models' commonsense ability. Unlike text, images capture commonsense information inherently but little effort has been paid to effectively utilize them. In this work, we propose a novel Multi-mOdal REtrieval (MORE) augmentation framework, to leverage both text and images to enhance the commonsense ability of LLMs. Extensive experiments on the Common-Gen task have demonstrated the efficacy of MORE based on the pre-trained models of both single and multiple modalities.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- Spice: Semantic propositional image caption evaluation. ArXiv, abs/1607.08822.
- Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In IEEvaluation@ACL.
- Optimizing retrieval-augmented reader models via token elimination. arXiv preprint arXiv:2310.13682.
- Paul Bloom. 2002. How children learn the meanings of words. MIT press.
- Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.
- A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing.
- Retrieve, caption, generate: Visual grounding for enhancing commonsense in text generation models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10618–10626.
- Linda B. Gambrell and Ruby J. Bales. 1986. Mental imagery and the comprehension-monitoring performance of fourth- and fifth-grade poor readers. Reading Research Quarterly, 21:454.
- Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Conference on Automated Knowledge Base Construction.
- Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill.
- Visually-augmented pretrained language models for nlp tasks without images. ArXiv, abs/2212.07937.
- Conceptnet 3: a flexible, multilingual semantic network for common sense knowledge. In Recent advances in natural language processing, pages 27–29. John Benjamins Philadelphia, PA.
- Metric-guided distillation: Distilling knowledge from the metric to ranker and retriever for generative commonsense reasoning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 839–852.
- Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685.
- Gautier Izacard and Édouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880.
- Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299.
- Comprehension problems in children with specific language impairment: Does mental imagery training help? International Journal of Language & Communication Disorders, 42(6):648–664.
- Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706–715.
- The power of scale for parameter-efficient prompt tuning. In Conference on Empirical Methods in Natural Language Processing.
- Kfcnet: Knowledge filtering and contrastive learning for generative commonsense reasoning. In Conference on Empirical Methods in Natural Language Processing.
- Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning.
- Commongen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823–1840.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics.
- Microsoft coco: Common objects in context. In European Conference on Computer Vision.
- Tcra-llm: Token compression retrieval augmented large language model for inference cost reduction. arXiv preprint arXiv:2310.15556.
- Gpt understands, too. ArXiv, abs/2103.10385.
- Kgr4: Retrieval, retrospect, refine and rethink for commonsense generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11029–11037.
- Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations.
- Discriminability objective for training descriptive captions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964–6974.
- When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, Toronto, Canada. Association for Computational Linguistics.
- OpenAI. 2022. Introducing chatgpt. https://openai.com/blog/chatgpt.
- OpenAI. 2023a. Gpt-3.5 turbo fine-tuning and api updates. https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates.
- OpenAI. 2023b. Gpt-4. https://openai.com/research/gpt-4.
- Bleu: a method for automatic evaluation of machine translation. In Annual Meeting of the Association for Computational Linguistics.
- Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.
- Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
- In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.
- The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends® in Information Retrieval, 3(4):333–389.
- Replug: Retrieval-augmented black-box language models. arXiv preprint arXiv:2301.12652.
- Prompting gpt-3 to be reliable. In The Eleventh International Conference on Learning Representations.
- Learning to imagine: Visually-augmented natural language generation. ArXiv, abs/2305.16944.
- Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
- Attention is all you need. Advances in neural information processing systems, 30.
- Cider: Consensus-based image description evaluation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566–4575.
- Contextualized scene imagination for generative commonsense reasoning. In International Conference on Learning Representations.
- Vatex: A large-scale, high-quality multilingual dataset for video-and-language research. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4580–4590.
- A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computational Linguistics.
- Recomp: Improving retrieval-augmented lms with compression and selective augmentation. arXiv preprint arXiv:2310.04408.
- Retrieval augmentation for commonsense reasoning: A unified approach. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4364–4377.
- Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068.
- Visualize before you write: Imagination-guided open-ended text generation. arXiv preprint arXiv:2210.03765.
Sponsor
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.