Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
156 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models (2312.17661v1)

Published 29 Dec 2023 in cs.CL, cs.AI, and cs.CV

Abstract: The burgeoning interest in Multimodal LLMs (MLLMs), such as OpenAI's GPT-4V(ision), has significantly impacted both academic and industrial realms. These models enhance LLMs with advanced visual understanding capabilities, facilitating their application in a variety of multimodal tasks. Recently, Google introduced Gemini, a cutting-edge MLLM designed specifically for multimodal integration. Despite its advancements, preliminary benchmarks indicate that Gemini lags behind GPT models in commonsense reasoning tasks. However, this assessment, based on a limited dataset (i.e., HellaSWAG), does not fully capture Gemini's authentic commonsense reasoning potential. To address this gap, our study undertakes a thorough evaluation of Gemini's performance in complex reasoning tasks that necessitate the integration of commonsense knowledge across modalities. We carry out a comprehensive analysis of 12 commonsense reasoning datasets, ranging from general to domain-specific tasks. This includes 11 datasets focused solely on language, as well as one that incorporates multimodal elements. Our experiments across four LLMs and two MLLMs demonstrate Gemini's competitive commonsense reasoning capabilities. Additionally, we identify common challenges faced by current LLMs and MLLMs in addressing commonsense problems, underscoring the need for further advancements in enhancing the commonsense reasoning abilities of these models.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. An in-depth look at gemini’s language abilities. arXiv preprint arXiv:2312.11444.
  2. Abductive commonsense reasoning. In International Conference on Learning Representations.
  3. Prajjwal Bhargava and Vincent Ng. 2022. Commonsense knowledge reasoning and generation with pre-trained language models: A survey. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 12317–12325.
  4. Chatgpt is a knowledgeable but inexperienced solver: An investigation of commonsense problem in large language models. arXiv preprint arXiv:2303.16421.
  5. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.
  6. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  7. A challenger to gpt-4v? early explorations of gemini in visual expertise. arXiv preprint arXiv:2312.12436.
  8. Large language models as zero-shot conversational recommenders. In Proceedings of the 32nd ACM international conference on information and knowledge management, pages 720–730.
  9. Aligning ai with shared human values. In International Conference on Learning Representations.
  10. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391–2401.
  11. Mvp-tuning: Multi-view knowledge retrieval with prompt tuning for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13417–13432.
  12. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274.
  13. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2.
  14. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082–8090.
  15. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213.
  16. Do language models learn commonsense knowledge? arXiv preprint arXiv:2111.00607.
  17. A comprehensive evaluation of gpt-4v on knowledge-intensive visual question answering. arXiv preprint arXiv:2311.07536.
  18. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862–6868.
  19. Riddlesense: Reasoning about riddle questions featuring linguistic creativity and commonsense knowledge. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1504–1515.
  20. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773.
  21. Hugo Liu and Push Singh. 2004. Conceptnet—a practical commonsense reasoning tool-kit. BT technology journal, 22(4):211–226.
  22. Mengchen Liu and Chongyan Chen. 2023. An evaluation of gpt-4v and gemini in online vqa. arXiv preprint arXiv:2312.10637.
  23. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  24. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys.
  25. OpenAI. 2023. Gpt-4 technical report.
  26. Artificial general intelligence: Roadmap to achieving human-level capabilities.
  27. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4463–4473.
  28. An experimental study measuring the generalization of fine-tuned language representation models across commonsense reasoning benchmarks. Expert Systems, page e13243.
  29. Vered Shwartz and Yejin Choi. 2020. Do neural language models overcome reporting bias? In Proceedings of the 28th International Conference on Computational Linguistics, pages 6863–6870.
  30. Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches. arXiv preprint arXiv:1904.01172, pages 1–60.
  31. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4149–4158.
  32. Pre-training is (almost) all you need: An application to commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3878–3887.
  33. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  34. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  35. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  36. Yuqing Wang and Yun Zhao. 2023a. Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05342.
  37. Yuqing Wang and Yun Zhao. 2023b. Tram: Benchmarking temporal reasoning for large language models. arXiv preprint arXiv:2310.00835.
  38. Integrating physiological time series and clinical notes with transformer for early prediction of sepsis. arXiv preprint arXiv:2203.14469.
  39. Enhancing transformer efficiency for multivariate time series classification. arXiv preprint arXiv:2203.14472.
  40. Are large language models ready for healthcare? a comparative study on clinical language understanding. arXiv preprint arXiv:2304.05368.
  41. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
  42. Large language models are better reasoners with self-verification. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2550–2575.
  43. Can gpt-4v (ision) serve medical applications? case studies on gpt-4v for multimodal medical diagnosis. arXiv preprint arXiv:2310.09909.
  44. Multimodal large language models: A survey. arXiv preprint arXiv:2311.13165.
  45. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 9(1).
  46. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601.
  47. Improving commonsense in vision-language models via knowledge graph riddles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2634–2645.
  48. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731.
  49. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800.
  50. A survey of large language models. arXiv preprint arXiv:2303.18223.
  51. Empirical quantitative analysis of covid-19 forecasting models. In 2021 International Conference on Data Mining Workshops (ICDMW), pages 517–526. IEEE.
  52. Commonsense knowledge transfer for pre-trained language models. arXiv preprint arXiv:2306.02388.
Citations (13)
View on Semantic Scholar

Summary

  • The paper evaluates Gemini's commonsense reasoning across diverse domains using 12 datasets and varied prompting techniques.
  • The paper finds that Gemini marginally outperforms GPT-3.5 Turbo in language-based tasks while struggling with temporal, social, and emotional cues in multimodal settings.
  • The paper highlights challenges in robust multimodal reasoning, suggesting clear avenues for future improvements in AI commonsense understanding.

Introduction

The paper presents an evaluation of a cutting-edge multimodal LLM (MLLM) known as Gemini, particularly focusing on its commonsense reasoning abilities. Commonsense reasoning is a core cognitive skill that humans use daily to make sense of ordinary situations and complex tasks. This proficiency is challenging to replicate in NLP systems. The research's central drive is to test Gemini's performance extensively and highlight the common challenges current LLMs and MLLMs face in commonsense tasks.

Commonsense Overview

Commonsense reasoning involves an array of domains including general, contextual, temporal, physical and numerical understanding, as well as social interactions and moral judgments. These aspects cover intuitive human understanding, predicting scenarios based on cause and effect, recognizing social cues, ethical reasoning, and interpreting visual information. It's crucial for AI systems to navigate these domains effectively to mirror human understanding and interaction.

Experimental Setup

The empirical paper encompasses the evaluation of Gemini against twelve different datasets. Four popular LLMs are assessed for language-based tasks, while two MLLMs are scrutinized for multimodal tasks. These tasks span various domains of commonsense reasoning such as general, specialized, social, ethical, and visual understanding, with a performance metric of accuracy used across all datasets. Different setups like zero-shot standard prompting and few-shot chain-of-thought prompting are employed to understand the inherent and enhanced commonsense capabilities of the models.

Results and Limitations

Findings reveal that Gemini's performance is akin to GPT-3.5 Turbo, outperforming it marginally in language-based commonsense reasoning tasks, but lagging behind GPT-4 Turbo. While demonstrating a sturdy understanding of most tested domains, Gemini struggles with temporal and social reasoning, as well as discerning emotion in images. Despite its notable logical reasoning, it often misunderstands context. The paper acknowledges limitations, such as the language and dataset scope potentially not covering all commonsense facets, and the resulting outcomes being bound to evolving AI capabilities.

Discussion

This comprehensive assessment indicates significant progress in AI's ability to reason with commonsense knowledge, yet highlights the nuanced and context-dependent nature of human reasoning remains a tough challenge. The area of multimodal reasoning, combining visual cues with language understanding, is still notably challenging. The detailed examination of the performance across diverse datasets provides valuable insights into the strengths and weaknesses of current LLMs and MLLMs, suggesting a path forward for improvements in natural, robust AI comprehension and interaction.

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube

Reddit Logo Streamline Icon: https://streamlinehq.com

Reddit

  1. [2312.17661] Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language Models (58 points, 19 comments)