Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
72 tokens/sec
GPT-4o
61 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria (2311.13951v3)

Published 23 Nov 2023 in cs.CL

Abstract: Multimodal LLMs (MLLMs) have broadened the scope of AI applications. Existing automatic evaluation methodologies for MLLMs are mainly limited in evaluating queries without considering user experiences, inadequately addressing the nuances of creative and associative multimodal tasks. However, the open-ended and subjective nature of such tasks poses a significant challenge to the evaluation methodology, where it is difficult to define the ground-truth answers for them. To this end, in our paper, we propose a new evaluation paradigm for MLLMs, which is evaluating MLLMs with per-sample criteria using potent MLLM as the judge. To validate the feasibility and effectiveness of this paradigm, we design a benchmark, dubbed MLLM-Bench, by curating the evaluation samples across six comprehensive cognitive levels. We benchmark 21 popular MLLMs in a pairwise-comparison fashion, showing diverse performance across models. Moreover, the validity of our benchmark manifests itself in reaching 88.02% agreement with human evaluation. We contend that the proposed paradigm explores the potential of MLLMs as effective evaluation tools with the help of per-sample criteria. See online leaderboard at \url{https://mLLM-bench.LLMzoo.com}.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (44)
  1. Falcon-40B: an open large language model with state-of-the-art performance. (2023).
  2. Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond. arXiv:2308.12966 [cs.CV]
  3. Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308.16890 (2023).
  4. Introducing our Multimodal Models. https://www.adept.ai/blog/fuyu-8b
  5. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use. arXiv preprint arXiv:2308.06595 (2023).
  6. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901.
  7. MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning. arXiv:2310.09478 [cs.CV]
  8. Shikra: Unleashing Multimodal LLM’s Referential Dialogue Magic. arXiv preprint arXiv:2306.15195 (2023).
  9. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 (2022).
  10. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. arXiv:2305.06500 [cs.CV]
  11. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023).
  12. Making LLaMA SEE and Draw with SEED Tokenizer. arXiv:2310.01218 [cs.CV]
  13. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556 (2022).
  14. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020).
  15. David R Krathwohl. 2002. A revision of Bloom’s taxonomy: An overview. Theory into practice 41, 4 (2002), 212–218.
  16. OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents. arXiv:2306.16527 [cs.IR]
  17. SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension. arXiv:2307.16125 [cs.CL]
  18. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023).
  19. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. arXiv:2301.12597 [cs.CV]
  20. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023).
  21. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models. arXiv preprint arXiv:2310.14566 (2023).
  22. Improved Baselines with Visual Instruction Tuning. arXiv:2310.03744 [cs.CV]
  23. Visual Instruction Tuning.
  24. MMBench: Is Your Multi-modal Model an All-around Player? arXiv preprint arXiv:2307.06281 (2023).
  25. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems 35 (2022), 27730–27744.
  26. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv:2306.14824 [cs.CL]
  27. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 3505–3506.
  28. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100 (2022).
  29. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761 (2023).
  30. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053 (2019).
  31. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 (2020), 3008–3021.
  32. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085 (2022).
  33. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023).
  34. To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuning. arXiv:2311.07574 [cs.CV]
  35. Large Language Models are not Fair Evaluators. arXiv:2305.17926 [cs.CL]
  36. CogVLM: Visual Expert for Pretrained Language Models. arXiv:2311.03079 [cs.CV]
  37. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022).
  38. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 (2022).
  39. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265 (2023).
  40. mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration. arXiv:2311.04257 [cs.CL]
  41. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490 (2023).
  42. GPT-4V (ision) as a Generalist Evaluator for Vision-Language Tasks. arXiv preprint arXiv:2311.01361 (2023).
  43. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023).
  44. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 [cs.CL]
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (17)
  1. Wentao Ge (2 papers)
  2. Shunian Chen (15 papers)
  3. Junying Chen (26 papers)
  4. Zhihong Chen (63 papers)
  5. Shuo Yan (13 papers)
  6. Chenghao Zhu (9 papers)
  7. Ziyue Lin (6 papers)
  8. Wenya Xie (8 papers)
  9. Xidong Wang (30 papers)
  10. Anningzhe Gao (22 papers)
  11. Jianquan Li (18 papers)
  12. Xiang Wan (93 papers)
  13. Benyou Wang (109 papers)
  14. Guiming Hardy Chen (8 papers)
  15. Nuo Chen (100 papers)
  16. Song Dingjie (1 paper)
  17. Zhang Zhiyi (2 papers)
Citations (3)
View on Semantic Scholar
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

  1. GitHub - FreedomIntelligence/MLLM-Bench: Evaluating Multi-modal LLMs using GPT-4V (70 stars)