Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-5
GPT-4o
DeepSeek R1 via Azure
2000 character limit reached

AutoBench-V: Can Large Vision-Language Models Benchmark Themselves? (2410.21259v4)

Published 28 Oct 2024 in cs.CV and cs.AI

Abstract: Large Vision-LLMs (LVLMs) have become essential for advancing the integration of visual and linguistic information. However, the evaluation of LVLMs presents significant challenges as the evaluation benchmark always demands lots of human cost for its construction, and remains static, lacking flexibility once constructed. Even though automatic evaluation has been explored in textual modality, the visual modality remains under-explored. As a result, in this work, we address a question: "Can LVLMs themselves be used to benchmark each other in the visual automatically domain?". We introduce AutoBench-V, an automated framework for serving evaluation on demand, i.e., benchmarking LVLMs based on specific aspects of model capability. AutoBench-V leverages text-to-image models to generate relevant image samples and then utilizes LVLMs to orchestrate visual question-answering (VQA) tasks, completing the evaluation process efficiently and flexibly. Through an extensive evaluation of nine popular LVLMs across five demanded user inputs (i.e., evaluation capabilities), the framework shows effectiveness and reliability.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Vqa: Visual question answering. arXiv preprint arXiv:1505.00468, 2016.
  3. Anthropic. Claude3family. Anthropic News, 2024. URL https://www.anthropic.com/news/claude-3-family.
  4. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023a.
  5. Benchmarking foundation models with language-model-as-an-examiner. arXiv preprint arXiv:2306.04181, 2023b.
  6. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023.
  7. blackforestlabs. flux_pro. BlackForestLabs News, 2024. URL https://blackforestlabs.ai/announcing-black-forest-labs/.
  8. Spatialbot: Precise spatial understanding with vision language models. arXiv preprint arXiv:2406.13642, 2024.
  9. Gui-world: A dataset for gui-oriented multimodal llm-based agents. arXiv preprint arXiv:2406.10819, 2024a.
  10. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024b.
  11. DeepMind. Gemini-1-5-flash. DeepMind Technologies, 2024. URL https://deepmind.google/technologies/gemini/flash/.
  12. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2024.
  13. Multimodal emotion recognition with deep learning: advancements, challenges, and future directions. Information Fusion, 105:102218, 2024.
  14. Deep learning approaches on image captioning: A review. ACM Computing Surveys, 56(3):1–39, 2023.
  15. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793, 2024.
  16. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. arXiv preprint arXiv:1612.00837, 2017.
  17. Regiongpt: Towards region understanding vision language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  13796–13806, 2024.
  18. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. arXiv preprint arXiv:2303.11897, 2023.
  19. Metatool benchmark for large language models: Deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128, 2023.
  20. Position: Trustllm: Trustworthiness in large language models. In International Conference on Machine Learning, pp.  20166–20270. PMLR, 2024.
  21. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. arXiv preprint arXiv:1412.2306, 2015.
  22. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  4401–4410, 2019.
  23. Aligning text-to-image models using human feedback. arXiv preprint arXiv:2302.12192, 2023.
  24. Seed-bench-2: Benchmarking multimodal large language models. arXiv preprint arXiv:2311.17092, 2023a.
  25. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023b.
  26. Autobencher: Creating salient, novel, difficult datasets for language models. arXiv preprint arXiv:2407.08351, 2024.
  27. Microsoft coco: Common objects in context. arXiv preprint arXiv:1405.0312, 2015.
  28. Visual instruction tuning. Advances in neural information processing systems, 36, 2024a.
  29. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743, 2023a.
  30. Alignbench: Benchmarking chinese alignment of large language models. arXiv preprint arXiv:2311.18743, 2024b.
  31. Sora: A review on background, technology, limitations, and opportunities of large vision models. arXiv preprint arXiv:2402.17177, 2024c.
  32. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023b.
  33. Deid-gpt: Zero-shot medical text de-identification by gpt-4. arXiv preprint arXiv:2303.11032, 2023c.
  34. Mmiu: Multimodal multi-image understanding for evaluating large vision-language models. arXiv preprint arXiv:2408.02718, 2024.
  35. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
  36. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789, 2023.
  37. M Ross Quillian. Semantic memory. Air Force Cambridge Research Laboratories, Office of Aerospace Research …, 1966.
  38. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  10684–10695, 2022.
  39. Imagenet large scale visual recognition challenge. arXiv preprint arXiv:1409.0575, 2015.
  40. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  41. Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
  42. Attention is all you need. arXiv preprint arXiv:1706.03762, 2023.
  43. Prompt2model: Generating deployable models from natural language instructions. arXiv preprint arXiv:2308.12261, 2023.
  44. Unigen: A unified framework for textual dataset generation using large language models. arXiv preprint arXiv:2406.18966, 2024.
  45. Online object tracking: A benchmark. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  2411–2418, 2013.
  46. Configurable foundation models: Building llms from a modular perspective. arXiv preprint arXiv:2409.02877, 2024a.
  47. C-pack: Packaged resources to advance general chinese embedding. arXiv preprint arXiv:2309.07597, 2024b.
  48. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023.
  49. A survey of scene understanding by event reasoning in autonomous driving. International Journal of Automation and Computing, 15(3):249–266, 2018.
  50. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024.
  51. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. arXiv preprint arXiv:2306.06687, 2023.
  52. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi. arXiv preprint arXiv:2404.16006, 2024.
  53. Idealgpt: Iteratively decomposing vision and language reasoning via large language models. arXiv preprint arXiv:2305.14985, 2023.
  54. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023.
  55. Mm-vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities. arXiv preprint arXiv:2408.00765, 2024.
  56. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.16502, 2024.
  57. Task me anything. arXiv preprint arXiv:2406.11775, 2024a.
  58. Debiasing multimodal large language models. arXiv preprint arXiv:2403.05262, 2024b.
  59. Self-guide: Better task-specific instruction following via self-synthetic finetuning. arXiv preprint arXiv:2407.12874, 2024.
  60. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  61. Navgpt: Explicit reasoning in vision-and-language navigation with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38(7), pp.  7641–7649, 2024.
  62. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
  63. Dyval: Dynamic evaluation of large language models for reasoning tasks. arXiv preprint arXiv:2309.17167, 2024a.
  64. Dynamic evaluation of large language models by meta probing agents. arXiv preprint arXiv:2402.14865, 2024b.
  65. Object detection in 20 years: A survey. arXiv preprint arXiv:1905.05055, 2023.
Citations (3)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube