Papers
Topics
Authors
Recent
Search
2000 character limit reached

DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models

Published 22 Feb 2024 in cs.CV | (2402.14767v1)

Abstract: We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal LLMs (MLLMs) to enhance vision-language task performance. Current MLLMs typically singularly focus on inputs at a predefined resolution, resulting in deficiencies in detailed questions involving local regions. We introduced a DualFocus mechanism where the model concentrates on the image from a macro perspective, responses to the question, and identifies suitable sub-regions to zoom in for subsequent micro perspective analysis. Via the integration of answers from both macro and micro perspectives, the model is adept at addressing tasks that encompass global, detailed, and combined considerations. To endows the DualFocus mechanism in MLLMs, we curated a tailored dataset derived from the Visual Genome (VG) and adapted it to align with the training regimen of DualFocus. Through comparative studies across different model sizes and benchmarks, we demonstrate DualFocus's superiority in balancing detailed examination with holistic insight, significantly reducing hallucination instances in MLLMs and improving their performance in various vision-language tasks.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv.org, 2023.
  2. Baichuan. Baichuan 2: Open large-scale language models. arXiv.org, 2023. URL https://arxiv.org/abs/2309.10305.
  3. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877–1901, 2020.
  4. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv.org, 2023a.
  5. Sharegpt4v: Improving large multi-modal models with better captions. arXiv.org, 2023b.
  6. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  7. Palm: Scaling language modeling with pathways. arXiv.org, 2022.
  8. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023.
  9. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv.org, 2018.
  10. Internlm-xcomposer2: Mastering free-form text-image composition and comprehension in vision-language large model. arXiv.org, 2024.
  11. Planting a seed of vision in large language model.
  12. Wanjuan: A comprehensive multimodal dataset for advancing english and chinese large models. arXiv.org, abs/2308.10755, 2023. URL https://api.semanticscholar.org/CorpusID:261049100.
  13. Cogagent: A visual language model for gui agents. arXiv.org, 2023.
  14. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9.
  15. Bliva: A simple multimodal llm for better handling of text-rich visual questions. ArXiv, abs/2308.09936, 2023. URL https://api.semanticscholar.org/CorpusID:261049015.
  16. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. arXiv.org, 2023.
  17. Gqa: A new dataset for real-world visual reasoning and compositional question answering. Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  18. Jelinek, F. Statistical methods for speech recognition. MIT press, 1998.
  19. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
  20. Otterhd: A high-resolution multi-modality model. Arxiv, 2023a.
  21. Otter: A multi-modal model with in-context instruction tuning. arXiv.org, 2023b.
  22. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine learning (ICML), pp.  12888–12900. PMLR, 2022.
  23. Evaluating object hallucination in large vision-language models. arXiv.org, 2023c.
  24. Monkey: Image resolution and text label are important things for large multi-modal models. Arxiv, 2023d.
  25. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744, 2023a.
  26. Visual instruction tuning. arXiv.org, 2023b.
  27. Mmbench: Is your multi-modal model an all-around player? arXiv:2307.06281, 2023c.
  28. OpenAI. Chatgpt. https://openai.com/blog/chatgpt, 2022.
  29. OpenAI. Gpt-4 technical report, 2023.
  30. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35:27730–27744, 2022.
  31. Kosmos-2: Grounding multimodal large language models to the world. arXiv.org, 2023.
  32. Gpt4point: A unified framework for point-language understanding and generation, 2023a.
  33. Gemini vs gpt-4v: A preliminary comparison and combination of vision-language models through qualitative cases, 2023b.
  34. Qwen. Introducing qwen-7b: Open foundation and human-aligned models (of the state-of-the-arts), 2023.
  35. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine learning (ICML), pp.  8748–8763. PMLR, 2021.
  36. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(1):5485–5551, 2020.
  37. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.  8317–8326, 2019.
  38. Alpha-CLIP: A clip model focusing on wherever you want. arXiv.org, 2023.
  39. Team, I. Internlm: A multilingual language model with progressively enhanced capabilities. https://github.com/InternLM/InternLM, 2023.
  40. Llama: Open and efficient foundation language models. arXiv.org, 2023a.
  41. Llama 2: Open foundation and fine-tuned chat models, 2023b.
  42. V3det: Vast vocabulary visual detection dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2023.
  43. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems (NIPS), 35:24824–24837, 2022.
  44. mplug-owl: Modularization empowers large language models with multimodality. arXiv.org, 2023.
  45. Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition. arXiv.org, 2023.
  46. Mllm-dataengine: An iterative refinement approach for mllm. arXiv.org, 2023.
  47. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv.org, 2023.
Citations (7)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up