Papers
Topics
Authors
Recent
2000 character limit reached

Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs (2311.13194v2)

Published 22 Nov 2023 in cs.CV

Abstract: In the field of document understanding, significant advances have been made in the fine-tuning of Multimodal LLMs (MLLMs) with instruction-following data. Nevertheless, the potential of text-grounding capability within text-rich scenarios remains underexplored. In this paper, we present a text-grounding document understanding model, termed TGDoc, which addresses this deficiency by enhancing MLLMs with the ability to discern the spatial positioning of text within images. Empirical evidence suggests that text-grounding improves the model's interpretation of textual content, thereby elevating its proficiency in comprehending text-rich images. Specifically, we compile a dataset containing 99K PowerPoint presentations sourced from the internet. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and LLM. Moreover, we curate a collection of text-rich images and prompt the text-only GPT-4 to generate 12K high-quality conversations, featuring textual locations within text-rich scenarios. By integrating text location data into the instructions, TGDoc is adept at discerning text locations during the visual question process. Extensive experiments demonstrate that our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. Flamingo: a visual language model for few-shot learning. Proceedings of the Advances in Neural Information Processing Systems, 35:23716–23736, 2022.
  2. OpenFlamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023.
  3. ICDAR 2019 competition on scene text visual question answering. In Proceedings of the International Conference on Document Analysis and Recognition, pages 1563–1570. IEEE, 2019.
  4. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023.
  5. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. URL https://vicuna. lmsys. org, 2023.
  6. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  7. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
  8. PaLM-E: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  9. EVA: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 19358–19369, 2023.
  10. UniDoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308.11592, 2023.
  11. ICDAR2019 competition on scanned receipt OCR and information extraction. In Proceedings of the International Conference on Document Analysis and Recognition, pages 1516–1520. IEEE, 2019.
  12. FUNSD: A dataset for form understanding in noisy scanned documents. In Proceedings of the International Conference on Document Analysis and Recognition Workshops, pages 1–6. IEEE, 2019.
  13. Visual information extraction in the wild: practical dataset and end-to-end solution. In Proceedings of the International Conference on Document Analysis and Recognition, pages 36–53. Springer, 2023.
  14. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning, pages 12888–12900. PMLR, 2022.
  15. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023.
  16. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023a.
  17. On the hidden mystery of OCR in large multimodal models. arXiv preprint arXiv:2305.07895, 2023b.
  18. SGDR: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983, 2016.
  19. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  20. KOSMOS-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419, 2023.
  21. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022.
  22. DocVQA: A dataset for VQA on document images. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pages 2200–2209, 2021.
  23. InfographicVQA. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022.
  24. OCR-VQA: Visual question answering by reading text in images. In Proceedings of the International Conference on Document Analysis and Recognition, pages 947–952. IEEE, 2019.
  25. OpenAI. Gpt-4 technical report, 2023.
  26. Training language models to follow instructions with human feedback. Proceedings of the Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  27. KOSMOS-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
  28. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  29. Ronald Rivest. The md5 message-digest algorithm. Technical report, 1992.
  30. LAION-400M: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021.
  31. Towards VQA models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019.
  32. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca, 2023.
  33. Plug-and-Play VQA: Zero-shot VQA by conjoining large pretrained models with zero training. arXiv preprint arXiv:2210.08773, 2022.
  34. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  35. Self-Instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  36. LayoutReader: Pre-training of text and layout for reading order detection. arXiv preprint arXiv:2108.11591, 2021.
  37. Chain-of-thought prompting elicits reasoning in large language models. Proceedings of the Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  38. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. arXiv preprint arXiv:2304.01196, 2023.
  39. LayoutLM: Pre-training of text and layout for document image understanding. In Proceedings of ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1192–1200, 2020a.
  40. LayoutLMv2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint arXiv:2012.14740, 2020b.
  41. The dawn of lmms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.17421, 2023.
  42. mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023a.
  43. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023b.
  44. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023c.
  45. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
  46. StrucTexTv2: Masked visual-textual prediction for document image pre-training. arXiv preprint arXiv:2303.00289, 2023.
  47. Monkey: Image resolution and text label are important things for large multi-modal models. arXiv preprint arXiv:2311.06607, 2023a.
  48. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023b.
  49. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
Citations (14)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up