Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding (2311.11810v4)

Published 20 Nov 2023 in cs.CV and cs.AI

Abstract: This work presents DocPedia, a novel large multimodal model (LMM) for versatile OCR-free document understanding, capable of parsing images up to 2,560$\times$2,560 resolution. Unlike existing work either struggle with high-resolution documents or give up the LLM thus vision or language ability constrained, our DocPedia directly processes visual input in the frequency domain rather than the pixel space. The unique characteristic enables DocPedia to capture a greater amount of visual and textual information using a limited number of visual tokens. To consistently enhance both perception and comprehension abilities of our model, we develop a dual-stage training strategy and enrich instructions/annotations of all training tasks covering multiple document types. Extensive quantitative and qualitative experiments conducted on various publicly available benchmarks confirm the mutual benefits of jointly learning perception and comprehension tasks. The results provide further evidence of the effectiveness and superior performance of our DocPedia over other methods.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (56)
  1. Discrete cosine transform. IEEE Transactions on Computers, 100(1):90–93, 1974.
  2. DocFormer: End-to-end transformer for document understanding. In ICCV, pages 993–1003, 2021.
  3. Wukong-Reader: Multi-modal pre-training for fine-grained visual document understanding. arXiv preprint arXiv:2212.09621, 2022.
  4. Qwen-VL: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023.
  5. ICDAR 2019 competition on scene text visual question answering. In ICDAR, pages 1563–1570, 2019.
  6. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020.
  7. Vicuna: An open-source chatbot impressing GPT-4 with 90%* ChatGPT quality. See https://vicuna. lmsys. org, 2023.
  8. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  9. UniDoc: A universal large multimodal model for simultaneous text detection, recognition, spotting and understanding. arXiv preprint arXiv:2308.11592, 2023.
  10. Masked autoencoders are scalable vision learners. In CVPR, pages 16000–16009, 2022.
  11. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. In AAAI, pages 10767–10775, 2022.
  12. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys, 51(6):1–36, 2019.
  13. LayoutLMv3: Pre-training for document ai with unified text and image masking. In ACM MM, pages 4083–4091, 2022.
  14. ICDAR 2019 competition on scanned receipt OCR and information extraction. In ICDAR, pages 1516–1520, 2019.
  15. Post-OCR parsing: building simple and robust parser via bio tagging. In NeurIPS, 2019.
  16. Funsd: A dataset for form understanding in noisy scanned documents. In ICDARW, pages 1–6, 2019.
  17. FigureQA: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017.
  18. OCR-free document understanding transformer. In ECCV, pages 498–517, 2022.
  19. Visual information extraction in the wild: practical dataset and end-to-end solution. In ICDAR, pages 36–53, 2023.
  20. Pix2Struct: Screenshot parsing as pretraining for visual language understanding. In ICML, pages 18893–18912, 2023.
  21. StrucTexT: Structured text understanding with multi-modal transformers. In ACM MM, pages 1912–1920, 2021.
  22. Real-time scene text detection with differentiable binarization. In AAAI, pages 11474–11481, 2020.
  23. NomMer: Nominate synergistic context in vision transformer for visual recognition. In CVPR, pages 12073–12082, 2022.
  24. The devil is in the frequency: Geminated gestalt autoencoder for self-supervised visual pre-training. In AAAI, pages 1649–1656, 2023a.
  25. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023b.
  26. Fots: Fast oriented text spotting with a unified network. In CVPR, pages 5676–5685, 2018.
  27. On the hidden mystery of OCR in large multimodal models. arXiv preprint arXiv:2305.07895, 2023c.
  28. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 10012–10022, 2021.
  29. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
  30. GeoLayoutLM: Geometric pre-training for visual information extraction. In CVPR, pages 7092–7101, 2023.
  31. KOSMOS-2.5: A multimodal literate model. arXiv preprint arXiv:2309.11419, 2023.
  32. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244, 2022.
  33. DocVQA: A dataset for VQA on document images. In WACV, pages 2200–2209, 2021.
  34. InfographicVQA. In WACV, pages 1697–1706, 2022.
  35. OCR-VQA: Visual question answering by reading text in images. In ICDAR, pages 947–952, 2019.
  36. Ernie-layout: Layout knowledge enhanced pre-training for visually-rich document understanding. arXiv preprint arXiv:2210.06155, 2022.
  37. KOSMOS-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023.
  38. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763, 2021.
  39. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. TPAMI, 39(11):2298–2304, 2016.
  40. Towards VQA models that can read. In CVPR, pages 8317–8326, 2019.
  41. Super-convergence: Very fast training of neural networks using large learning rates. In AIMLMOA, pages 369–386, 2019.
  42. Document image understanding. In FJCC, pages 87–95, 1986.
  43. Unifying vision, text, and layout for universal document processing. In CVPR, pages 19254–19264, 2023.
  44. Hierarchical multimodal transformers for multipage DocVQA. PR, 144:109834, 2023.
  45. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  46. Attention is all you need. NeurIPS, 30, 2017.
  47. Gregory K Wallace. The jpeg still picture compression standard. Commun. ACM, 34(4):30–44, 1991.
  48. End-to-end scene text recognition. In ICCV, pages 1457–1464, 2011.
  49. LayoutLM: Pre-training of text and layout for document image understanding. In KDD, pages 1192–1200, 2020.
  50. LayoutxLM: Multimodal pre-training for multilingual visually-rich document understanding. arXiv preprint arXiv:2104.08836, 2021.
  51. mPLUG-DocOwl: Modularized multimodal large language model for document understanding. arXiv preprint arXiv:2307.02499, 2023a.
  52. UReader: Universal OCR-free visually-situated language understanding with multimodal large language model. arXiv preprint arXiv:2310.05126, 2023b.
  53. mPLUG-Owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023c.
  54. StrucTexTv2: Masked visual-textual prediction for document image pre-training. arXiv preprint arXiv:2303.00289, 2023.
  55. LLaVAR: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023.
  56. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (7)
  1. Hao Feng (83 papers)
  2. Qi Liu (485 papers)
  3. Hao Liu (497 papers)
  4. Wengang Zhou (153 papers)
  5. Houqiang Li (236 papers)
  6. Can Huang (43 papers)
  7. Jingqun Tang (22 papers)
Citations (39)
View on Semantic Scholar