GeoLayoutLM: Geometric Pre-training for Visual Information Extraction (2304.10759v1)

Abstract: Visual information extraction (VIE) plays an important role in Document Intelligence. Generally, it is divided into two tasks: semantic entity recognition (SER) and relation extraction (RE). Recently, pre-trained models for documents have achieved substantial progress in VIE, particularly in SER. However, most of the existing models learn the geometric representation in an implicit way, which has been found insufficient for the RE task since geometric information is especially crucial for RE. Moreover, we reveal another factor that limits the performance of RE lies in the objective gap between the pre-training phase and the fine-tuning phase for RE. To tackle these issues, we propose in this paper a multi-modal framework, named GeoLayoutLM, for VIE. GeoLayoutLM explicitly models the geometric relations in pre-training, which we call geometric pre-training. Geometric pre-training is achieved by three specially designed geometry-related pre-training tasks. Additionally, novel relation heads, which are pre-trained by the geometric pre-training tasks and fine-tuned for RE, are elaborately designed to enrich and enhance the feature representation. According to extensive experiments on standard VIE benchmarks, GeoLayoutLM achieves highly competitive scores in the SER task and significantly outperforms the previous state-of-the-arts for RE (\eg, the F1 score of RE on FUNSD is boosted from 80.35\% to 89.45\%). The code and models are publicly available at https://github.com/AlibabaResearch/AdvancedLiterateMachinery/tree/main/DocumentUnderstanding/GeoLayoutLM

GeoLayoutLM: Geometric Pre-training for Visual Information Extraction

In the field of Document Intelligence, visual information extraction (VIE) serves a pivotal function, primarily encompassing semantic entity recognition (SER) and relation extraction (RE). The paper "GeoLayoutLM: Geometric Pre-training for Visual Information Extraction" proposes a novel multi-modal framework that addresses the limitations of existing pre-trained models, particularly in relation extraction tasks.

Key Contributions and Methodology

GeoLayoutLM introduces an explicit geometric pre-training methodology to improve document layout representation. This is achieved via three innovative geometry-related pre-training tasks designed to capture geometric relations effectively. The model's architecture consists of independent visual and text-layout modules, refined through interactive co-attention layers, synthesizing the document's visual and textual information.

1. Geometric Relations and Pre-training Tasks:
   • GeoPair: This captures the relation between two text segments, refined through Direction and Distance Modeling (DDM) tasks that enhance directional and proximity understanding.
   • GeoMPair: Extends GeoPair by considering multiple text segment pairs, employing the Detection of Direction Exceptions (DDE) task to recognize common geometric patterns within document areas.
   • GeoTriplet: Examines relations among three text segments via Collinearity Identification of Triplets (CIT), crucial for modeling multi-segment relations.
2. Relation Heads:
   • A Coarse Relation Prediction (CRP) head and an advanced Relation Feature Enhancement (RFE) head are introduced. These are pre-trained to mitigate the gap between pre-training and fine-tuning, crucial for enhancing relation extraction capabilities.
3. Fine-tuning Strategy:
   • The model incorporates a Restriction on the Selection of Fathers (RSF) strategy during RE inference to refine relation predictions, further improving accuracy.

Experimental Results

The experimental evaluations on widely recognized benchmarks such as FUNSD and CORD underscore GeoLayoutLM's efficiency. For the FUNSD dataset, GeoLayoutLM excels with an F1 score of 89.45% in the RE task, marking a significant improvement over previous benchmarks. The SER task results were also optimal, showcasing the model's comprehensive capability in different VIE tasks.

Implications and Future Directions

The implications of this research are twofold. Practically, GeoLayoutLM offers a robust tool for document-based information extraction, showing potential in enhancing applications that require accurate document analysis such as automated invoice processing or form recognition. Theoretically, it paves the way for future research on geometric pre-training techniques in visual document understanding tasks.

For future advancements, exploration could delve into more effective geometric pre-training tasks or the adaptation of GeoLayoutLM's framework to other domains within visually-rich document understanding. Additionally, adjusting the current architecture to leverage different vision modules could further enhance its adaptability and efficiency.

In conclusion, the paper presents a comprehensive approach to overcoming the challenges of relation extraction within document intelligence, providing a meaningful stride towards the efficient processing of visually enriched documents.