Aligning benchmark datasets for table structure recognition (2303.00716v2)

Abstract: Benchmark datasets for table structure recognition (TSR) must be carefully processed to ensure they are annotated consistently. However, even if a dataset's annotations are self-consistent, there may be significant inconsistency across datasets, which can harm the performance of models trained and evaluated on them. In this work, we show that aligning these benchmarks$\unicode{x2014}$removing both errors and inconsistency between them$\unicode{x2014}$improves model performance significantly. We demonstrate this through a data-centric approach where we adopt one model architecture, the Table Transformer (TATR), that we hold fixed throughout. Baseline exact match accuracy for TATR evaluated on the ICDAR-2013 benchmark is 65% when trained on PubTables-1M, 42% when trained on FinTabNet, and 69% combined. After reducing annotation mistakes and inter-dataset inconsistency, performance of TATR evaluated on ICDAR-2013 increases substantially to 75% when trained on PubTables-1M, 65% when trained on FinTabNet, and 81% combined. We show through ablations over the modification steps that canonicalization of the table annotations has a significantly positive effect on performance, while other choices balance necessary trade-offs that arise when deciding a benchmark dataset's final composition. Overall we believe our work has significant implications for benchmark design for TSR and potentially other tasks as well. Dataset processing and training code will be released at https://github.com/microsoft/table-transformer.

• The paper demonstrates that aligning datasets increases TATR's accuracy from 42%-69% to 65%-81% by reducing annotation errors.
• It employs canonicalization of table annotations to ensure consistency within and across benchmarks for robust performance.
• The study highlights that improving dataset quality, rather than altering model architectures, leads to significant gains in model reliability.

Aligning Benchmark Datasets for Table Structure Recognition

The paper "Aligning Benchmark Datasets for Table Structure Recognition" explores the crucial task of enhancing the consistency and accuracy of benchmark datasets used in Table Structure Recognition (TSR). The authors underscore the adverse effects of both annotation errors and inconsistencies across datasets, highlighting the impact on model performance. By focusing on harmonizing these datasets, the paper aims to significantly boost the efficacy of TSR models, particularly the Table Transformer (TATR).

Key Contributions

The paper primarily addresses inconsistencies within and across benchmark datasets, emphasizing the repercussions such inconsistencies can have on TSR models. The authors adopt a fixed model architecture, TATR, and undertake a rigorous examination of prominent TSR benchmarks such as PubTables-1M, FinTabNet, and ICDAR-2013.

1. Error Reduction and Consistency Alignment:
   • The research demonstrates a substantial improvement in TATR's performance upon rectifying annotation mistakes. Initial exact match accuracies on the ICDAR-2013 benchmark range from 42% (FinTabNet) to 69% (combined datasets). Post-alignment, these figures rise to 65% and 81%, respectively.
2. Canonicalization Impacts:
   • Canonicalization of table annotations emerged as a critical factor, ensuring that datasets are not only internally consistent but also aligned with each other. This step is instrumental in achieving the reported performance gains.
3. Data-Centric Performance Enhancements:
   • By maintaining a fixed model architecture, the paper highlights the potential of data-centric strategies. It underscores that rectifying the data itself, rather than modifying models, can lead to considerable performance improvements.
4. Implications for Benchmark Design:
   • The work suggests significant implications for the design and use of benchmark datasets in TSR and possibly other machine learning tasks. It stresses the importance of consistent and error-free data for robust model evaluation and training.

Implications and Future Directions

The findings hold profound implications for practitioners and researchers in the field of document intelligence. Enhanced consistency in benchmark datasets reduces noise, thus allowing models to more accurately learn from data and perform in real-world scenarios. This suggests that ongoing efforts in TSR should prioritize dataset quality, potentially revisiting older datasets to rectify inconsistencies instead of merely querying new data.

The insight provided by this paper lays a foundation for further exploration into the effects of dataset quality across various AI tasks. Additionally, it opens avenues for developing automated tools and methodologies for dataset alignment and error detection.

In conclusion, this paper effectively demonstrates the pivotal role of dataset integrity in the performance of machine learning models. By addressing alignment and consistency in TSR benchmark datasets, the paper provides a model for enhancing dataset quality across AI domains, paving the way for more accurate, reliable machine learning models in practice.