Handwritten Text Recognition from Crowdsourced Annotations (2306.10878v1)
Abstract: In this paper, we explore different ways of training a model for handwritten text recognition when multiple imperfect or noisy transcriptions are available. We consider various training configurations, such as selecting a single transcription, retaining all transcriptions, or computing an aggregated transcription from all available annotations. In addition, we evaluate the impact of quality-based data selection, where samples with low agreement are removed from the training set. Our experiments are carried out on municipal registers of the city of Belfort (France) written between 1790 and 1946. % results The results show that computing a consensus transcription or training on multiple transcriptions are good alternatives. However, selecting training samples based on the degree of agreement between annotators introduces a bias in the training data and does not improve the results. Our dataset is publicly available on Zenodo: https://zenodo.org/record/8041668.
- Individual vs. Collaborative Methods of Crowdsourced Transcription. In Journal of Data Mining and Digital Humanities. https://doi.org/10.46298/jdmdh.5759
- Multiple Document Datasets Pre-training Improves Text Line Detection With Deep Neural Networks. In 25th International Conference on Pattern Recognition. 2134–2141. https://doi.org/10.1109/ICPR48806.2021.9412447
- DAN: a Segmentation-free Document Attention Network for Handwritten Document Recognition. In IEEE Transactions on Pattern Analysis and Machine Intelligence. https://doi.org/10.1109/TPAMI.2023.3235826
- Quality Control in Crowdsourcing. In ACM Computing Surveys. 1–40. https://doi.org/10.1145/3148148
- Automatic text summarization: A comprehensive survey. In Expert Systems with Applications. 113679. https://doi.org/10.1016/j.eswa.2020.113679
- J.G. Fiscus. 1997. A post-processing system to yield reduced word error rates: Recognizer Output Voting Error Reduction (ROVER). In IEEE Workshop on Automatic Speech Recognition and Understanding Proceedings. 347–354. https://doi.org/10.1109/ASRU.1997.659110
- Learning from Multiple Annotator Noisy Labels via Sample-wise Label Fusion. In Computer Vision – ECCV 2022. 407–422. https://doi.org/10.1007/978-3-031-20053-3_24
- The Interedition Development Group. 2010-2019. https://collatex.net/ Collatex software home page. Accessed: 2023-04-14.
- Jiyi Li. 2020. Crowdsourced Text Sequence Aggregation Based on Hybrid Reliability and Representation. In 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. 1761–1764. https://doi.org/10.1145/3397271.3401239
- Jiyi Li and Fumiyo Fukumoto. 2019. A Dataset of Crowdsourced Word Sequences: Collections and Answer Aggregation for Ground Truth Creation. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP. 24–28. https://doi.org/10.18653/v1/D19-5904
- Joan Puigcerver. 2017. Are Multidimensional Recurrent Layers Really Necessary for Handwritten Text Recognition?. In 14th International Conference on Document Analysis and Recognition. 67–72. https://doi.org/10.1109/ICDAR.2017.20
- Filipe Rodrigues and Francisco Pereira. 2018. Deep learning from crowds. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. 1611–1618. https://doi.org/10.5555/3504035.3504232
- Learning from Crowds with Crowd-Kit. https://arxiv.org/abs/2109.08584
- Deep Learning From Multiple Noisy Annotators as A Union. In IEEE transactions on neural networks and learning systems.
- Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ArXiv abs/2110.12088 (2021).
- To Aggregate or Not? Learning with Separate Noisy Labels. In CSW@WSDM.
- Detecting Corrupted Labels Without Training a Model to Predict. In International Conference on Machine Learning.