An Assessment of the Impact of OCR Noise on Language Models

Abstract: Neural LLMs are the backbone of modern-day natural language processing applications. Their use on textual heritage collections which have undergone Optical Character Recognition (OCR) is therefore also increasing. Nevertheless, our understanding of the impact OCR noise could have on LLMs is still limited. We perform an assessment of the impact OCR noise has on a variety of LLMs, using data in Dutch, English, French and German. We find that OCR noise poses a significant obstacle to language modelling, with LLMs increasingly diverging from their noiseless targets as OCR quality lowers. In the presence of small corpora, simpler models including PPMI and Word2Vec consistently outperform transformer-based models in this respect.