GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages

Abstract: The need for large text corpora has increased with the advent of pretrained LLMs and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languages with large dominant communities. However, there is no corpus available that (i) covers a wide range of minority languages; (ii) is generated by an open-source reproducible pipeline; and (iii) is rigorously cleaned from noise, making it trustworthy to use. We present GlotCC, a clean, document-level, 2TB general domain corpus derived from CommonCrawl, covering more than 1000 languages. We make GlotCC and the system used to generate it - including the pipeline, language identification model, and filters - available to the research community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1, Pipeline v. 3.0 https://github.com/cisnlp/GlotCC.

• The paper presents a novel corpus, GlotCC, that significantly enhances the availability and quality of minority language data extracted from CommonCrawl.
• It introduces GlotLID v3.0, a robust language identification model that supports over 2000 language-script pairs and effectively manages web noise.
• Its comprehensive processing pipeline and rigorous self-audit yield high in-language accuracy, setting new standards for multilingual NLP resources.

A Comprehensive Analysis of GlotCC: Advancing Corpus Resources for Minority Languages

The paper "GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages" addresses the pressing need for a diverse and substantial language corpus tailored to minority languages. This work introduces GlotCC, a comprehensive and clean corpus derived from CommonCrawl, alongside its supporting system that includes the GlotLID language identification model and an open-source processing pipeline. Through various enhancements and extensive evaluations, this study extends the accessibility and reliability of corpora for over 1000 languages, significantly contributing to multilingual language technology research.

Core Contributions

1. Corpus Development: GlotCC emerges as a dataset covering an expansive range of languages, particularly minority ones, compiled through an improved language identification strategy and noise reduction processes. This initiative addresses the scarcity of multilingual resources in low-resource contexts, often neglected due to the dominance of a few high-resource languages.
2. Enhanced Language Identification: The introduction of GlotLID v3.0 represents a substantial advancement over existing language identification models like FastText and CLD3. By supporting more than 2000 language-script pairs and integrating specialized noise-handling mechanisms (e.g., noise detection labels "zxx" and "UND"), GlotLID v3.0 offers a higher accuracy and coverage. Such improvements are pivotal for minimizing misidentification issues and ensuring cleaner data extraction from web sources.
3. Pipeline and Filtering Innovations: The paper details an elaborate pipeline based on Ungoliant, and extensions tailored to address the limitations of previous tools. New quality control warnings and filters are integrated to ensure content consistency and remove residual noise, enriching the corpus with high-quality linguistic data free from prevalent web noise artifacts like Mojibake and mis-rendered PDFs.
4. Self-Audit and Evaluation: Evaluating GlotCC, the authors audit the dataset by analyzing random samples from different language subcorpora. This audit confirms a high in-language content accuracy with macro-average and median scores showcasing minimal misclassification, thus attesting to the robustness of the GlotLID model and filtering processes. The comparison with other LID models displays significant improvements in identifying minority languages.

Quantitative Results

The GlotLID v3.0 model has consistently shown robust performance across various benchmarks. It achieved an F1 score of 0.991 with a false positive rate of 0.000003 on the GlotTest evaluation set. A notable increase in coverage for minority languages was achieved, as evidenced by GlotCC's compilation statistics, which include more than 1275 LID labels. GlotCC surpasses traditional models while retaining accuracy, thus enhancing its reliability for training multilingual LLMs.

Implications and Further Developments

The theoretical implications of GlotCC lie in its ability to unify language processing methodologies for low-resource languages by providing clean and labeled corpora. Practically, GlotCC broadens language inclusion in NLP tasks, facilitates the development of more sophisticated LLMs, and helps democratize AI applications across diverse linguistic landscapes.

The paper suggests several paths for future developments. It aims to extend the corpus to incorporate additional CommonCrawl snapshots, thereby continuously updating the language database and further enhancing coverage. Additionally, collaborations with linguistic communities could help fine-tune language identification and data filtering processes, ensuring cultural accuracy and authenticity.

In conclusion, GlotCC and its underpinning systems mark a significant stride forward in corpus linguistics and language technology for minority languages. The research effectively bridges a critical resource gap, enabling broader LLM training while upholding ethical standards and focusing on inclusivity in the digital era.