Wikimedia data for AI: a review of Wikimedia datasets for NLP tasks and AI-assisted editing (2410.08918v1)

Abstract: Wikimedia content is used extensively by the AI community and within the LLMing community in particular. In this paper, we provide a review of the different ways in which Wikimedia data is curated to use in NLP tasks across pre-training, post-training, and model evaluations. We point to opportunities for greater use of Wikimedia content but also identify ways in which the LLMing community could better center the needs of Wikimedia editors. In particular, we call for incorporating additional sources of Wikimedia data, a greater focus on benchmarks for LLMs that encode Wikimedia principles, and greater multilingualism in Wikimedia-derived datasets.

• The paper reviews how Wikimedia datasets, especially from Wikipedia, form a robust foundation for training large language models in NLP and AI-assisted editing.
• It emphasizes the need to diversify data use by incorporating multilingual and multimodal elements to better serve both AI applications and Wikimedia editors.
• The study advocates for developing refined benchmarks aligned with Wikipedia's content policies to more effectively evaluate AI model outputs.

Analyzing the Role of Wikimedia Data in NLP and AI-Assisted Editing

The paper "Wikimedia data for AI: a review of Wikimedia datasets for NLP tasks and AI-assisted editing" by Isaac Johnson, Lucie-Aimée Kaffee, and Miriam Redi provides a thorough examination of how Wikimedia data, particularly from Wikipedia, has been utilized for NLP and related AI applications. The authors focus on the alignment between AI advancements and the needs of the Wikimedia community, emphasizing opportunities for future collaboration and improvement.

Overview of Wikimedia's Contributions to AI

Wikimedia projects, notably Wikipedia, have been foundational in offering high-quality, well-structured data essential for NLP tasks. The paper highlights the extensive use of English Wikipedia in pre-training LLMs, such as BERT, setting benchmarks for data quality and multilingual availability. This data has primarily served the AI community, but there has been less emphasis on mutual benefits for Wikimedia contributors.

Current Challenges and Opportunities

The authors identify several opportunities for better integrating Wikimedia contributions within AI research:

1. Diversification of Data Use: They argue for expanding the types of Wikimedia data utilized in AI research, such as Wikimedia Commons images and talk pages, to enhance multimodal models and community interaction analysis.
2. Enhanced Benchmark Representation: While Wikipedia data is prevalent in LLM benchmarks, these are primarily tailored to end-user applications rather than the specific needs of Wikimedia editors. Developing evaluation methods that help verify the adherence of AI-generated content to Wikimedia's content policies remains a significant opportunity.
3. Multilingual Model Expansion: As Wikipedia operates in over 300 languages, creating multilingual models that support non-English content would greatly benefit the Wikimedia community and align with Wikimedia's global reach.

Data Processing and Training Stages

The paper categorizes the NLP use of Wikimedia data into three stages:

• Pre-training: Raw Wikimedia data undergoes minimal preprocessing to form datasets that enable understanding of language patterns. Despite available English datasets, there is room for more preprocessed, standardized datasets across different languages.
• Post-training: At this stage, pre-processed datasets are transformed into specific tasks such as classification, recommendation, and text generation suited to the goals of Wikimedia projects. However, many tasks remain heavily English-centric.
• Evaluation: Effective benchmarks are vital for assessing AI models' adherence to the principles valued by Wikimedia editors. Current benchmarks need refinement to fully represent the nuanced content policies of Wikipedia and ensure meaningful assessments of LLM capability.

Implications for Future Research

The integration of Wikimedia data into AI research opens avenues for more sophisticated LLMs that respect and reflect Wikimedia's values. Future research should aim to:

• Develop more comprehensive multilingual and multimodal datasets.
• Create benchmarks aligned with editorial needs and Wikimedia's content policies.
• Encourage open-source and accessible models that the Wikimedia Foundation and its contributors can readily use and improve upon.

Conclusion

This paper underscores the mutually beneficial relationship that could flourish between the NLP community and Wikimedia projects. By addressing current gaps in data usage, multilingual capacity, and evaluation metrics, AI researchers can not only drive technical advancements but also enrich Wikimedia communities, supporting their mission of open, reliable knowledge dissemination.