- The paper presents a comprehensive 5M-word Kashmiri pretraining dataset using a custom InPage-to-Unicode conversion to recover archival texts.
- The authors implement an 11-stage cleaning pipeline achieving high script purity (0.9965) and perform detailed tokenization analysis.
- The study highlights significant implications for improving LLM efficiency and offers a replicable blueprint for low-resource language processing.
KS-PRET-5M: Construction and Implications of a 5 Million Word Kashmiri Pretraining Dataset
Introduction
KS-PRET-5M establishes a new resource benchmark for Kashmiri language processing, compiling 5.09M words (12.13M subword tokens) of high-fidelity text for NLP research. This dataset directly addresses Kashmiri's chronic underrepresentation in public multilingual corpora, an impediment that has limited both monolingual modeling and cross-lingual transfer for Indo-Aryan languages. Notably, the work resolves structural obstacles in Kashmiri text digitization: decades of literature and journalism rendered in InPage's proprietary encoding, previously inaccessible to Unicode-based NLP workflows, are now systematically recovered. The strategic integration of both archival InPage-recovered texts and Unicode-native web material results in a corpus with comprehensive coverage across genres and registers.
Dataset Construction
Source Material and Recovery Strategy
Two encoding classes underpin KS-PRET-5M. The first comprises professionally typeset, editorially validated texts archived in InPage format. Prior approaches failed to extract orthographically accurate Kashmiri due to proprietary diacritic handling and semantic distortions in generic Urdu/Persian mappings. The authors deploy a custom InPage-to-Unicode converter, implementing reverse-engineered parsing and precise combining-character logic. This unlocks content spanning narrative fiction, poetry, journalism, biography, academic writing, and religious scholarship, all preserving diacritic completeness.
The second component supplements this material with Unicode-native data mined from web portals, blogs, and institutional sites. While expanding temporal and stylistic range, web-sourced material introduces greater noise and code-mixing, necessitating robust post-processing.
Cleaning Pipeline
The empirical fidelity of KS-PRET-5M is maintained through an eleven-stage Python-based pipeline. This process encompasses encoding repair (ftfy), markup and noise symbol removal, Latin-script and digit stripping, and strict script-purity filtering based on Unicode block analysis. The script purity gate (mean Kashmiri-script ratio 0.9965) eliminates non-Kashmiri content to a negligible residue, e.g., only 146 Devanagari characters remain among nearly 28 million total, ensuring the dataset's syntactic and orthographic reliability.
Tokenization Analysis
The paper conducts an explicit, empirical measurement of Kashmiri subword tokenization using google/muril-base-cased. Contrary to prior heuristics based on non-Kashmiri Perso-Arabic analogues (1.3–1.8 tokens per word), the measured ratio is 2.383, yielding 12.13M subword tokens. This discrepancy is directly attributable to the diacritic (harakaat) density in formal Kashmiri: generic multilingual BPE tokenizers segment these abundantly, whereas a Kashmiri-native BPE would merge them, reducing fragmentation. This establishes the first reproducible baseline for Kashmiri tokenization rates relevant to dataset scaling and compute estimation.
Results
KS-PRET-5M sets a new standard in dataset size and quality for Kashmiri, characterized by:
- 5.09M words and 27.7M characters
- 295K unique word types (TTR 0.0580)
- High hapax legomena rate (47.4%), consistent with the language's morphological complexity
- 12.13M subword tokens under muril-base-cased
- Script purity (mean KS ratio 0.9965), with negligible Devanagari contamination
- Comprehensive genre coverage—including rare technical and classical registers absent from web scrapes
Discussion
InPage Recovery and NLP Resource Dynamics
The principal insight is that Kashmiri's data bottleneck is not a lack of professionally authored material but inaccessibility resulting from the InPage encoding barrier. The methodology applied demonstrates that reverse engineered tooling makes these repositories computationally tractable. This suggests that further expansion of high-quality Kashmiri datasets depends on scaling InPage recovery operations, providing a replicable pathway for analogous low-resource languages with proprietary text archives.
Tokenization and Model Efficiency
The subword tokenization analysis is of particular significance. Any cross-lingual or compute-resource comparison based on naive token multipliers systematically understates the effective scale of formally diacritized Kashmiri corpora. This inflates the perceived resource gap and inhibits realistic allocation for pretraining and benchmarking. Recommending tokenizer-specific empirical baselining for non-mainstream languages corrects this recurring methodological error. Additionally, training a Kashmiri-native BPE tokenizer on this dataset constitutes an immediately actionable step toward improving model efficiency and reducing redundant subword proliferation.
Morphological Representation
The dataset's hapax rate reflects robust representation of Kashmiri's morphological richness, a prerequisite for LLMs to generalize inflectional and derivational patterns. Its coverage is sufficient for initial pretraining but remains constrained compared to better-resourced Indian languages.
Limitations
Major limitations persist. KS-PRET-5M only supports pretraining; the absence of labeled fine-tuning resources (e.g., QA, NER, NLI) precludes systematic evaluation of task-specific model adaptation. The dataset's 5M-word scale, while unprecedented for Kashmiri, remains modest against the corpora available for Hindi/Bengali. There is also limited representation of regional/dialectal variants and absence of fine-grained annotation such as genre, provenance, or speaker metadata.
Implications and Future Directions
Practically, KS-PRET-5M enables the training of LLMs and tokenizers specifically adapted to Kashmiri's formal variety, acting as a seed resource for further pretraining, tokenizer construction, or intrinsic linguistic analysis. Theoretically, the resource and its construction pipeline can be replicated for other low-resource languages where editorially controlled, non-Unicode archives exist. Future research should focus on:
- Expansion through continued InPage recovery and diversifying with domain-/dialect-marked data
- Building supervised datasets for downstream task evaluation and adaptation
- Training Kashmiri-specific BPE tokenizers and assessing their impact on compression and downstream accuracy
Conclusion
KS-PRET-5M establishes Kashmiri as a computationally accessible language for large-scale NLP research. The work addresses nearly all prior barriers to systematic pretraining by combining a technically rigorous recovery strategy with empirical validation of script and tokenization properties. The dataset becomes the de facto foundation for Kashmiri LLM research and provides a reproducible methodology for analogous efforts in other under-resourced language communities. Further advances now depend on task-oriented resource construction, tokenizer adaptation, and the sustained unlocking of archival text in non-standard encodings.
Reference:
"ks-pret-5m: a 5 million word, 12 million token kashmiri pretraining dataset" (2604.11066)