- The paper presents the SIPAKOSA corpus with 786,839 sentences and 9.25M tokens, using dual-source methods (OCR and web scraping) for data collection.
- It demonstrates rigorous language classification, metadata curation, and segmentation to support both monolingual and mixed-language Buddhist text analysis.
- The corpus sets benchmarks for low-resource NLP in Sinhala and Pali, enhancing model evaluation, domain adaptation, and cultural preservation.
SIPAKOSA: A Critical Resource for Sinhala and Pali Buddhist NLP
Introduction
"SiPaKosa: A Comprehensive Corpus of Canonical and Classical Buddhist Texts in Sinhala and Pali" (2603.29221) presents the construction and analysis of SIPAKOSA, the most expansive digital corpus dedicated to canonical and classical Buddhist texts in Sinhala and Pali. Leveraging dual-source collection—systematic web scraping of canonical Tripitaka repositories and high-fidelity OCR on copyright-cleared historical manuscripts—the resource comprises 786,839 sentences amounting to 9.25 million tokens, split into Sinhala and Mixed Sinhala-Pali subcorpora. This work addresses a critical bottleneck for NLP research in Sri Lanka, offering domain-specific data for low-resource languages and facilitating advanced study of historical, religious, and code-mixed/scripture-based text.
Construction Methodology
The dual-source pipeline integrates 16 historical Buddhist documents and the digitally available Tripitaka canonical texts:
- Historical Text Processing: 83 PDF texts from IFBCnet were subjected to strict copyright filtering (70-year post-mortem auctoris rule), reducing the set to 16 public domain works. PDF text extraction was attempted directly where possible; for scanned content, Google Document AI was adopted, yielding an average character-level OCR confidence of 99.8%. Non-content material (e.g., covers, indices) and low-confidence pages were systematically filtered or quarantined.
- Canonical Source Collection: The entire Tripitaka corpus was harvested via a custom web-scraper targeting tripitaka.online, parsing JavaScript-embedded content with structural metadata preservation, ultimately extracting 539,326 well-structured sentences.
- Language Classification and Metadata: A dual-lexicon, weighted confidence approach (70% lexicon coverage, 30% morphological cues) classified segments as Sinhala, Pali, or Mixed. Notably, no pure Pali pages met the empirical threshold, reflecting the domain-specific distribution of script and vocabulary. Rich metadata encapsulated bibliographic details, provenance, extraction metrics, and copyright status.
Corpus Analysis
The finalized SIPAKOSA corpus includes:
- Scale: 9.25M tokens over 786,839 sentences.
- Language Distribution: 465,539 sentences (59.2%; 5.42M tokens) Sinhala; 320,805 sentences (40.8%; 3.83M tokens) Mixed Sinhala-Pali.
- Structural Integrity: Separation into subcorpora enables targeted monolingual and mixed-language modeling, with fixed 80/10/10 train/val/test splits for replicable benchmarks.
- Coverage: The corpus spans three major textual classes—canonical scripture, commentary/philosophy, and hagiographical works—covering monastic, doctrinal, and narrative genres essential for comprehensive Buddhist studies in the region.
Comparative Context
Cross-corpus comparison positions SIPAKOSA within the landscape of world religious and Indic language corpora:
- Innovation Relative to Prior Sinhala Corpora: SiDiaC [Jayatilleke and de Silva, 2025a, 2026] provides diachronic secular and some religious Sinhala, but at lower scale (241K words) and without direct handling of code-mixed Pali-Sinhala or canonical structure.
- Domain-Specificity over Existing Indic/Religious Datasets: Unlike GRETIL or DCS, SIPAKOSA encodes Buddhist text in Sinhala and Pali (Sinhala script), a configuration previously absent. Unlike the Digital Pali Canon or CBETA, it leverages NLP-compatible sentence segmentation, language-class annotation, and metadata critical for downstream processing.
Model Evaluation and Baseline Results
Corpus-specific language modeling benchmarks were established using nine models: four proprietary GPT-family variants and five open-source LLMs (Llama-3, Aya-Expanse, Qwen2.5, Gemma, SinLlama).
Key findings:
- Performance Gap: Proprietary GPT models substantially outperform open models by factors of 3–6 in perplexity (e.g., GPT-3.5-Turbo achieves PPL 1.09 on Buddhist Sinhala; best open LLaMA model, 3.29).
- Domain Sensitivity: Proprietary models demonstrate robust generalization (Buddhist Sinhala / General Sinhala PPL ratios ≈ 1.01), confirming effective register adaptation. Open-source models manifest significant degradation, particularly on Mixed Sinhala-Pali content (e.g., M/G ratio for Aya-Expanse-8B: 1.80).
- Language Mixing Challenge: All models exhibit increased perplexity—sometimes >80%—on Mixed Sinhala-Pali, highlighting the inherent complexity of intra-sentential and inter-sentential language mixing. This effect is most pronounced in open-source models lacking explicit code-mixing handling.
- Model Size vs. Performance: Larger parameter counts do not guarantee improved performance (e.g., Llama-3.2-3B outperforms the larger Gemma-2-9B), underscoring the importance of domain-specific pretraining and transfer adaptation over scale alone.
Practical and Theoretical Implications
- Benchmark for Sinhala/Pali Buddhist NLP: SIPAKOSA sets a replicable benchmark for both monolingual and code-mixed Buddhist language modeling, facilitating the development of IR, QA, and translation systems for Buddhist scholastic content.
- Advancement in Low-resource NLP: The pronounced gap between proprietary and open models elucidates the data deficiency in open Sinhala and Pali resources. SIPAKOSA enables fine-tuning, continued pretraining, and parameter-efficient adaptation (e.g., LoRA, QLoRA), particularly for lightweight and locally-deployable models.
- Diachronic and Register Studies: Rich metadata and genre/language stratification allow corpus-driven studies of linguistic change, code-mixing phenomena, and doctrinal concept integration into vernacular Sinhala, advancing both computational and descriptive linguistics.
- Cultural Preservation through NLP: By securing, curating, and annotating historical Buddhist materials, SIPAKOSA contributes critically to digital preservation and accessibility of Sri Lankan heritage for computational analysis, education, and global Buddhist scholarship.
Limitations and Potential Future Directions
- Temporal Scope: Due to copyright constraints, post-1950s scholarly and modern doctrinal works are underrepresented; contemporary Buddhist discourse remains largely outside the present corpus.
- Post-processing and Error Correction: Although OCR accuracy is high, absence of expert manual normalization may leave residual orthographic ambiguity, especially in historical Sinhala and Pali; future involvement of philological experts could further enhance corpus quality.
- Language Identification Granularity: Code-mixing detection remains lexicon-based, not deep context-aware; future approaches could integrate neural sequence labeling or unsupervised clustering for improved intra-sentence language attribution.
- Evaluation Breadth: Only classical perplexity benchmarks are reported; comprehensive evaluation on semantic and information-seeking tasks (e.g., QA, IR, domain-specific retrieval, and translation) remains for future work, including the use of human expert validation.
Conclusion
SIPAKOSA (2603.29221) delivers a rigorously constructed, large-scale, and richly annotated resource for Sinhala and Pali Buddhist NLP. Leveraging advanced digitization and meticulous quality filtering, it addresses the persistent lack of in-domain, high-recall training and benchmark data for low-resource South Asian religious text. Preliminary evaluations reveal both the challenge of code-mixed canonical literature and the urgent necessity for deeper domain adaptation in open-source LLMs. SIPAKOSA establishes a foundation upon which future efforts in language modeling, historical linguistics, and Buddhist studies can be robustly advanced, with direct implications for cultural preservation, digital humanities, and broader Indic low-resource NLP research.