Swa-Bhasha Resource Hub for South Asian NLP
- Swa-Bhasha Resource Hub is a comprehensive repository consolidating datasets, algorithms, and tools for low-resource South Asian language technology, with an initial focus on Sinhala transliteration.
- The hub employs transparent pipelines ranging from rule-based to neural models, achieving robust evaluation metrics like WER, CER, and BLEU in multilingual settings.
- It serves as a blueprint for standardizing low-resource NLP by integrating metadata, licensing practices, and community-driven evaluation across diverse language resources.
Swa-bhasha Resource Hub denotes a resource-centered infrastructure for low-resource South Asian language technology. In its explicitly released form, it consolidates datasets, algorithms, and ready-to-use tools for Romanized Sinhala (“Singlish”) to Sinhala script transliteration developed between 2020 and 2025, with an emphasis on ad-hoc typing patterns, contextual disambiguation, and standardized evaluation (Sumanathilaka et al., 12 Jul 2025). In related work, the same hub is proposed as a broader cataloging and benchmarking layer for speech corpora, multilingual translation assets, language identification resources, metadata schemas, leaderboards, and community-governed integration workflows for Indic and neighboring languages (Sharma et al., 8 Mar 2026, Kumar et al., 20 Apr 2026).
1. Definition, scope, and institutional setting
The hub’s most concrete definition appears in the Sinhala transliteration literature, where it is described as consolidating datasets, algorithms, and ready-to-use tools for Romanized Sinhala to Sinhala script transliteration. Its stated purpose is to enable robust back-transliteration under informal, ad-hoc typing patterns prevalent in social media and messaging, and to provide standardized benchmarks for evaluating context-aware disambiguation. The principal contributors named in that work are Deshan Sumanathilaka, Sameera Perera, Sachithya Dharmasiri, Maneesha Athukorala, Anuja Dilrukshi Herath, Rukshan Dias, Pasindu Gamage, Ruvan Weerasinghe, and Y.H.P.P. Priyadarshana (Sumanathilaka et al., 12 Jul 2025).
Related blueprints expand the hub from a transliteration repository into a task-centric platform for Indian and South Asian NLP. The survey literature organizes the surrounding landscape into text, speech, and multimodal modalities, and identifies a resource environment spanning 200+ datasets, 50+ benchmarks, and 100+ models, tools, and systems. Integration-oriented papers further recommend that the hub maintain standardized metadata, reproducible splits, model cards, evaluation templates, and ethical notes, especially for languages with sparse digital infrastructure (Kumar et al., 20 Apr 2026, Poria et al., 15 Sep 2025).
This suggests that “Swa-bhasha Resource Hub” is best understood not as a single dataset or model family, but as a unifying repository-and-benchmark concept whose first mature implementation is in Sinhala back-transliteration and whose proposed extensions cover ASR, MT, LID, and cross-lingual transfer.
2. Sinhala transliteration as the hub’s foundational implementation
The Sinhala component is the most fully specified part of the hub. It covers informal Singlish inputs, standardized romanization, contextual ambiguity, and shared-task evaluation. The resource base combines large lexicons, sentence-level corpora, ambiguity test sets, and multiple transliteration systems ranging from rule-based pipelines to BERT-based contextual disambiguation and Transformer seq2seq models (Sumanathilaka et al., 12 Jul 2025, Athukorala et al., 2024).
| Resource family | Contents | Notable details |
|---|---|---|
| Swa-Bhasha Dataset (2024) | Word-level, sentence-pair, and disambiguation resources | 7,134,803 Romanized words from 440,024 unique Sinhala root words |
| Augmented Sinhala–Romanized dataset | Standardized romanization corpus | ~7.24 million sentence pairs |
| IndoNLP 2025 evaluation sets | Reverse transliteration test sets | 10,000 general-pattern items and 5,000 ad-hoc-pattern items |
| Transliteration Disambiguation dataset | Contextual ambiguity benchmark | 660 sentence pairs, 22 ambiguous Romanized words |
The data construction pipeline is unusually explicit. Sources include user-generated social media content, especially YouTube, and Google’s Dakshina dataset. Surveys with 215 participants and a follow-up with 25 participants were used to capture real-world typing behavior. Rule repositories include 92 general rules and 26 special rules, plus a rule-based generator with 60 vowel/consonant rules, 18 hal marker rules, 18 special character rules, and an ad-hoc generator with 12 character-pattern rules, 6 vowel combination rules, and 8 special character rules. Validation used the LTRL framework and a Sinhala lexicon from the University of Colombo NLP Society (Sumanathilaka et al., 12 Jul 2025).
The algorithmic timeline is also well defined. The rule-based “message-based” system maps Romanized inputs through numeric coding and fuzzy matching, including vowel-absent forms such as khmd → කොහොමද and variant spellings such as kiyanna / kianna / kiynna → කියන්න (Athukorala et al., 2024). The 2023 hybrid system adds N-gram tagging and a Trie-based suggester. “Swa Bhasha 2.0” introduces a GRU seq2seq model with attention. A 2025 BERT-based masked-language-model pipeline handles contextual ambiguity by generating candidate sentences and ranking them with Sinhala BERT logits. Experimental mBART-50 models extend the stack with multilingual seq2seq and a custom Romanized Sinhala tokenizer (Sumanathilaka et al., 12 Jul 2025).
Benchmarking strongly favors contextual neural models. On the IndoNLP 2025 Sinhala test sets, Fine-tuned BERT attains WER 0.0867, CER 0.0200, and BLEU 0.9133 on Set 1, and WER 0.0903, CER 0.0215, and BLEU 0.9099 on Set 2. On the Transliteration Disambiguation dataset, Fine-tuned BERT reaches F1 0.9626 on Set 1 and 0.9389 on Set 2, substantially above rule-only or hybrid baselines (Sumanathilaka et al., 12 Jul 2025).
3. Core technical principles and evaluation design
A recurring design principle in the hub materials is script fidelity. For Sinhala, the target is native Sinhala Unicode rather than Romanized output. For Nepal Bhasha ASR, the benchmark is explicitly script-preserving in Devanagari, with no transliteration in either training or scoring. More generally, integration documents recommend Unicode normalization, script-aware tokenization, and explicit metadata for language, script, dialect, and domain (Sharma et al., 8 Mar 2026, Mujadia et al., 2024).
The evaluation layer is correspondingly metric-centered. For transliteration, the reported metrics include Accuracy, Character Error Rate, Word Error Rate, BLEU, and F1 for disambiguation. For speech tasks, the recommended primary metrics are CER and WER. The formulas are stated in the source materials as
with the denominator interpreted at the character or word level depending on the metric (Sumanathilaka et al., 12 Jul 2025, Sharma et al., 8 Mar 2026).
The transliteration systems also illustrate the hub’s preference for modular, benchmarkable pipelines. The original Sinhala word-level system tokenizes a Singlish word, assigns each Roman letter a unique numeric code, detects whether vowels are absent, inserts candidate vowels according to pre-defined templates, merges the resulting sequence into a numeric string, and retrieves candidate Sinhala words through fuzzy matching. The paper formalizes this as
followed by template-based vowel insertion and merged-value lookup (Athukorala et al., 2024).
At the repository level, integration plans emphasize dataset cards, checksums, versioning, file-level metadata, speaker-level metadata for speech, and leaderboards that record model, training hours, augmentation, LM usage, decoding settings, parameters, and compute budget. This is proposed both for ASR and for multilingual translation assets, and it reflects an attempt to make low-resource work reproducible rather than merely downloadable (Sharma et al., 8 Mar 2026, Mujadia et al., 2024).
4. Speech, language identification, and benchmark expansion
The hub’s proposed expansion beyond transliteration is most explicit in speech-resource integration. “Nwāchā Munā” introduces a 5.39-hour manually transcribed Devanagari speech corpus for Nepal Bhasha comprising 5,727 utterances from 18 native speakers, with train, validation, and test splits of 4.31, 0.54, and 0.54 hours respectively. The benchmark compares a Nepali Conformer-CTC model against Whisper-Small and shows that proximal transfer from Nepali reduces CER from 52.54% zero-shot to 17.59% with augmentation, effectively matching Whisper-Small at 17.88% while using far fewer parameters and compute (Sharma et al., 8 Mar 2026).
That work also gives a direct hub integration recipe. Recommended per-utterance metadata include utt_id, speaker_id, split, audio_path, transcript, and duration_sec; per-speaker metadata include speaker_id, gender, age_group, region, and dialect notes. It further recommends mirroring the corpus into hub storage, publishing a dataset card, versioning the initial release as v1.0, and exposing standardized CER/WER scoring scripts (Sharma et al., 8 Mar 2026).
A much larger candidate speech ingestion is “SPRING-INX,” which contributes about 2000 hours of legally sourced and manually transcribed speech data across Assamese, Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Odia, Punjabi, and Tamil. The delivered corpus uses WAV, 16-bit Linear PCM, mono audio at 16 kHz, and provides fixed 5-hour test sets per language. The associated guidance recommends Kaldi/ESPnet-style data directories, metadata fields for speech mode, code-mixing, dialect tags, and domain labels, and direct linkage to ESPnet recipes (R et al., 2023).
Accent-robust English ASR can also be positioned within the hub. “Svarah” supplies a 9.6-hour evaluation benchmark of Indian-accented English from 117 speakers across 65 districts, with both read and spontaneous speech. On this benchmark, Whisper-large records WER 7.2, outperforming all other evaluated systems, while use-case utterances containing entities such as bank names, product names, and IDs remain notably harder than read or extempore speech (Javed et al., 2023).
Language identification is treated as another enabling layer. “Bhasha-Abhijnaanam” provides native-script and romanized test sets for the 22 constitutional languages, while IndicLID supplies fastText-based native and romanized classifiers, an IndicBERT-based roman classifier, and a two-stage ensemble. The datasets are released under CC0 1.0 Universal and the code and models under the MIT License. The ensemble uses a probability threshold of 0.6 to route difficult romanized examples from fastText to IndicBERT, improving romanized accuracy to about 80.40 with macro-F1 about 74.72 (Madhani et al., 2023).
5. Translation, cross-lingual transfer, and deployment-oriented extensions
The hub’s proposed translation layer is heterogeneous. “VAKTA-SETU” contributes a deployment-ready speech-to-speech and text-to-text machine translation service for English, Hindi, and Marathi. Its cascade is ASR Disfluency Correction MT TTS, and its deployment profile is unusually concrete: on an NVIDIA DGX A100 with 8 A100 80GB GPUs, each SSMT pipeline occupies about 6 GB of GPU memory, 13 pipelines are deployed per GPU, and the system reports a median response time of 4.4 seconds for 1000 concurrent users (Mhaskar et al., 2023).
At the corpus level, “BhashaSetu: A Data-Centric Approach to Low-Resource Machine Translation” adds a linguistically enriched English–Marathi dataset with 2,779,901 aligned sentence pairs after preprocessing, released as CSV with raw, tokenized, stemmed, and lemmatized columns. Its main technical finding is that corpus-level deduplication is the single largest preprocessing contributor to downstream quality: removing it reduces performance by 1.17 BLEU and 2.21 chrF++ in the ablation summary, and the paired table also reports a 6.51 to 4.87 BLEU drop and a 42.02 to 38.69 chrF++ drop when deduplication is removed (Thakkar et al., 26 May 2026).
For extreme low-resource learning, a different “BhashaSetu” studies cross-lingual transfer from Hindi and English into languages with only about 100 labeled training instances. It combines HAL, TET, and GETR, and reports improvements of 13 percentage points on Mizo and Khasi POS tagging, and 20 and 27 percentage point gains in macro-F1 on simulated low-resource sentiment and NER settings. These materials are directly relevant to a hub that aims to support languages for which benchmark construction is itself difficult (Maji et al., 5 Feb 2026).
The most ambitious hub-wide translation blueprint is “BhashaVerse,” which proposes a translation ecosystem for 36 Indian languages. Its developed corpora include 1B+ parallel sentence pairs across 325 language pairs and 15 scripts, 2M human post-edited domain-parallel sentences, and 72,000 human-rated translations on a 1–100 scale. The accompanying pipeline prescribes COMET-QE-based alignment, multi-script normalization for languages such as Kashmiri, Sindhi, and Manipuri, paragraph-level back-translation, APE training, REST/gRPC serving, and a metadata schema keyed by language code, script, domain, source, license, quality metrics, and synthetic-data flags (Mujadia et al., 2024).
Deployment under resource constraints is addressed by “Bhasha-Rupantarika,” which couples distilled NLLB-200 with FPGA-oriented ultra-low-precision execution. In FP4, the model footprint is reduced to 0.56 GB, corresponding to a 4.1× size reduction, and measured throughput reaches 66 tokens/s, corresponding to a 4.2× speedup and a 4.8× throughput improvement. For a hub that aspires to serve low-resource settings rather than only benchmark them, these numbers are significant because they connect multilingual coverage to deployable hardware profiles (Lokhande et al., 12 Oct 2025).
6. Limitations, governance, and unresolved questions
A recurrent limitation across hub-related resources is incomplete distribution metadata. Several papers state that datasets or checkpoints are openly released but do not specify a license, DOI, or stable download URL. This is explicit for Nwāchā Munā, SPRING-INX, Svarah, VAKTA-SETU, and several MT corpora. As a result, many integration documents recommend requesting official links, license terms, and checksums before mirroring assets into hub storage (Sharma et al., 8 Mar 2026, R et al., 2023).
Coverage is also uneven. Survey work stresses that high-resource languages such as Hindi, Bengali, Tamil, Telugu, Marathi, Urdu, Gujarati, Kannada, and Malayalam dominate across tasks, while Assamese, Odia, Punjabi, Kashmiri, Sindhi, Nepali, Sinhala, Santali, Mizo, Bhojpuri, Braj, Magahi, and other low-resource or marginalized varieties remain patchily represented. The same surveys identify persistent gaps in healthcare, legal, and culturally grounded evaluation, along with continuing problems in script diversity, code-mixing, dialect variation, and evaluation fragmentation (Kumar et al., 20 Apr 2026, Poria et al., 15 Sep 2025).
Community governance is treated as a technical as well as ethical requirement. Hub integration plans recommend culturally respectful deployments, participation of language communities in evaluation, mechanisms for orthography corrections and dialect feedback, explicit ethics statements, and restrictions against sensitive deployments. In the Nepal Bhasha ASR plan, for example, community feedback, orthography correction, and local-benefit applications such as voice-search, dictation for local government, and education in Bagmati Province are specifically encouraged (Sharma et al., 8 Mar 2026).
A common misconception would be to treat Swa-bhasha Resource Hub as only a Romanized Sinhala transliteration repository. The released Sinhala work does document a mature transliteration stack, but related papers explicitly position the hub as a much broader infrastructure for speech corpora, ASR benchmarks, multilingual translation, cross-lingual transfer, LID, metadata standardization, and reproducible leaderboards (Sumanathilaka et al., 12 Jul 2025, Sharma et al., 8 Mar 2026). A second misconception would be to equate openness with immediate reusability: the source materials repeatedly show that openness without explicit licensing, checksums, or stable URLs remains operationally incomplete.
In that sense, Swa-bhasha Resource Hub is both a concrete resource collection and a larger program of standardization. Its present substance lies in released transliteration datasets, models, and evaluation sets; its broader significance lies in the attempt to turn low-resource South Asian NLP from a disconnected set of one-off corpora into a benchmarked, interoperable, and community-accountable ecosystem.