Breaking Language Barriers: A Question Answering Dataset for Hindi and Marathi (2308.09862v3)
Abstract: The recent advances in deep-learning have led to the development of highly sophisticated systems with an unquenchable appetite for data. On the other hand, building good deep-learning models for low-resource languages remains a challenging task. This paper focuses on developing a Question Answering dataset for two such languages- Hindi and Marathi. Despite Hindi being the 3rd most spoken language worldwide, with 345 million speakers, and Marathi being the 11th most spoken language globally, with 83.2 million speakers, both languages face limited resources for building efficient Question Answering systems. To tackle the challenge of data scarcity, we have developed a novel approach for translating the SQuAD 2.0 dataset into Hindi and Marathi. We release the largest Question-Answering dataset available for these languages, with each dataset containing 28,000 samples. We evaluate the dataset on various architectures and release the best-performing models for both Hindi and Marathi, which will facilitate further research in these languages. Leveraging similarity tools, our method holds the potential to create datasets in diverse languages, thereby enhancing the understanding of natural language across varied linguistic contexts. Our fine-tuned models, code, and dataset will be made publicly available.
- The question answering systems: A survey. International Journal of Research and Reviews in Information Sciences (IJRRIS), 2(3).
- HindiRC: A Dataset for Reading Comprehension in Hindi. In 20th International Conference on Computational Linguistics and Intelligent Text.
- On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
- Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8440–8451. Online: Association for Computational Linguistics.
- Question Answering System Using Ontology in Marathi Language.
- MMQA: A Multi-domain Multi-lingual Question-Answering Framework for English and Hindi. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association (ELRA).
- MMQA: A Multi-domain Multi-lingual Question-Answering Framework for English and Hindi. In chair), N. C. C.; Choukri, K.; Cieri, C.; Declerck, T.; Goggi, S.; Hasida, K.; Isahara, H.; Maegaard, B.; Mariani, J.; Mazo, H.; Moreno, A.; Odijk, J.; Piperidis, S.; and Tokunaga, T., eds., Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018). Miyazaki, Japan: European Language Resources Association (ELRA). ISBN 979-10-95546-00-9.
- BERT Based Multilingual Machine Comprehension in English and Hindi.
- chaii - Hindi and Tamil Question Answering.
- Joshi, R. 2022a. L3Cube-HindBERT and DevBERT: Pre-Trained BERT Transformer models for Devanagari based Hindi and Marathi Languages. arXiv preprint arXiv:2211.11418.
- Joshi, R. 2022b. L3Cube-MahaCorpus and MahaBERT: Marathi Monolingual Corpus, Marathi BERT Language Models, and Resources. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, 97–101. Marseille, France: European Language Resources Association.
- Joshi, R. 2022c. L3Cube-MahaNLP: Marathi Natural Language Processing Datasets, Models, and Library. arXiv preprint arXiv:2205.14728.
- Kamble S., B. S. 2018. Learning to Classify Marathi Questions and Identify Answer Type Using Machine Learning Technique.
- Adam: A Method for Stochastic Optimization. arXiv:1412.6980.
- MuCoT: Multilingual Contrastive Training for Question-Answering in Low-resource Languages.
- Natural Questions: a Benchmark for Question Answering Research. Transactions of the Association of Computational Linguistics.
- MLQA: Evaluating Cross-lingual Extractive Question Answering. arXiv preprint arXiv:1910.07475.
- MLQA: Evaluating Cross-lingual Extractive Question Answering.
- Lin, C.-Y. 2004. ROUGE: A Package for Automatic Evaluation of summaries. 10.
- L3Cube-MahaNER: A Marathi Named Entity Recognition Dataset and BERT models. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, 29–34. Marseille, France: European Language Resources Association.
- RoBERTa: A Robustly Optimized BERT Pretraining Approach. arXiv:1907.11692.
- BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, 311–318. USA: Association for Computational Linguistics.
- PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv:1912.01703.
- Question Answering System for low resource language using Transfer Learning. In 2021 International Conference on Computational Intelligence and Computing Applications (ICCICA), 1–6.
- How multilingual is Multilingual BERT? arXiv:1906.01502.
- Know What You Don’t Know: Unanswerable Questions for SQuAD.
- SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv:1606.05250.
- Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages. Transactions of the Association for Computational Linguistics, 10: 145–162.
- CoQA: A Conversational Question Answering Challenge.
- DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108.
- Database Creation for Marathi QA System.
- Attention Is All You Need. arXiv:1706.03762.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.
