Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
169 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task (2407.20663v1)

Published 30 Jul 2024 in cs.CL

Abstract: This paper presents an overview of the Arabic Natural Language Understanding (ArabicNLU 2024) shared task, focusing on two subtasks: Word Sense Disambiguation (WSD) and Location Mention Disambiguation (LMD). The task aimed to evaluate the ability of automated systems to resolve word ambiguity and identify locations mentioned in Arabic text. We provided participants with novel datasets, including a sense-annotated corpus for WSD, called SALMA with approximately 34k annotated tokens, and the IDRISI-DA dataset with 3,893 annotations and 763 unique location mentions. These are challenging tasks. Out of the 38 registered teams, only three teams participated in the final evaluation phase, with the highest accuracy being 77.8% for WSD and the highest MRR@1 being 95.0% for LMD. The shared task not only facilitated the evaluation and comparison of different techniques, but also provided valuable insights and resources for the continued advancement of Arabic NLU technologies.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (60)
  1. Reem Abdel-Salam. 2024. Rematchka at arabicnlu shared task: Evaluating large language models for arabicword sense and location sense disambiguation. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024) Part of ACL 2024.
  2. Mohammed Alaeddine Abderrahim and Mohammed El Amine Abderrahim. 2022. Arabic word sense disambiguation for information retrieval. ACM Trans. Asian Low Resour. Lang. Inf. Process., 21(4):69:1–69:19.
  3. You tweet what you speak: A city-level dataset of Arabic dialects. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  4. NADI 2023: The fourth nuanced Arabic dialect identification shared task. In Proceedings of ArabicNLP 2023, pages 600–613, Singapore (Hybrid). Association for Computational Linguistics.
  5. ARBERT & MARBERT: deep bidirectional transformers for arabic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 7088–7105. Association for Computational Linguistics.
  6. Miuru Abeysiriwardana and Deshan Sumanathilaka. 2024. A survey on lexical ambiguity detection and word sense disambiguation. CoRR, abs/2403.16129.
  7. Abdul-Ghani Abul-Azm. 2014. Al-ghani al-zaher dictionary. Rabat: Al-Ghani Publishing Institution.
  8. AI@Meta. 2024. Llama 3 model card.
  9. Moustafa Al-Hajj and Mustafa Jarrar. 2021a. ArabGlossBERT: Fine-Tuning BERT on Context-Gloss Pairs for WSD. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 40–48, Online. INCOMA Ltd.
  10. Moustafa Al-Hajj and Mustafa Jarrar. 2021b. LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2vec Performance in Arabic Word-in-Context Disambiguation. In Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021), pages 748–755, Online. Association for Computational Linguistics.
  11. Arabert: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15.
  12. Vedanth Baiju. 2022. Word sense disambiguation in the domain of sentiment analysis through deep learning.
  13. Implementing an arabic question answering system using conceptual graphs. In Hybrid Intelligent Systems - 21st International Conference on Hybrid Intelligent Systems (HIS 2021), December 14-16, 2021, volume 420 of Lecture Notes in Networks and Systems, pages 295–304. Springer.
  14. Are Large Language Models the New Interface for Data Pipelines? In Proceedings of the International Workshop on Big Data in Emergent Distributed Environments, BiDEDE ’24, New York, NY, USA. Association for Computing Machinery.
  15. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: on meaning, form, and understanding in the age of data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 5185–5198. Association for Computational Linguistics.
  16. Comparative study of rocchio classifier applied to supervised wsd using arabic lexical samples. In Proceedings of the tenth conference of language engeneering (SEOLEC’2010), Cairo, Egypt.
  17. Arabic gloss wsd using bert. Applied Sciences, 11(6):2567.
  18. Bilel Elayeb. 2019. Arabic word sense disambiguation: a review. Artif. Intell. Rev., 52(4):2475–2532.
  19. Curras + Baladi: Towards a Levantine Corpus. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  20. Acegpt, localizing large language models in arabic. CoRR, abs/2309.12053.
  21. Mustafa Jarrar. 2021. The Arabic Ontology - An Arabic Wordnet with Ontologically Clean Content. Applied Ontology Journal, 16(1):1–26.
  22. WojoodNER 2023: The First Arabic Named Entity Recognition Shared Task. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 748–758. ACL.
  23. ArBanking77: Intent Detection Neural Model and a New Dataset in Modern and Dialectical Arabic. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 276–287. ACL.
  24. WojoodNER 2024: The Second Arabic Named Entity Recognition Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  25. Mustafa Jarrar and Tymaa Hasanain Hammouda. 2024. Qabas: An Open-Source Arabic Lexicographic Database. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 13363–13370, Torino, Italy. ELRA and ICCL.
  26. Wojood: Nested Arabic Named Entity Corpus and Recognition using BERT. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2022), Marseille, France.
  27. SALMA: Arabic Sense-annotated Corpus and WSD Benchmarks. In Proceedings of the 1st Arabic Natural Language Processing Conference (ArabicNLP), Part of the EMNLP 2023, pages 359–369. ACL.
  28. Lisan: Yemeni, Irqi, Libyan, and Sudanese Arabic Dialect Copora with Morphological Annotations. In The 20th IEEE/ACS International Conference on Computer Systems and Applications (AICCSA). IEEE.
  29. Joint recognition and linking of fine-grained locations from tweets. In Proceedings of the 25th International Conference on World Wide Web, WWW ’16, page 1271–1281, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  30. Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  31. Abstractive text summarization: Enhancing sequence-to-sequence models using word sense disambiguation and semantic content generalization. Comput. Linguistics, 47(4):813–859.
  32. Large language models and multimodal retrieval for visual word sense disambiguation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 13053–13077. Association for Computational Linguistics.
  33. Effective location identification from microblogs. In 2014 IEEE 30th International Conference on Data Engineering, pages 880–891, Chicago, IL, USA. Institute of Electrical and Electronics Engineers (IEEE).
  34. UniMelb at SemEval-2019 task 12: Multi-model combination for toponym resolution. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1313–1318, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  35. AraFinNLP 2024: The First Arabic Financial NLP Shared Task. In Proceedings of the Second Arabic Natural Language Processing Conference (ArabicNLP 2024), Bangkok, Thailand. Association for Computational Linguistics.
  36. Context-Gloss Augmentation for Improving Arabic Target Sense Verification. In Proceedings of the 12th International Global Wordnet Conference (GWC2023). Global Wordnet Association.
  37. State of art for semantic analysis of natural language processing. Qubahan academic journal, 1(2):21–28.
  38. Location extraction from social media: Geoparsing, location disambiguation, and geotagging. ACM Transactions on Information Systems, 36(4):1–27.
  39. Introduction to wordnet: An on-line lexical database. International journal of lexicography, 3(4):235–244.
  40. Sangita S Modi and Sudhir B Jagtap. 2018. Web page classification using wsd and yago and ontology. In 2018 3rd International Conference on Communication and Electronics Systems (ICCES), pages 887–891. IEEE.
  41. Ahmed Mukhtar Omar. 2008. Contemporary arabic dictionary.(i1). World of Books, Cairo, Egypt. Retrieval Date, 14(8):2020.
  42. OpenAI. 2023. GPT-4 technical report. CoRR, abs/2303.08774.
  43. Contextad: Context-aware acronym disambiguation with siamese BERT network. Int. J. Intell. Syst., 2023:1–14.
  44. Setfit: A robust approach for offensive content detection in tamil-english code-mixed conversations using sentence transfer fine-tuning. In Proceedings of the Fourth Workshop on Speech, Vision, and Language Technologies for Dravidian Languages, pages 35–42.
  45. Fake news detection and sentiment analysis in twitter. International Journal for Research in Applied Science & Engineering Technology (IJRASET), 8:72–75.
  46. An evaluation benchmark for testing the word sense disambiguation capabilities of machine translation systems. In Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 3668–3675. European Language Resources Association.
  47. Upaya at arabicnlu shared-task: Arabic lexical disambiguation using large language models. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024) Part of ACL 2024.
  48. Coarse lexical semantic annotation with supersenses: An arabic case study. In The 50th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference, July 8-14, 2012, Jeju Island, Korea - Volume 2: Short Papers, pages 253–258. The Association for Computer Linguistics.
  49. Yujia Sun and Jan Platoš. 2023. Attention-based stacked bidirectional long short-term memory model for word sense disambiguation. ACM Transactions on Asian and Low-Resource Language Information Processing.
  50. IDRISI-D: arabic and english datasets and benchmarks for location mention disambiguation over disaster microblogs. In Proceedings of ArabicNLP 2023, Singapore (Hybrid), December 7, 2023, pages 158–169. Association for Computational Linguistics.
  51. IDRISI-RA: The first Arabic location mention recognition dataset of disaster tweets. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16298–16317, Toronto, Canada. Association for Computational Linguistics.
  52. Pirates at arabicnlu2024: Enhancing arabic word sense disambiguation using transformer-based approaches. In The Second Arabic Natural Language Processing Conference (ArabicNLP 2024) Part of ACL 2024.
  53. Openchat: Advancing open-source language models with mixed-quality data. CoRR, abs/2309.11235.
  54. Jimin Wang and Yingjie Hu. 2019. Are we there yet? evaluating state-of-the-art neural network based geoparsers using eupeg as a benchmarking platform. In Proceedings of the 3rd ACM SIGSPATIAL International Workshop on Geospatial Humanities, GeoHumanities ’19, New York, NY, USA. Association for Computing Machinery.
  55. DM_NLP at SemEval-2018 task 12: A pipeline system for toponym resolution. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 917–923, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  56. Ontonotes release 5.0 ldc2013t19. Philadelphia: Linguistic Data Consortium.
  57. Wizardlm: Empowering large language models to follow complex instructions. CoRR, abs/2304.12244.
  58. Dlocrl: A deep learning pipeline for fine-grained location recognition and linking in tweets. In The World Wide Web Conference, WWW 2019, pages 3391–3397, San Francisco, CA, USA. ACM.
  59. University of Arizona at SemEval-2019 task 12: Deep-affix named entity recognition of geolocation entities. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 1319–1323, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  60. Wei Zhang and Judith Gelernter. 2014. Geocoding location expressions in Twitter messages: A preference learning method. Journal of Spatial Information Science, 2014(9):37–70.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now