Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
132 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

UMBCLU at SemEval-2024 Task 1A and 1C: Semantic Textual Relatedness with and without machine translation (2402.12730v2)

Published 20 Feb 2024 in cs.CL, cs.AI, and cs.LG

Abstract: The aim of SemEval-2024 Task 1, "Semantic Textual Relatedness for African and Asian Languages" is to develop models for identifying semantic textual relatedness (STR) between two sentences using multiple languages (14 African and Asian languages) and settings (supervised, unsupervised, and cross-lingual). LLMs have shown impressive performance on several natural language understanding tasks such as multilingual machine translation (MMT), semantic similarity (STS), and encoding sentence embeddings. Using a combination of LLMs that perform well on these tasks, we developed two STR models, $\textit{TranSem}$ and $\textit{FineSem}$, for the supervised and cross-lingual settings. We explore the effectiveness of several training methods and the usefulness of machine translation. We find that direct fine-tuning on the task is comparable to using sentence embeddings and translating to English leads to better performance for some languages. In the supervised setting, our model performance is better than the official baseline for 3 languages with the remaining 4 performing on par. In the cross-lingual setting, our model performance is better than the baseline for 3 languages (leading to $1{st}$ place for Africaans and $2{nd}$ place for Indonesian), is on par for 2 languages and performs poorly on the remaining 7 languages. Our code is publicly available at https://github.com/dipta007/SemEval24-Task8.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. What makes sentences semantically related: A textual relatedness dataset and empirical study. CoRR, abs/2110.04845.
  2. Semantic relatedness retroactively boosts memory and promotes memory interdependence across episodes. eLife, 11:e72519.
  3. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
  4. Satanjeev Banerjee and Ted Pedersen. 2003. Extended gloss overlaps as a measure of semantic relatedness. In Proceedings of the 18th International Joint Conference on Artificial Intelligence, IJCAI’03, page 805–810, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  5. Paracrawl: Web-scale acquisition of parallel corpora. Association for Computational Linguistics (ACL).
  6. Translation semantic variability: How semantic relatedness affects learning of translation-ambiguous words. Bilingualism: Language and Cognition, 20(4):783–794.
  7. Alexander Budanitsky and Graeme Hirst. 2006. Evaluating WordNet-based Measures of Lexical Semantic Relatedness. Computational Linguistics, 32(1):13–47.
  8. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  9. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
  10. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.
  11. C-sts: Conditional semantic textual similarity.
  12. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  13. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852.
  14. The state of the art in semantic relatedness: a framework for comparison. The Knowledge Engineering Review, 32:e10.
  15. Survey of low-resource machine translation. Computational Linguistics, 48(3):673–732.
  16. Mario Jarmasz and Stanialaw Szpakowicz. 2012. Roget’s thesaurus and semantic similarity. In Recent Advances in Natural Language Processing.
  17. Mistral 7b. arXiv preprint arXiv:2310.06825.
  18. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796, Brussels, Belgium. Association for Computational Linguistics.
  19. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
  20. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  21. George A. Miller. 1992. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992.
  22. Toan Q Nguyen and David Chiang. 2017. Transfer learning across low-resource, related languages for neural machine translation. arXiv preprint arXiv:1708.09803.
  23. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. CoRR, abs/2108.08877.
  24. C. E. Osgood. 1949. The similarity paradox in human learning: a resolution. 56:132–143.
  25. Semrel2024: A collection of semantic textual relatedness datasets for 14 languages.
  26. SemEval-2024 task 1: Semantic textual relatedness. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024).
  27. Transforming machine translation: a deep learning system reaches news translation quality comparable to human professionals. 11.
  28. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1–67.
  29. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  30. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv: Computation and Language.
  31. Semantically-informed hierarchical event modeling. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 353–369, Toronto, Canada. Association for Computational Linguistics.
  32. Ccmatrix: Mining billions of high-quality parallel sentences on the web. arXiv preprint arXiv:1911.04944.
  33. Attention is all you need. Advances in neural information processing systems, 30.
  34. Just rank: Rethinking evaluation with word and sentence similarities. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6060–6077, Dublin, Ireland. Association for Computational Linguistics.
  35. Learning semantic textual similarity from conversations. In Proceedings of the Third Workshop on Representation Learning for NLP, pages 164–174, Melbourne, Australia. Association for Computational Linguistics.
  36. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36.
  37. Transfer learning for low-resource neural machine translation. arXiv preprint arXiv:1604.02201.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets