Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SSA-COMET: Do LLMs Outperform Learned Metrics in Evaluating MT for Under-Resourced African Languages? (2506.04557v1)

Published 5 Jun 2025 in cs.CL and cs.AI

Abstract: Evaluating machine translation (MT) quality for under-resourced African languages remains a significant challenge, as existing metrics often suffer from limited language coverage and poor performance in low-resource settings. While recent efforts, such as AfriCOMET, have addressed some of the issues, they are still constrained by small evaluation sets, a lack of publicly available training data tailored to African languages, and inconsistent performance in extremely low-resource scenarios. In this work, we introduce SSA-MTE, a large-scale human-annotated MT evaluation (MTE) dataset covering 13 African language pairs from the News domain, with over 63,000 sentence-level annotations from a diverse set of MT systems. Based on this data, we develop SSA-COMET and SSA-COMET-QE, improved reference-based and reference-free evaluation metrics. We also benchmark prompting-based approaches using state-of-the-art LLMs like GPT-4o and Claude. Our experimental results show that SSA-COMET models significantly outperform AfriCOMET and are competitive with the strongest LLM (Gemini 2.5 Pro) evaluated in our study, particularly on low-resource languages such as Twi, Luo, and Yoruba. All resources are released under open licenses to support future research.

User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (10)
  1. Senyu Li (5 papers)
  2. Jiayi Wang (74 papers)
  3. Felermino D. M. A. Ali (3 papers)
  4. Colin Cherry (38 papers)
  5. Daniel Deutsch (28 papers)
  6. Eleftheria Briakou (21 papers)
  7. Rui Sousa-Silva (2 papers)
  8. Henrique Lopes Cardoso (13 papers)
  9. Pontus Stenetorp (68 papers)
  10. David Ifeoluwa Adelani (59 papers)

Summary

We haven't generated a summary for this paper yet.