Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Prometheus: Inducing Fine-grained Evaluation Capability in Language Models (2310.08491v2)

Published 12 Oct 2023 in cs.CL and cs.LG
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

Abstract: Recently, using a powerful proprietary LLM (e.g., GPT-4) as an evaluator for long-form responses has become the de facto standard. However, for practitioners with large-scale evaluation tasks and custom criteria in consideration (e.g., child-readability), using proprietary LLMs as an evaluator is unreliable due to the closed-source nature, uncontrolled versioning, and prohibitive costs. In this work, we propose Prometheus, a fully open-source LLM that is on par with GPT-4's evaluation capabilities when the appropriate reference materials (reference answer, score rubric) are accompanied. We first construct the Feedback Collection, a new dataset that consists of 1K fine-grained score rubrics, 20K instructions, and 100K responses and language feedback generated by GPT-4. Using the Feedback Collection, we train Prometheus, a 13B evaluator LLM that can assess any given long-form text based on customized score rubric provided by the user. Experimental results show that Prometheus scores a Pearson correlation of 0.897 with human evaluators when evaluating with 45 customized score rubrics, which is on par with GPT-4 (0.882), and greatly outperforms ChatGPT (0.392). Furthermore, measuring correlation with GPT-4 with 1222 customized score rubrics across four benchmarks (MT Bench, Vicuna Bench, Feedback Bench, Flask Eval) shows similar trends, bolstering Prometheus's capability as an evaluator LLM. Lastly, Prometheus achieves the highest accuracy on two human preference benchmarks (HHH Alignment & MT Bench Human Judgment) compared to open-sourced reward models explicitly trained on human preference datasets, highlighting its potential as an universal reward model. We open-source our code, dataset, and model at https://kaistai.github.io/prometheus/.

PDF HTML Abstract

Introducing Prometheus: Enabling Fine-grained Evaluation with Open-source LLMs

Overview

Recent advances in NLP have positioned LLMs as potent tools for evaluating machine-generated text. However, reliance on proprietary models like GPT-4 presents challenges including lack of transparency, version control issues, and financial barriers. Addressing these concerns, this paper introduces Prometheus, a 13B parameter open-source LLM designed to rival the evaluation capabilities of GPT-4. Prometheus, leveraging the newly compiled Feedback Collection dataset, demonstrates remarkable proficiency in evaluating long-form responses across diverse custom score rubrics, showcasing its potential as a versatile and accessible evaluator.

The Feedback Collection Dataset

The Feedback Collection dataset stands out for its unique architecture, designed specifically to enhance fine-grained evaluation capabilities of LLMs. Consisting of 1K fine-grained score rubrics, 20K instructions, and 100K responses with language feedback generated using GPT-4, it introduces a sophisticated framework for instruction-based evaluation. Each dataset instance comprises multiple components, including sophisticated rubrics and reference answers, to enable nuanced understanding and evaluation of text responses. This design not only facilitates detailed feedback generation but also quantitatively rates responses, paving the way for comprehensive and tailored text evaluation.

Experimental Validation

Prometheus’s evaluation prowess was rigorously tested across various benchmarks and compared with human evaluators as well as GPT-4 and ChatGPT models. Key findings are as follows:

  • Correlation with Human Evaluators: Prometheus achieves a Pearson correlation coefficient of 0.897 on 45 customized score rubrics, demonstrating parity with GPT-4 (0.882) and significantly surpassing ChatGPT (0.392).
  • Feedback Quality: In pairwise comparisons, feedback generated by Prometheus is preferred over GPT-4 58.67% of the time, highlighting its superior ability to generate meaningful and critical feedback.
  • Universal Reward Model Potential: Prometheus also excels in two human preference benchmarks, outperforming open-source reward models trained explicitly on human preferences.

These results underscore Prometheus’s adeptness not only in emulating human evaluation standards but also in its utility as a universal reward model, offering insights into its potential applications in model training and development.

Implications and Future Directions

Prometheus challenges the prevailing dependence on proprietary LLMs for text evaluation by offering an open-source alternative that does not sacrifice performance. The inclusion of reference materials such as score rubrics and reference answers proves vital in its success, suggesting avenues for further enhancing LLM evaluators. Additionally, the performance on ranking grading schemes suggests Prometheus's adaptability as a reward model for various AI training methodologies, marking a significant stride toward developing versatile, transparent, and accessible evaluation tools in NLP.

The open-sourcing of Prometheus, along with the Feedback Collection dataset, not only democratizes access to high-quality evaluation tools but also encourages community collaboration in refining and expanding upon this foundational work. Future research could explore domain-specific evaluator models, further diversify evaluation criteria, and integrate Prometheus into broader AI training and development workflows, paving the way for innovative applications and methodologies in artificial intelligence research.

Acknowledgeably, this research initiates a crucial discussion on transparency, autonomy, and accessibility in AI evaluation, setting a precedent for future endeavors in the field. Prometheus not only signifies a step forward in LLM evolution but also embodies the collaborative spirit essential for sustainable progress in AI research and development.

File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. A general language assistant as a laboratory for alignment, 2021.
  2. A survey on evaluation of large language models. arXiv preprint arXiv:2307.03109, 2023.
  3. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint arXiv:2306.04757, 2023.
  4. Can large language models be an alternative to human evaluations?, 2023.
  5. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/.
  6. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.
  7. Chatgpt outperforms crowd-workers for text-annotation tasks. arXiv preprint arXiv:2303.15056, 2023.
  8. Large language models are reasoning teachers. arXiv preprint arXiv:2212.10071, 2022.
  9. Generative models as a complex systems science: How can we make sense of large language model behavior? arXiv preprint arXiv:2308.00189, 2023.
  10. Unnatural instructions: Tuning language models with (almost) no human labor. arXiv preprint arXiv:2212.09689, 2022.
  11. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning. arXiv preprint arXiv:2305.14045, 2023a.
  12. Aligning large language models through synthetic feedback. arXiv preprint arXiv:2305.13735, 2023b.
  13. Hurdles to progress in long-form question answering. arXiv preprint arXiv:2103.06332, 2021.
  14. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 2023.
  15. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110, 2022.
  16. Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp.  74–81, Barcelona, Spain, July 2004a. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013.
  17. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.  74–81, 2004b.
  18. Gpteval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634, 2023.
  19. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023.
  20. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp.  311–318, 2002.
  21. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023.
  22. On the challenges of using black-box apis for toxicity evaluation in research, 2023.
  23. Bleurt: Learning robust metrics for text generation. In Annual Meeting of the Association for Computational Linguistics, 2020. URL https://api.semanticscholar.org/CorpusID:215548699.
  24. Llama 2: Open foundation and fine-tuned chat models, 2023.
  25. Shepherd: A critic for language model generation. arXiv preprint arXiv:2308.04592, 2023a.
  26. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. arXiv preprint arXiv:2306.05087, 2023b.
  27. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  28. Selfee: Iterative self-revising llm empowered by self-feedback generation. Blog post, May 2023a. URL https://kaistai.github.io/SelFee/.
  29. Flask: Fine-grained language model evaluation based on alignment skill sets. arXiv preprint arXiv:2307.10928, 2023b.
  30. BARTScore: Evaluating generated text as text generation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=5Ya8PbvpZ9.
  31. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019.
  32. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.05685, 2023.
  33. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (11)
  1. Seungone Kim (34 papers)
  2. Jamin Shin (24 papers)
  3. Yejin Cho (3 papers)
  4. Joel Jang (30 papers)
  5. Shayne Longpre (49 papers)
  6. Hwaran Lee (31 papers)
  7. Sangdoo Yun (71 papers)
  8. Seongjin Shin (5 papers)
  9. Sungdong Kim (30 papers)
  10. James Thorne (48 papers)
  11. Minjoon Seo (82 papers)
Citations (137)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube