Papers
Topics
Authors
Recent
Search
2000 character limit reached

CPsyExam: A Chinese Benchmark for Evaluating Psychology using Examinations

Published 16 May 2024 in cs.CL | (2405.10212v3)

Abstract: In this paper, we introduce a novel psychological benchmark, CPsyExam, constructed from questions sourced from Chinese language examinations. CPsyExam is designed to prioritize psychological knowledge and case analysis separately, recognizing the significance of applying psychological knowledge to real-world scenarios. From the pool of 22k questions, we utilize 4k to create the benchmark that offers balanced coverage of subjects and incorporates a diverse range of case analysis techniques.Furthermore, we evaluate a range of existing LLMs~(LLMs), spanning from open-sourced to API-based models. Our experiments and analysis demonstrate that CPsyExam serves as an effective benchmark for enhancing the understanding of psychology within LLMs and enables the comparison of LLMs across various granularities.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (47)
  1. CHQ- SocioEmo: Identifying Social and Emotional Support Needs in Consumer-Health Questions. Scientific Data, 10(1):329.
  2. Findings of WASSA 2023 shared task on empathy, emotion and personality detection in conversation and reactions to news articles. In Proceedings of the 13th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 511–525, Toronto, Canada. Association for Computational Linguistics.
  3. WASSA 2022 shared task: Predicting empathy, emotion and personality in reaction to news stories. In Proceedings of the 12th Workshop on Computational Approaches to Subjectivity, Sentiment & Social Media Analysis, pages 214–227, Dublin, Ireland. Association for Computational Linguistics.
  4. Nicolas Bertagnolli. 2020. Counsel chat: Bootstrapping high-quality therapy data.
  5. Modeling empathy and distress in reaction to news stories. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4758–4765, Brussels, Belgium. Association for Computational Linguistics.
  6. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330, Dublin, Ireland. Association for Computational Linguistics.
  7. A survey on evaluation of large language models.
  8. A Benchmark for Automatic Medical Consultation System: Frameworks, Tasks and Datasets. Bioinformatics. Btac817.
  9. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass.
  10. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  11. Xiezhi: An ever-updating benchmark for holistic domain knowledge evaluation.
  12. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.
  13. Applying deep matching networks to chinese medical question answering: A study and a dataset. BMC Medical Informatics and Decision Making, 19(2):52.
  14. Measuring massive multitask language understanding. In International Conference on Learning Representations.
  15. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Processing Systems.
  16. Financebench: A new benchmark for financial question answering.
  17. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. arXiv preprint arXiv:2009.13081.
  18. Psyeval: A comprehensive large language model evaluation benchmark for mental health. ArXiv, abs/2311.09189.
  19. Psy-llm: Scaling up global mental health psychological services with ai-based large language models.
  20. Towards medical machine reading comprehension with structural knowledge and plain text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1427–1438, Online. Association for Computational Linguistics.
  21. Cmmlu: Measuring massive multitask language understanding in chinese.
  22. Holistic evaluation of language models. Transactions on Machine Learning Research. Featured Certification, Expert Certification.
  23. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
  24. M3ke: A massive multi-level multi-subject knowledge evaluation benchmark for chinese large language models.
  25. Towards emotional support dialog systems. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3469–3483, Online. Association for Computational Linguistics.
  26. PAIR: Prompt-aware margIn ranking for counselor reflection scoring in motivational interviewing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 148–158, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  27. Empathic conversations: A multi-level dataset of contextualized conversations.
  28. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 248–260. PMLR.
  29. Smile: Single-turn to multi-turn inclusive language expansion via chatgpt for mental health support.
  30. Improving language understanding by generative pre-training.
  31. Language models are unsupervised multitask learners.
  32. Towards empathetic open-domain conversation models: A new benchmark and dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5370–5381, Florence, Italy. Association for Computational Linguistics.
  33. A computational approach to understanding empathy expressed in text-based mental health support. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5263–5276, Online. Association for Computational Linguistics.
  34. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research.
  35. PsyQA: A Chinese dataset for generating long counseling text for mental health support. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1489–1503, Online. Association for Computational Linguistics.
  36. WASSA 2021 shared task: Predicting empathy and emotion in reaction to news stories. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 92–104, Online. Association for Computational Linguistics.
  37. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. Curran Associates Inc., Red Hook, NY, USA.
  38. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
  39. Cmb: A comprehensive medical benchmark in chinese. arXiv preprint arXiv:2308.08833.
  40. A large-scale dataset for empathetic response generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1251–1264, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  41. Pixiu: A large language model, instruction data and evaluation benchmark for finance.
  42. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  43. Hui Zeng. 2023. Measuring massive multitask chinese understanding.
  44. Psybench: a balanced and in-depth psychological chinese evaluation benchmark for foundation models.
  45. Fineval: A chinese financial domain knowledge evaluation benchmark for large language models.
  46. AugESC: Dialogue augmentation with large language models for emotional support conversation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1552–1568, Toronto, Canada. Association for Computational Linguistics.
  47. Agieval: A human-centric benchmark for evaluating foundation models.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free