Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
51 tokens/sec
GPT-4o
60 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
8 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Can Large Language Model Comprehend Ancient Chinese? A Preliminary Test on ACLUE (2310.09550v1)

Published 14 Oct 2023 in cs.CL

Abstract: LLMs have showcased remarkable capabilities in understanding and generating language. However, their ability in comprehending ancient languages, particularly ancient Chinese, remains largely unexplored. To bridge this gap, we present ACLUE, an evaluation benchmark designed to assess the capability of LLMs in comprehending ancient Chinese. ACLUE consists of 15 tasks cover a range of skills, spanning phonetic, lexical, syntactic, semantic, inference and knowledge. Through the evaluation of eight state-of-the-art LLMs, we observed a noticeable disparity in their performance between modern Chinese and ancient Chinese. Among the assessed models, ChatGLM2 demonstrates the most remarkable performance, achieving an average score of 37.4%. We have made our code and data public available.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Falcon-40B: an open large language model with state-of-the-art performance.
  2. Automated reconstruction of ancient languages using probabilistic models of sound change. Proc. Natl. Acad. Sci. USA, 110(11):4224–4229.
  3. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
  4. Think you have solved question answering? try arc, the AI2 reasoning challenge. CoRR, abs/1803.05457.
  5. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335.
  6. Jiuge: A human-machine collaborative chinese classical poetry generation system. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28 - August 2, 2019, Volume 3: System Demonstrations, pages 25–30. Association for Computational Linguistics.
  7. Sentence segmentation for classical chinese based on LSTM with radical embedding. CoRR, abs/1810.03479.
  8. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  9. Knowledge representation and sentence segmentation of ancient chinese based on deep language models. Journal of Chinese Information Processing, 35(4):8–15.
  10. Cmmlu: Measuring massive multitask language understanding in chinese.
  11. Ancient-modern chinese translation with a new large training dataset. ACM Trans. Asian Low Resour. Lang. Inf. Process., 19(1):6:1–6:13.
  12. Yu sheng: Human-in-loop classical chinese poetry generation system. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. EACL 2023 - System Demonstrations, Dubrovnik, Croatia, May 2-4, 2023, pages 57–66. Association for Computational Linguistics.
  13. Crosslingual generalization through multitask finetuning. CoRR, abs/2211.01786.
  14. OpenAI. 2023. Gpt-4 technical report.
  15. OpenLMLab. 2023. Moss.
  16. The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116.
  17. Coupgan: Chinese couplet generation via encoder-decoder model and adversarial training under global control. Soft Comput., 26(15):7423–7433.
  18. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100.
  19. A sentiment and style controllable approach for chinese poetry generation. In CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 4784–4788. ACM.
  20. 古汉语词义标注语料库的构建及应用研究(the construction and application of Ancient Chinese corpus with word sense annotation). In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 549–563, Huhhot, China. Chinese Information Processing Society of China.
  21. Anchibert: A pre-trained model for ancient chinese language understanding and generation. In International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021, pages 1–8. IEEE.
  22. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  23. Roger D Woodard. 2008. The ancient languages of Europe. Cambridge University Press.
  24. From shakespeare to li-bai: Adapting a sonnet model to chinese poetry. In Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association, ALTA 2019, Sydney, Australia, December 4-6, 2019, pages 10–18. Australasian Language Technology Association.
  25. Chinese couplet generation with neural network structures. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
  26. Stylistic chinese poetry generation via unsupervised style disentanglement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 3960–3969. Association for Computational Linguistics.
  27. Generating chinese classical poems with RNN encoder-decoder. In Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data - 16th China National Conference, CCL 2017, - and - 5th International Symposium, NLP-NABD 2017, Nanjing, China, October 13-15, 2017, Proceedings, volume 10565 of Lecture Notes in Computer Science, pages 211–223. Springer.
  28. Automatic poetry generation with mutual reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3143–3153, Brussels, Belgium.
  29. Automatic generation of chinese couplets with attention based encoder-decoder model. In 2nd IEEE Conference on Multimedia Information Processing and Retrieval, MIPR 2019, San Jose, CA, USA, March 28-30, 2019, pages 65–70. IEEE.
  30. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations.
  31. Poetrybert: Pre-training with sememe knowledge for classical chinese poetry. In Data Mining and Big Data - 7th International Conference, DMBD 2022, Beijing, China, November 21-24, 2022, Proceedings, Part II, volume 1745 of Communications in Computer and Information Science, pages 369–384. Springer.
  32. Bao Zhiming. 1990. Language and world view in ancient china. Philosophy East and West, 40(2):195–219.
  33. Agieval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (2)
  1. Yixuan Zhang (94 papers)
  2. Haonan Li (43 papers)
Citations (6)
View on Semantic Scholar