Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

How many words does ChatGPT know? The answer is ChatWords (2309.16777v1)

Published 28 Sep 2023 in cs.CL and cs.AI

Abstract: The introduction of ChatGPT has put AI NLP in the spotlight. ChatGPT adoption has been exponential with millions of users experimenting with it in a myriad of tasks and application domains with impressive results. However, ChatGPT has limitations and suffers hallucinations, for example producing answers that look plausible but they are completely wrong. Evaluating the performance of ChatGPT and similar AI tools is a complex issue that is being explored from different perspectives. In this work, we contribute to those efforts with ChatWords, an automated test system, to evaluate ChatGPT knowledge of an arbitrary set of words. ChatWords is designed to be extensible, easy to use, and adaptable to evaluate also other NLP AI tools. ChatWords is publicly available and its main goal is to facilitate research on the lexical knowledge of AI tools. The benefits of ChatWords are illustrated with two case studies: evaluating the knowledge that ChatGPT has of the Spanish lexicon (taken from the official dictionary of the "Real Academia Espa~nola") and of the words that appear in the Quixote, the well-known novel written by Miguel de Cervantes. The results show that ChatGPT is only able to recognize approximately 80% of the words in the dictionary and 90% of the words in the Quixote, in some cases with an incorrect meaning. The implications of the lexical knowledge of NLP AI tools and potential applications of ChatWords are also discussed providing directions for further work on the study of the lexical knowledge of AI tools.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909, 2023.
  2. Will affective computing emerge from foundation models and general artificial intelligence? a first evaluation of chatgpt. IEEE Intelligent Systems, 38(2):15–23, 2023.
  3. Partha Pratim Ray. Chatgpt: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope. Internet of Things and Cyber-Physical Systems, 3:121–154, 2023.
  4. A brief overview of chatgpt: The history, status quo and potential future development. IEEE/CAA Journal of Automatica Sinica, 10(5):1122–1136, 2023.
  5. Ali Borji. A categorical archive of chatgpt failures. arXiv preprint arXiv:2303.03494, 2023.
  6. Is ChatGPT capable of solving classical Communications and Networking problems? techrxiv preprint, 7 2023.
  7. ChatGPT: Jack of all trades, master of none. Information Fusion, 99:101861, nov 2023.
  8. In chatgpt we trust? measuring and characterizing the reliability of chatgpt. 2023.
  9. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, pages 7176–7185. PMLR, 2020.
  10. Combining generative artificial intelligence (ai) and the internet: Heading towards evolution or degradation? arXiv preprint arXiv:2303.01255, 2023.
  11. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305.17493, 2023.
  12. Towards understanding the interplay of generative artificial intelligence and the internet. arXiv preprint arXiv:2306.06130, 2023.
  13. Does chatgpt resemble humans in language use? arXiv preprint arXiv:2303.08014, 2023.
  14. Juan Manuel Toro. Emergence of a phonological bias in chatgpt. arXiv preprint arXiv:2305.15929, 2023.
  15. Contrasting linguistic patterns in human and llm-generated text. arXiv preprint arXiv:2308.09067, 2023.
  16. Playing with words: Comparing the vocabulary and lexical richness of chatgpt and humans. arXiv preprint arXiv:2308.07462, 2023.
  17. How many words do we know? practical estimates of vocabulary size dependent on word definition, the degree of language input and the participant’s age. Frontiers in Psychology, 7, 2016.
  18. How do spanish speakers read words? insights from a crowdsourced lexical decision megastudy. Behavior research methods, 52:1867–1882, 2020.
  19. Cervantes Miguel. Don Quijote De La Mancha - Edición IV Centenario. Alfaguara, 2015.
  20. David Singleton. Language and the lexicon: An introduction. Routledge, 2016.
  21. Harry C. Schnur. The greek thesaurus. The Classical World, 56(3):65–67, 1962.
  22. Maria Pantelia. Noûs, INTO CHAOS’: THE CREATION OF THE THESAURUS OF THE GREEK LANGUAGE. International Journal of Lexicography, 13(1):1–11, 03 2000.
  23. El banco de datos de la rae: Crea y corde. Per Abbat: boletín filológico de actualización académica y didáctica, ISSN 1886-5046, Nº. 2, 2007, pags. 137-148, 01 2007.
  24. The Corpus de référence du français contemporain (CRFC) as the First Genre-Diverse Mega-Corpus of French. International Journal of Lexicography, 30(1):63–84, 12 2015.
  25. Cristina Buenafuentes de la Mata. La décima edición del diccionario de la lengua castellana de la real academia española (1852): el aumento y la supresión de voces. 2019.
  26. Lynda Mugglestone. Dictionaries: a very short introduction. Oxford University Press, USA, 2011.
  27. Ludwig Wittgenstein. Tractatus logico-philosophicus. Routledge, London, 1990.
  28. George Kingsley Zipf. Selected Studies of the Principle of Relative Frequency in Language. Harvard University Press, 2013.
  29. Worldlex: Twitter and blog word frequencies for 66 languages. Behavior research methods, 48:963–972, 2016.
  30. Boolq: Exploring the surprising difficulty of natural yes/no questions, 2019.
  31. Perplexity—a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America, 62, 1977.
  32. Víctor García de la Concha. La Real Academia Española. Vida e historia. Espasa, Madrid, 2014.
  33. RAE. Diccionario de la lengua española. Espasa, Madrid, 2014.
  34. Is’Haaq Akbarian. Jiménez catalán, rosa maría (ed.). 2014. lexical availability in english and spanish as a second language. new york: Springer. xiv + 205 pages. isbn: 978-94-007-7157-4. International Journal of English Studies, 15:97, 12 2015.
  35. Abdu Mohammad Talib Al-kadi and Rashad Ali Ahmed. Evolution of english in the internet age. Indonesian Journal of Applied Linguistics, 7:727–736, 2018.
  36. 358,534 nonwords: The arc nonword database. The Quarterly Journal of Experimental Psychology Section A, 55(4):1339–1362, 2002.
  37. Wuggy: A multilingual pseudoword generator. Behavior research methods, 42:627–33, 08 2010.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Gonzalo Martínez (18 papers)
  2. Javier Conde (28 papers)
  3. Pedro Reviriego (36 papers)
  4. Elena Merino-Gómez (10 papers)
  5. José Alberto Hernández (25 papers)
  6. Fabrizio Lombardi (12 papers)
Citations (5)
View on Semantic Scholar