Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
102 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

ChatGPT Role-play Dataset: Analysis of User Motives and Model Naturalness (2403.18121v1)

Published 26 Mar 2024 in cs.CL and cs.HC

Abstract: Recent advances in interactive LLMs like ChatGPT have revolutionized various domains; however, their behavior in natural and role-play conversation settings remains underexplored. In our study, we address this gap by deeply investigating how ChatGPT behaves during conversations in different settings by analyzing its interactions in both a normal way and a role-play setting. We introduce a novel dataset of broad range of human-AI conversations annotated with user motives and model naturalness to examine (i) how humans engage with the conversational AI model, and (ii) how natural are AI model responses. Our study highlights the diversity of user motives when interacting with ChatGPT and variable AI naturalness, showing not only the nuanced dynamics of natural conversations between humans and AI, but also providing new avenues for improving the effectiveness of human-AI communication.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. Leveraging transitions of emotions for sarcasm detection. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1505–1508.
  2. Amos Azaria. 2022. Chatgpt usage and limitations.
  3. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 675–718, Nusa Dua, Bali. Association for Computational Linguistics.
  4. Penelope Brown and Stephen C Levinson. 1987. Politeness: Some universals in language usage, volume 4. Cambridge university press.
  5. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Do conversational agents have a theory of mind? a single case study of chatgpt with the hinting, false beliefs and false photographs, and strange stories paradigms.
  7. Does chatgpt resemble humans in language use? arXiv preprint arXiv:2303.08014.
  8. Assessing cross-cultural alignment between ChatGPT and human societies: An empirical study. In Proceedings of the First Workshop on Cross-Cultural Considerations in NLP (C3NLP), pages 53–67, Dubrovnik, Croatia. Association for Computational Linguistics.
  9. Collaborative effort towards common ground in situated human-robot dialogue. 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI), pages 33–40.
  10. Chatgpt goes to law school. Available at SSRN.
  11. Noam chomsky: The false promise of chatgpt. The New York Times.
  12. H.H. Clark. 1996. Using Language. ACLS Humanities E-Book. Cambridge University Press.
  13. François Cooren. 2018. Edda weigand (ed.). 2017. the routledge handbook of language and dialogue. Language and Dialogue, 8:468–482.
  14. Common ground, cooperation, and recipient design in human-computer interactions. Journal of Pragmatics, 193:4–20.
  15. Gunther Eysenbach et al. 2023. The role of chatgpt, generative language models, and artificial intelligence in medical education: a conversation with chatgpt and a call for papers. JMIR Medical Education, 9(1):e46885.
  16. MEEP: Is this engaging? prompting large language models for dialogue evaluation in multilingual settings. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2078–2100, Singapore. Association for Computational Linguistics.
  17. Kerstin Fischer. 2016. Designing Speech for a Recipient: The roles of partner modeling, alignment and feedback in so-called ’simplified registers’.
  18. Kerstin Fischer. 2017. The Situatedness of Pragmatic Acts: Explaining a Lamp to a Robot, Perspectives in Pragmatics, Psychology & Philosophy, pages 901–910. Springer, Germany.
  19. Mindful tutors: Linguistic choice and action demonstration in speech to infants and a simulated robot. Interaction Studies, 12:134–161.
  20. The match corpus: a corpus of older and younger users’ interactions with spoken dialogue systems. Language Resources and Evaluation, 44:221–261.
  21. Accommodation theory: Communication, context, and consequence, Studies in Emotion and Social Interaction, page 1–68. Cambridge University Press.
  22. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill.
  23. Paul Grice. 1989. Studies in the Way of Words. Harvard University Press.
  24. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597.
  25. Regulating chatgpt and other large generative ai models. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’23, page 1112–1123, New York, NY, USA. Association for Computing Machinery.
  26. Johan F Hoorn and Juliet J-Y Chen. 2023. Epistemic considerations when ai answers questions for us. arXiv preprint arXiv:2304.14352.
  27. William S. Horton. 2005. Conversational common ground and memory processes in language production. Discourse Processes, 40(1):1–35.
  28. Distinguishing human generated text from chatgpt generated text using machine learning. arXiv preprint arXiv:2306.01761.
  29. Chatgpt makes medicine easy to swallow: An exploratory case study on simplified radiology reports. arXiv preprint arXiv:2212.14882.
  30. Analysis of humanoid appearances in human-robot interaction. IEEE Transactions on Robotics, 24:725–735.
  31. Ardavan Kasirzadeh and Iason Gabriel. 2023. In conversation with artificial intelligence: aligning language models with human values. Philosophy & Technology, 36(2):1–24.
  32. Activating, seeking, and creating common ground: A socio-cognitive approach. Pragmatics & Cognition, 172:331–355.
  33. "is the pope catholic?" applying chain-of-thought reasoning to understanding conversational implicatures. ArXiv, abs/2305.13826.
  34. Better zero-shot reasoning with role-play prompting.
  35. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13171–13189, Singapore. Association for Computational Linguistics.
  36. Chatgpt: A meta-analysis after 2.5 months. arXiv preprint arXiv:2302.13795.
  37. What makes pre-trained language models better zero-shot learners? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2288–2303, Toronto, Canada. Association for Computational Linguistics.
  38. Large language models are superpositions of all characters: Attaining arbitrary role-play via self-alignment.
  39. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint arXiv:2301.06627.
  40. Developing chatgpt’s theory of mind. Frontiers in Robotics and AI, 10.
  41. Chatgpt or human? detect and explain. explaining decisions of machine learning model for detecting short chatgpt-generated text. arXiv preprint arXiv:2301.13852.
  42. The media inequality: Comparing the initial human-human and human-ai social interactions.
  43. Clifford Nass and Youngme Moon. 2000. Machines and mindlessness: Social responses to computers. Journal of Social Issues, 56:81–103.
  44. OpenAI. 2023. Teaching with AI. https://openai.com/blog/teaching-with-ai. [Online; accessed 17-September-2023].
  45. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
  46. To chatgpt, or not to chatgpt: That is the question! arXiv preprint arXiv:2304.01487.
  47. Steven Piantadosi. 2023. Modern language models refute chomsky’s approach to language. Lingbuzz Preprint, lingbuzz, 7180.
  48. Is ChatGPT a general-purpose natural language processing task solver? In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1339–1384, Singapore. Association for Computational Linguistics.
  49. Pragmatic implicature processing in chatgpt.
  50. Improving language understanding by generative pre-training.
  51. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  52. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by llms.
  53. John R Searle. 1980. Minds, brains, and programs. Behavioral and Brain Sciences, 3(3):417–457.
  54. Sakib Shahriar and Kadhim Hayawi. 2023. Let’s have a chat! a conversation with chatgpt: Technology, applications, and limitations. arXiv preprint arXiv:2302.13817.
  55. Role-play with large language models.
  56. Teo Susnjak. 2022. Chatgpt: The end of online exam integrity? arXiv preprint arXiv:2212.09292.
  57. Spoken dialogue systems and chatgpt for second language pragmatics research. In K. Sadeghi, editor, The Routledge handbook of technological advances and considerations in second language/applied linguistics research. Routledge. In press.
  58. Wilbert Tabone and Joost De Winter. 2023. Using chatgpt for human–computer interaction research: A primer. Manuscript submitted for publication.
  59. H Holden Thorp. 2023. Chatgpt is fun, but not an author. Science, 379(6630):313–313.
  60. Chatgpt: five priorities for research. Nature, 614(7947):224–226.
  61. On the robustness of chatgpt: An adversarial and out-of-distribution perspective. arXiv preprint arXiv:2302.12095.
  62. Joseph Weizenbaum. 1966. Eliza—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45.
  63. Reducing working memory load in spoken dialogue systems. Interact. Comput., 21:276–287.
  64. Exploring ai ethics of chatgpt: A diagnostic analysis. arXiv preprint arXiv:2301.12867.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Yufei Tao (16 papers)
  2. Ameeta Agrawal (23 papers)
  3. Judit Dombi (1 paper)
  4. Tetyana Sydorenko (1 paper)
  5. Jung In Lee (2 papers)
Citations (4)
View on Semantic Scholar
Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube