2000 character limit reached
Are LLMs Effective Negotiators? Systematic Evaluation of the Multifaceted Capabilities of LLMs in Negotiation Dialogues (2402.13550v2)
Published 21 Feb 2024 in cs.CL and cs.AI
Abstract: A successful negotiation requires a range of capabilities, including comprehension of the conversation context, Theory-of-Mind (ToM) skills to infer the partner's motives, strategic reasoning, and effective communication, making it challenging for automated systems. Despite the remarkable performance of LLMs in various NLP tasks, there is no systematic evaluation of their capabilities in negotiation. Such an evaluation is critical for advancing AI negotiation agents and negotiation research, ranging from designing dialogue systems to providing pedagogical feedback and scaling up data collection practices. This work aims to systematically analyze the multifaceted capabilities of LLMs across diverse dialogue scenarios throughout the stages of a typical negotiation interaction. Our analysis highlights GPT-4's superior performance in many tasks while identifying specific challenges, such as making subjective assessments and generating contextually appropriate, strategically advantageous responses.
- Llm-deliberation: Evaluating llms with interactive multi-agent negotiation games.
- On the cultural basis of gender differences in negotiation. Experimental Economics, 21:757–778.
- Language models are few-shot learners.
- Sparks of artificial general intelligence: Early experiments with gpt-4.
- Towards emotion-aware agents for negotiation dialogues. In 2021 9th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–8. IEEE.
- Towards emotion-aware agents for improved user satisfaction and partner perception in negotiation dialogues. IEEE Transactions on Affective Computing.
- Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems.
- CaSiNo: A corpus of campsite negotiation dialogues for automatic negotiation systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3167–3185, Online. Association for Computational Linguistics.
- Social influence dialogue systems: A survey of datasets and models for social influence tasks.
- Social influence dialogue systems: A survey of datasets and models for social influence tasks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 750–766, Dubrovnik, Croatia. Association for Computational Linguistics.
- Be selfish, but wisely: Investigating the impact of agent personality in mixed-motive human-agent interactions. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13078–13092.
- Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
- Scaling instruction-finetuned language models.
- Toward natural turn-taking in a virtual human negotiation agent. In 2015 AAAI Spring Symposium Series.
- Chaim Fershtman. 1990. The importance of the agenda in bargaining. Games and Economic Behavior, 2(3):224–238.
- Chris Frith and Uta Frith. 2005. Theory of mind. Current biology, 15(17):R644–R645.
- Improving language model negotiation with self-play and in-context learning from ai feedback.
- Strategic reasoning with language models.
- Reasoning with language model is planning with world model.
- Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333–2343.
- Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey.
- Mistral 7b.
- Intelligent tutoring system for negotiation skills training. In International Conference on Artificial Intelligence in Education, pages 122–127. Springer.
- Large language models are zero-shot reasoners.
- Michal Kosinski. 2023. Theory of mind might have spontaneously emerged in large language models.
- Essentials of negotiation. McGraw-Hill New York.
- Deal or no deal? end-to-end learning for negotiation dialogues.
- Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization.
- Peng Luo. 2008. Analysis of cultural differences between west and east in international business negotiation. International Journal of Business and Management, 3(11):103–106.
- What makes chain-of-thought prompting effective? a counterfactual study. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1448–1535.
- Johnathan Mell and Jonathan Gratch. 2017. Grumpy & pinocchio: answering human-agent negotiation questions through realistic agent design. In Proceedings of the 16th conference on autonomous agents and multiagent systems, pages 401–409.
- The likeability-success tradeoff: Results of the 2 nd annual human-agent automated negotiating agents competition. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII), pages 1–7. IEEE.
- David Noever and Forrest McKee. 2023. Numeracy from literacy: Data science as an emergent skill from large language models.
- Outcome satisfaction in negotiation: A test of expectancy disconfirmation. Organizational Behavior and Human Decision Processes, 60(2):252–275.
- OpenAI. 2022. Openai: Introducing chatgpt.
- OpenAI. 2023. Openai: Gpt-4.
- Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks.
- A survey of the evolution of language model-based dialogue systems. arXiv preprint arXiv:2311.16789.
- Emergent abilities of large language models.
- Chain-of-thought prompting elicits reasoning in large language models.
- Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.
- Dialogue act-based breakdown detection in negotiation dialogues. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 745–757, Online. Association for Computational Linguistics.
- Self-discover: Large language models self-compose reasoning structures. arXiv preprint arXiv:2402.03620.
- Can large language models transform computational social science? arXiv preprint arXiv:2305.03514.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.