Measuring Bargaining Abilities of LLMs: A Benchmark and A Buyer-Enhancement Method (2402.15813v3)
Abstract: Bargaining is an important and unique part of negotiation between humans. As LLM-driven agents learn to negotiate and act like real humans, how to evaluate agents' bargaining abilities remains an open problem. For the first time, we formally described the Bargaining task as an asymmetric incomplete information game, defining the gains of the Buyer and Seller in multiple bargaining processes. It allows us to quantitatively assess an agent's performance in the Bargain task. We collected a real product price dataset, AmazonHistoryPrice, and conducted evaluations of various LLM agents' bargaining abilities. We find that playing a Buyer is much harder than a Seller, and increasing model size can not effectively improve the Buyer's performance. To address the challenge, we propose a novel approach called OG-Narrator that integrates a deterministic Offer Generator to control the price range of Buyer's offers, and an LLM Narrator to create natural language sentences for generated offers. Experimental results show that OG-Narrator improves the buyer's deal rates from 26.67% to 88.88% and brings a ten times multiplication of profits on all baselines, even a model that has not been aligned.
- Qwen technical report.
- Noncooperative models of bargaining. Handbook of game theory with economic applications, 1:179–225.
- The nash bargaining solution in economic modelling. The RAND Journal of Economics, pages 176–188.
- Put your money where your mouth is: Evaluating strategic planning and execution of llm agents in an auction arena.
- GLM: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335, Dublin, Ireland. Association for Computational Linguistics.
- Chaim Fershtman. 1990. The importance of the agenda in bargaining. Games and Economic Behavior, 2(3):224–238.
- Improving language model negotiation with self-play and in-context learning from ai feedback.
- Francisca Gayà Torres. 2021. Rubinstein’s bargaining model.
- Textbooks are all you need. ArXiv preprint, abs/2306.11644.
- Decoupling strategy and generation in negotiation dialogues. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2333–2343, Brussels, Belgium. Association for Computational Linguistics.
- Language of bargaining.
- Mistral 7b.
- Mixtral of experts.
- Large language models are zero-shot reasoners.
- Efficient memory management for large language model serving with pagedattention.
- Deal or no deal? end-to-end learning for negotiation dialogues.
- Amogh Mannekote. 2023. Towards a neural era in dialogue management for collaboration: A literature survey.
- OpenAI. 2023. Gpt-4 technical report.
- Training language models to follow instructions with human feedback.
- Generative agents: Interactive simulacra of human behavior.
- Train short, test long: Attention with linear biases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
- Ariel Rubinstein. 1982. Perfect equilibrium in a bargaining model. Econometrica: Journal of the Econometric Society, pages 97–109.
- Alfworld: Aligning text and embodied environments for interactive learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
- Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint, abs/2307.09288.
- Voyager: An open-ended embodied agent with large language models.
- Chain-of-thought prompting elicits reasoning in large language models.
- Baichuan 2: Open large-scale language models.
- Auto-gpt for online decision making: Benchmarks and additional opinions.
- Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.
- A dynamic strategy coach for effective negotiation. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 367–378, Stockholm, Sweden. Association for Computational Linguistics.