Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Enhancing Dialogue State Tracking Models through LLM-backed User-Agents Simulation (2405.13037v1)

Published 17 May 2024 in cs.CL and cs.AI

Abstract: Dialogue State Tracking (DST) is designed to monitor the evolving dialogue state in the conversations and plays a pivotal role in developing task-oriented dialogue systems. However, obtaining the annotated data for the DST task is usually a costly endeavor. In this paper, we focus on employing LLMs to generate dialogue data to reduce dialogue collection and annotation costs. Specifically, GPT-4 is used to simulate the user and agent interaction, generating thousands of dialogues annotated with DST labels. Then a two-stage fine-tuning on LLaMA 2 is performed on the generated data and the real data for the DST prediction. Experimental results on two public DST benchmarks show that with the generated dialogue data, our model performs better than the baseline trained solely on real data. In addition, our approach is also capable of adapting to the dynamic demands in real-world scenarios, generating dialogues in new domains swiftly. After replacing dialogue segments in any domain with the corresponding generated ones, the model achieves comparable performance to the model trained on real data.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (59)
  1. A slot-shared span prediction-based neural network for multi-domain dialogue state tracking. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE.
  2. Task-optimized adapters for an end-to-end task-oriented dialogue system. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7355–7369, Toronto, Canada. Association for Computational Linguistics.
  3. Björn Bebensee and Haejun Lee. 2023. Span-selective linear attention transformers for effective and robust schema-guided dialogue state tracking. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 78–91, Toronto, Canada. Association for Computational Linguistics.
  4. MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling. In Proc. of EMNLP.
  5. Schema-guided multi-domain dialogue state tracking with graph attention neural networks. In Proc. of AAAI.
  6. Unified language model pre-training for natural language understanding and generation. Advances in neural information processing systems, 32.
  7. MultiWOZ 2.1: A consolidated multi-domain dialogue dataset with state corrections and state tracking baselines. In Proc. of LREC.
  8. Towards llm-driven dialogue state tracking. In Proc. of EMNLP.
  9. From machine reading comprehension to dialogue state tracking: Bridging the gap. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI.
  10. Dialog state tracking: A neural reading comprehension approach. In Proc. of SIGDIAL.
  11. Multiwoz 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation. In Proc. of NLPCC.
  12. Unified dialog model pre-training for task-oriented dialog understanding and generation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 187–200.
  13. TripPy: A triple copy strategy for value independent neural dialog state tracking. In Proceedings of the 21th Annual Meeting of the Special Interest Group on Discourse and Dialogue.
  14. The second dialog state tracking challenge. In Proc. of SIGDIAL.
  15. Lora: Low-rank adaptation of large language models. In Proc. of ICLR.
  16. Vojtěch Hudeček and Ondřej Dušek. 2023. Are large language models all you need for task-oriented dialogue? In Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue.
  17. Is gpt-3 all you need for low-data discovery in chemistry?
  18. Synthetic data generation in low-resource settings via fine-tuning of large language models. ArXiv preprint.
  19. Brendan King and Jeffrey Flanigan. 2023. Diverse retrieval-augmented in-context learning for dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5570–5585.
  20. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
  21. Audio augmentation for speech recognition. In Proc. of Interspeech.
  22. Imagenet classification with deep convolutional neural networks. In Proc. of NeurIPS.
  23. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
  24. Dialogue state tracking with a language model using schema-driven prompting. In Proc. of EMNLP.
  25. Diable: Efficient dialogue state tracking as operations on tables. In Proc. of ACL Findings.
  26. Controllable dialogue simulation with in-context learning. In Proc. of EMNLP.
  27. MinTL: Minimalist transfer learning for task-oriented dialogue systems. In Proc. of EMNLP.
  28. Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations.
  29. Implicit discourse relation identification for open-domain dialogues. In Proc. of ACL.
  30. Neural belief tracker: Data-driven dialogue state tracking. In Proc. of ACL.
  31. OpenAI. 2023. Chatgpt. https://chat.openai.com.
  32. Specaugment: A simple data augmentation method for automatic speech recognition. In Proc. of INTERSPEECH.
  33. Soloist: Building task bots at scale with transfer learning and machine teaching. Transactions of the Association for Computational Linguistics.
  34. Towards universal dialogue state tracking. In Proc. of EMNLP.
  35. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. In Proc. of NeurIPS.
  36. Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data.
  37. Significant-gravitas. 2023. auto-gpt: An experimental open-source attempt to make gpt-4 fully autonomous. https://github.com/Significant-Gravitas/AutoGPT.
  38. Choice fusion as knowledge for zero-shot dialogue state tracking. In Proc. of ICASSP.
  39. On tracking dialogue state by inheriting slot values in mentioned slot pools. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 4375–4382. International Joint Conferences on Artificial Intelligence Organization. Main Track.
  40. Amendable generation for dialogue state tracking. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI.
  41. Llama: Open and efficient foundation language models. ArXiv preprint.
  42. Llama 2: Open foundation and fine-tuned chat models. ArXiv preprint.
  43. Bootstrapping llm-based task-oriented dialogue agents via self-talk. ArXiv preprint.
  44. Divide, conquer, and combine: Mixture of semantic-independent experts for zero-shot dialogue state tracking. In Proc. of ACL.
  45. LUNA: Learning slot-turn alignment for dialogue state tracking. In Proc. of NAACL.
  46. Jason Wei and Kai Zou. 2019. EDA: Easy data augmentation techniques for boosting performance on text classification tasks. In Proc. of EMNLP.
  47. Transferable multi-domain state generator for task-oriented dialogue systems. In Proc. of ACL.
  48. Refgpt: Dialogue generation of gpt, by gpt, and for gpt. In Proc. of EMNLP Findings.
  49. Multi-domain dialogue state tracking with disentangled domain-slot attention. In Proc. of ACL Findings.
  50. MultiWOZ 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue.
  51. Slot self-attentive dialogue state tracking. In Proc. of WWW.
  52. MultiWOZ 2.2 : A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI.
  53. MoNET: Tackle state momentum via noise-enhanced training for dialogue state tracking. In Findings of the Association for Computational Linguistics: ACL 2023, pages 520–534, Toronto, Canada. Association for Computational Linguistics.
  54. Find or classify? dual strategy for slot-value predictions on multi-domain dialog state tracking. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics.
  55. Character-level convolutional networks for text classification. In Proc. of NeurIPS.
  56. Description-driven task-oriented dialog modeling. ArXiv preprint.
  57. Pytorch fsdp: Experiences on scaling fully sharded data parallel. Proc. VLDB Endow.
  58. XQA-DST: Multi-domain and multi-lingual dialogue state tracking. In Proc. of ACL Findings.
  59. Continual prompt tuning for dialog state tracking. In Proc. of ACL.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Cheng Niu (15 papers)
  2. Xingguang Wang (2 papers)
  3. Xuxin Cheng (42 papers)
  4. Juntong Song (5 papers)
  5. Tong Zhang (569 papers)
Citations (6)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

Youtube Logo Streamline Icon: https://streamlinehq.com

YouTube