Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DiffusionDialog: A Diffusion Model for Diverse Dialog Generation with Latent Space (2404.06760v1)

Published 10 Apr 2024 in cs.CL and cs.AI

Abstract: In real-life conversations, the content is diverse, and there exists the one-to-many problem that requires diverse generation. Previous studies attempted to introduce discrete or Gaussian-based continuous latent variables to address the one-to-many problem, but the diversity is limited. Recently, diffusion models have made breakthroughs in computer vision, and some attempts have been made in natural language processing. In this paper, we propose DiffusionDialog, a novel approach to enhance the diversity of dialogue generation with the help of diffusion model. In our approach, we introduce continuous latent variables into the diffusion model. The problem of using latent variables in the dialog task is how to build both an effective prior of the latent space and an inferring process to obtain the proper latent given the context. By combining the encoder and latent-based diffusion model, we encode the response's latent representation in a continuous space as the prior, instead of fixed Gaussian distribution or simply discrete ones. We then infer the latent by denoising step by step with the diffusion model. The experimental results show that our model greatly enhances the diversity of dialog responses while maintaining coherence. Furthermore, in further analysis, we find that our diffusion model achieves high inference efficiency, which is the main challenge of applying diffusion models in natural language processing.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (37)
  1. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558–7567.
  2. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981–17993.
  3. Plato: Pre-trained dialogue generation model with discrete latent variable. arXiv preprint arXiv:1910.07931.
  4. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint arXiv:2208.04202.
  5. Dialogved: A pre-trained latent variable encoder-decoder model for dialog response generation. arXiv preprint arXiv:2204.13031.
  6. Prafulla Dhariwal and Alexander Nichol. 2021. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 34:8780–8794.
  7. Implicit deep latent variable models for text generation. arXiv preprint arXiv:1908.11527.
  8. Large-scale transfer learning for natural language generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6053–6058.
  9. Diffuseq: Sequence to sequence text generation with diffusion models. arXiv preprint arXiv:2210.08933.
  10. Dialogwae: Multimodal response generation with conditional wasserstein auto-encoder. arXiv preprint arXiv:1805.12352.
  11. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840–6851.
  12. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv:2009.09761.
  13. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.
  14. Optimus: Organizing sentences via pre-trained modeling of a latent space. arXiv preprint arXiv:2004.04092.
  15. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055.
  16. Diffusion-lm improves controllable text generation. Advances in Neural Information Processing Systems, 35:4328–4343.
  17. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957.
  18. Composable text controls in latent space with odes. arXiv preprint arXiv:2208.00638.
  19. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101.
  20. Latent diffusion for language generation. arXiv preprint arXiv:2212.09462.
  21. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  22. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
  23. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551.
  24. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125.
  25. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695.
  26. Cmu sinbad’s submission for the dstc7 avsd challenge. In DSTC7 at AAAI2019 workshop, volume 6.
  27. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909.
  28. A hierarchical latent variable encoder-decoder model for generating dialogues. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
  29. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502.
  30. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456.
  31. Generating relevant and coherent dialogue responses using self-separated conditional variational autoencoders. arXiv preprint arXiv:2106.03410.
  32. Attention is all you need. Advances in neural information processing systems, 30.
  33. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
  34. Seqdiffuseq: Text diffusion with encoder-decoder transformers. arXiv preprint arXiv:2212.10325.
  35. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243.
  36. Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960.
  37. The design and implementation of xiaoice, an empathetic social chatbot. Computational Linguistics, 46(1):53–93.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (6)
  1. Jianxiang Xiang (2 papers)
  2. Zhenhua Liu (47 papers)
  3. Haodong Liu (11 papers)
  4. Yin Bai (2 papers)
  5. Jia Cheng (20 papers)
  6. Wenliang Chen (33 papers)