A Survey of Diffusion Models in Natural Language Processing (2305.14671v2)
Abstract: This survey paper provides a comprehensive review of the use of diffusion models in NLP. Diffusion models are a class of mathematical models that aim to capture the diffusion of information or signals across a network or manifold. In NLP, diffusion models have been used in a variety of applications, such as natural language generation, sentiment analysis, topic modeling, and machine translation. This paper discusses the different formulations of diffusion models used in NLP, their strengths and limitations, and their applications. We also perform a thorough comparison between diffusion models and alternative generative models, specifically highlighting the autoregressive (AR) models, while also examining how diverse architectures incorporate the Transformer in conjunction with diffusion models. Compared to AR models, diffusion models have significant advantages for parallel generation, text interpolation, token-level controls such as syntactic structures and semantic contents, and robustness. Exploring further permutations of integrating Transformers into diffusion models would be a valuable pursuit. Also, the development of multimodal diffusion models and large-scale diffusion LLMs with notable capabilities for few-shot learning would be important directions for the future advance of diffusion models in NLP.
- Structured denoising diffusion models in discrete state-spaces.
- Structured denoising diffusion models in discrete state-spaces. In Advances in Neural Information Processing Systems.
- Structured denoising diffusion models in discrete state-spaces. ArXiv, abs/2107.03006.
- A neural probabilistic language model. In Journal of machine learning research.
- Language models are few-shot learners.
- A cheaper and better diffusion language model with soft-masked noise.
- Analog bits: Generating discrete data using diffusion models with self-conditioning. In International Conference on Learning Representations.
- Bert: Pre-training of deep bidirectional transformers for language understanding.
- Prafulla Dhariwal and Alex Nichol. 2021. Diffusion models beat gans on image synthesis.
- Continuous diffusion for categorical data.
- Difformer: Empowering diffusion models on the embedding space for text generation.
- Diffuseq: Sequence to sequence text generation with diffusion models.
- Generative adversarial networks.
- Ssd-lm: Semi-autoregressive simplex-based diffusion language model for text generation and modular control.
- Diffusionbert: Improving generative masked language models with diffusion models.
- Denoising diffusion probabilistic models.
- Training compute-optimal large language models. ArXiv, abs/2203.15556.
- Argmax flows and multinomial diffusion: Learning categorical distributions.
- On density estimation with diffusion models. In Advances in Neural Information Processing Systems.
- Diederik P Kingma and Max Welling. 2022. Auto-encoding variational bayes.
- Durk P Kingma and Prafulla Dhariwal. 2018. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31.
- Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
- Diffusion-lm improves controllable text generation.
- Text generation with diffusion language models: A pre-training approach with continuous paragraph denoise.
- Latent diffusion for language generation.
- Eliya Nachmani and Shaked Dovrat. 2021. Zero-shot translation using diffusion models.
- A systematic characterization of sampling algorithms for open-ended language generation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 334–346, Suzhou, China. Association for Computational Linguistics.
- Alex Nichol and Prafulla Dhariwal. 2021. Improved denoising diffusion probabilistic models.
- Masked autoregressive flow for density estimation. Advances in neural information processing systems, 30.
- Mauve: Measuring the gap between neural text and human text using divergence frontiers.
- Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872–1897.
- Exploring the limits of transfer learning with a unified text-to-text transformer.
- DiffusER: Diffusion via edit-based reconstruction. In International Conference on Learning Representations.
- Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages 1278–1286, Bejing, China. PMLR.
- High-resolution image synthesis with latent diffusion models.
- Step-unrolled denoising autoencoders for text generation.
- Deep unsupervised learning using nonequilibrium thermodynamics. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256–2265, Lille, France. PMLR.
- Deep unsupervised learning using nonequilibrium thermodynamics.
- Denoising diffusion implicit models.
- Self-conditioned embedding diffusion for text generation.
- Generating text with recurrent neural networks. In International Conference on Machine Learning.
- Dinoiser: Diffused conditional sequence learning by manipulating noises.
- Seqdiffuseq: Text diffusion with encoder-decoder transformers.
- Diffusum: Generation enhanced extractive summarization with diffusion.
- Trading off diversity and quality in natural language generation. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 25–33, Online. Association for Computational Linguistics.
- A reparameterized discrete diffusion model for text generation.