Latent Diffusion for Language Generation (2212.09462v2)

Abstract: Diffusion models have achieved great success in modeling continuous data modalities such as images, audio, and video, but have seen limited use in discrete domains such as language. Recent attempts to adapt diffusion to language have presented diffusion as an alternative to existing pretrained LLMs. We view diffusion and existing LLMs as complementary. We demonstrate that encoder-decoder LLMs can be utilized to efficiently learn high-quality language autoencoders. We then demonstrate that continuous diffusion models can be learned in the latent space of the language autoencoder, enabling us to sample continuous latent representations that can be decoded into natural language with the pretrained decoder. We validate the effectiveness of our approach for unconditional, class-conditional, and sequence-to-sequence language generation. We demonstrate across multiple diverse data sets that our latent language diffusion models are significantly more effective than previous diffusion LLMs.

The paper "Latent Diffusion for Language Generation" explores the adaptation of diffusion models, which have proven highly effective in continuous data modalities like images and audio, to the discrete domain of language. Traditionally, diffusion models have seen limited application in generating text, but this work aims to address that gap by presenting a framework where diffusion processes and pretrained LLMs are viewed as complementary rather than competing approaches.

The authors propose integrating encoder-decoder LLMs to develop high-quality language autoencoders. This integration allows continuous diffusion models to operate in the latent space of these autoencoders. Specifically, the model learns continuous latent representations through a diffusion process, which can then be decoded into human-readable text by the pretrained decoder.

The approach is validated across several types of language generation tasks, including:

• Unconditional Language Generation: Generating text without any specific input prompt or constraints.
• Class-Conditional Language Generation: Generating text conditioned on class labels.
• Sequence-to-Sequence Language Generation: Generating a sequence of text based on an input sequence, such as in translation tasks.

The results obtained from experiments on multiple diverse datasets indicate that the proposed latent language diffusion models significantly outperform previous diffusion-based LLMs. The success of the method shows its potential in improving the quality of language generation across various applications by leveraging the strengths of both diffusion models and pretrained LLMs.