AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation (2305.09515v3)

Abstract: Diffusion models have gained significant attention in the realm of image generation due to their exceptional performance. Their success has been recently expanded to text generation via generating all tokens within a sequence concurrently. However, natural language exhibits a far more pronounced sequential dependency in comparison to images, and the majority of existing LLMs are trained with a left-to-right auto-regressive approach. To account for the inherent sequential characteristic of natural language, we introduce Auto-Regressive Diffusion (AR-Diffusion). AR-Diffusion ensures that the generation of tokens on the right depends on the generated ones on the left, a mechanism achieved through employing a dynamic number of denoising steps that vary based on token position. This results in tokens on the left undergoing fewer denoising steps than those on the right, thereby enabling them to generate earlier and subsequently influence the generation of tokens on the right. In a series of experiments on various text generation tasks, including text summarization, machine translation, and common sense generation, AR-Diffusion clearly demonstrated its superiority over existing diffusion LLMs and that it can be $100\times\sim600\times$ faster when achieving comparable results. Our code is available at https://github.com/microsoft/ProphetNet/tree/master/AR-diffusion.

• The paper introduces AR-Diffusion, a novel model that fuses diffusion and auto-regressive approaches to enhance text generation by preserving sequential dependencies.
• The paper employs a multi-level diffusion strategy with dynamic timestep adjustments and a skipping mechanism to accelerate decoding up to 600× faster without sacrificing quality.
• The paper demonstrates through benchmarks that integrating noise-to-embedding dynamics improves both speed and naturalness in text generation, paving the way for further multimodal advancements.

AR-Diffusion: Auto-Regressive Diffusion Model for Text Generation

The paper presents AR-Diffusion, a novel approach integrating diffusion models within an auto-regressive (AR) framework for text generation tasks such as text summarization, machine translation, and common sense generation. Diffusion models, traditionally successful in image generation, have gained attention in text generation due to their parallel generation capabilities. However, existing models often lack the sequential dependence characteristic inherent in natural language, typically handled using left-to-right AR models. AR-Diffusion is designed to address this deficiency by incorporating sequential token dependencies into the diffusion process.

Methodology and Contributions

AR-Diffusion innovatively combines AR and diffusion methodologies by introducing a multi-level diffusion strategy. This approach involves both sentence-level and token-level diffusion, adjusting the number of denoising steps dynamically based on token position. This enables tokens on the left, which are generated earlier, to influence subsequent token generations on the right. The model employs a dynamic timestep function to ensure tokens at the end of a sentence benefit from the information gained from earlier tokens.

A critical component of this process is AR-Diffusion's use of a dynamic movement speed principle which dictates that tokens on the left move faster from a state of Gaussian noise to their target embedding compared to tokens on the right. This ability to adjust the diffusion process based on token position significantly improves the efficiency and naturalness of text generation.

Moreover, to counteract the traditionally slow diffusion model inference times, AR-Diffusion incorporates a skipping mechanism to accelerate decoding by selectively traversing certain timesteps.

Experimental Results

Experiments across several benchmarks demonstrate that AR-Diffusion not only surpasses existing diffusion models in terms of speed and quality but also matches or exceeds AR models. It achieves results $100\times\sim600\times$ faster without sacrificing performance, producing high-quality outputs even with a small number of inference steps.

Implications and Future Work

The implications of AR-Diffusion are profound, particularly in its ability to balance the intricate sequential dependencies of language generation with the computational efficiency akin to NAR methods. The model introduces possibilities for further exploration within AI, particularly regarding its application to other sequence-to-sequence tasks or integrating more sophisticated skipping mechanisms.

For future developments, the research community could explore optimizing sampling strategies to minimize candidate generation without quality reduction, or extending AR-Diffusion to multimodal tasks potentially combining text with other data types like images or audio.

In conclusion, AR-Diffusion serves as a notable contribution to the field, enhancing the capabilities of diffusion models in text generation by elegantly incorporating the strengths of AR methodologies. This approach opens pathways for further exploration and refinement within the landscape of natural language processing.