Blockwise Parallel Decoding for Deep Autoregressive Models (1811.03115v1)

Abstract: Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the length of the critical path at training time, generation still remains an inherently sequential process. To overcome this limitation, we propose a novel blockwise parallel decoding scheme in which we make predictions for multiple time steps in parallel then back off to the longest prefix validated by a scoring model. This allows for substantial theoretical improvements in generation speed when applied to architectures that can process output sequences in parallel. We verify our approach empirically through a series of experiments using state-of-the-art self-attention models for machine translation and image super-resolution, achieving iteration reductions of up to 2x over a baseline greedy decoder with no loss in quality, or up to 7x in exchange for a slight decrease in performance. In terms of wall-clock time, our fastest models exhibit real-time speedups of up to 4x over standard greedy decoding.

Blockwise Parallel Decoding for Deep Autoregressive Models

The paper, authored by Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit, introduces a novel algorithm for enhancing the decoding speed of deep autoregressive models by utilizing a blockwise parallel decoding mechanism. This work addresses a persistent challenge in autoregressive sequence generation: the inherently sequential nature of decoding, which limits efficiency on parallel hardware. The authors propose a method to exploit parallelism during decoding, thereby achieving substantial improvements in speed without compromising quality.

Core Methodology

The crux of the research is the implementation of a blockwise parallel decoding algorithm. This method leverages auxiliary models trained to propose multiple future predictions in parallel, which are then verified against a base model to determine the longest accepted prefix. This unique block structure allows significant reductions in the number of sequential steps required during decoding. The paper demonstrates that this technique can be seamlessly integrated into existing architectures like Transformers, which are well-suited for parallel computations due to their structure, thus enhancing inference efficiency significantly.

Experimental Verification

The efficacy of the proposed decoding strategy was validated through extensive experiments on two tasks: machine translation and image super-resolution. For machine translation, using the WMT 2014 English-German dataset, the authors achieved up to a 7x reduction in decoding iterations with slight quality sacrifice, while real-time speedup topped at 4x compared to standard greedy decoding. Notably, the technique required minimal architectural changes and was tested using the Tensor2Tensor library to ensure replicability and ease of integration.

In the domain of image super-resolution, the technique achieved similar iteration reductions with CelebA dataset, reflecting potential applicability across various sequence-to-sequence tasks beyond natural language processing. Human evaluations complemented quantitative analyses, revealing a subjective preference shift toward outputs generated by models using the new decoding strategy, albeit with minor perceptual quality improvement.

Implications and Future Directions

This work has significant implications for practical applications where generation speed is critical, such as real-time translation or interactive AI systems. By efficiently harnessing parallelism, the proposed method presents a transformative approach to deploying autoregressive models in production environments where latency is a crucial factor.

Theoretically, this approach opens avenues for further research into hybrid decoding strategies that combine parallel and sequential processing to optimize both speed and quality. Additionally, exploring integration with other model acceleration techniques, such as non-autoregressive or latent variable-based models, could potentially yield synergistic benefits.

In conclusion, the authors present a compelling and practical advancement in the field of sequence-to-sequence modeling. The blockwise parallel decoding strategy is a significant contribution to the field, aligning model capabilities with the computational strengths of modern hardware. Future research will likely build upon this framework, exploring its full potential across diverse applications and model architectures.