SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models (2411.05007v3)

Abstract: Diffusion models can effectively generate high-quality images. However, as they scale, rising memory demands and higher latency pose substantial deployment challenges. In this work, we aim to accelerate diffusion models by quantizing their weights and activations to 4 bits. At such an aggressive level, both weights and activations are highly sensitive, where existing post-training quantization methods like smoothing become insufficient. To overcome this limitation, we propose SVDQuant, a new 4-bit quantization paradigm. Different from smoothing, which redistributes outliers between weights and activations, our approach absorbs these outliers using a low-rank branch. We first consolidate the outliers by shifting them from activations to weights. Then, we use a high-precision, low-rank branch to take in the weight outliers with Singular Value Decomposition (SVD), while a low-bit quantized branch handles the residuals. This process eases the quantization on both sides. However, naively running the low-rank branch independently incurs significant overhead due to extra data movement of activations, negating the quantization speedup. To address this, we co-design an inference engine Nunchaku that fuses the kernels of the low-rank branch into those of the low-bit branch to cut off redundant memory access. It can also seamlessly support off-the-shelf low-rank adapters (LoRAs) without re-quantization. Extensive experiments on SDXL, PixArt-$\Sigma$, and FLUX.1 validate the effectiveness of SVDQuant in preserving image quality. We reduce the memory usage for the 12B FLUX.1 models by 3.5$\times$, achieving 3.0$\times$ speedup over the 4-bit weight-only quantization (W4A16) baseline on the 16GB laptop 4090 GPU with INT4 precision. On the latest RTX 5090 desktop with Blackwell architecture, we achieve a 3.1$\times$ speedup compared to the W4A16 model using NVFP4 precision.

• The paper introduces SVDQuant, a novel low-rank SVD-based approach that absorbs quantization outliers in weights and activations for efficient 4-bit diffusion models.
• It reduces memory usage by up to 3.6x and accelerates inference by 3.0x on NVIDIA RTX 4090 GPUs using the tailored Nunchaku engine.
• The method ensures high image fidelity and real-time performance on resource-limited devices, opening avenues for advanced AI applications.

Overview of SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models

The paper presents SVDQuant, a novel approach aimed at enabling efficient deployment of diffusion models by leveraging a 4-bit quantization paradigm that simultaneously quantizes both weights and activations. This approach addresses the significant increase in memory and latency challenges posed by deploying large-scale diffusion models, which necessitates aggressive compression schemes.

Diffusion models have demonstrated remarkable efficacy in generating high-quality images, but their computational demands escalate with increased parameter sizes, unlike LLMs for which weight-only quantization is often adequate. The paper advances the quantization methods by introducing a low-rank component strategy that effectively absorbs outliers, a key issue with diffusion models where both weights and activations are outlier-sensitive.

Core Contributions

The main contribution of this paper is the innovative SVDQuant technique. This technique departs from conventional approaches such as smoothing, where outliers are redistributed between model weights and activations, a method found to be insufficient due to the magnitude and nature of these outliers. Instead, SVDQuant employs a low-rank approximation, attained through Singular Value Decomposition (SVD), to absorb these outlier values, facilitating smoother quantization.

The method involves shifting the burden of quantization difficulty from activations to weights, followed by a low-rank decomposition of the transformed weights, effectively minimizing the residual magnitude for quantization. This is realized through the introduction of a low-rank branch, which operates in a higher precision (16 bits), while only the residuals are quantized to 4 bits.

Key to this approach is the innovative inference engine, Nunchaku, which co-designs kernel operations to minimize the latency overhead typically incurred by additional branches, thereby preserving the speedup offered by low-precision inference.

Empirical Evaluation

SVDQuant demonstrates significant memory and computational efficiency across a range of diffusion models, reliable in maintaining image fidelity. The evaluation spans various architectures such as FLUX.1, PixArt-Σ, and Stable Diffusion XL (SDXL), showcasing substantial reductions in memory usage—up to 3.6 times less than the original BF16 model—and latency improvements by a factor of 3.0x on NVIDIA RTX 4090 GPUs. Notably, for FLUX.1, the method eliminates the need for CPU offloading on GPUs with constrained memory, yielding an overall speedup of 10.1x.

The paper's empirical results show that the proposed method outperforms traditional weight-only quantization approaches and other baselines, such as NF4 4-bit quantization, especially in preserving visual quality as measured by various metrics including FID, LPIPS, and PSNR.

Implications and Future Direction

The proposed SVDQuant expands the potential for real-time AI applications by making the deployment of large-scale diffusion models feasible on resource-limited devices such as edge devices. The approach is versatile, applicable to various diffusion architectures, and well-suited for integration with additional model components such as LoRAs without necessitating re-quantization.

Looking forward, the SVDQuant method paves the way for further exploration of low-precision quantization techniques, particularly focusing on floating-point representations enabled by emerging hardware architectures such as NVIDIA's Blackwell, potentially enhancing efficiency beyond integer quantization.

In conclusion, the paper introduces a refined quantization paradigm that effectively balances computational efficiency with the fidelity demands of large diffusion models, making it a valuable asset in the AI research and deployment communities. This advancement holds promise for expanding interactive applications and improving model accessibility across diversified computational platforms.