LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy (2410.03111v1)

Abstract: The Key-Value (KV) cache is a crucial component in serving transformer-based autoregressive LLMs, enabling faster inference by storing previously computed KV vectors. However, its memory consumption scales linearly with sequence length and batch size, posing a significant bottleneck in LLM deployment. Existing approaches to mitigate this issue include: (1) efficient attention variants integrated in upcycling stages, which requires extensive parameter tuning thus unsuitable for pre-trained LLMs; (2) KV cache compression at test time, primarily through token eviction policies, which often overlook inter-layer dependencies and can be task-specific. This paper introduces an orthogonal approach to KV cache compression. We propose a low-rank approximation of KV weight matrices, allowing for plug-in integration with existing transformer-based LLMs without model retraining. To effectively compress KV cache at the weight level, we adjust for layerwise sensitivity and introduce a progressive compression strategy, which is supported by our theoretical analysis on how compression errors accumulate in deep networks. Our method is designed to function without model tuning in upcycling stages or task-specific profiling in test stages. Extensive experiments with LLaMA models ranging from 8B to 70B parameters across various tasks show that our approach significantly reduces the GPU memory footprint while maintaining performance.

Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy

The paper "LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy" introduces a new approach for compressing the Key-Value (KV) cache in LLMs. The authors focus on autoregressive transformers, which require efficient management of the KV cache to balance computational efficiency and memory usage. This paper circumvents complex attention-based optimizations and costly retraining by leveraging the inherent low-rank characteristics of weight matrices for KV cache compression.

Overview

Modern transformer-based LLMs have remarkable proficiency across numerous tasks but are constrained by substantial memory demands, particularly from the KV cache. This cache is crucial for efficient inference since it stores previously computed attention vectors, thereby streamlining subsequent computations. The traditional approaches to mitigate these memory challenges include designing more efficient attention variants or implementing dynamic token eviction policies. However, these techniques either necessitate extensive retraining or introduce task-specific constraints, thus limiting their applicability to pre-trained models.

The innovative method presented by the authors involves a low-rank approximation to compress weight matrices, specifically targeting the transformation matrices used in the computation of the KV cache. By applying Singular Value Decomposition (SVD) to these matrices, they can reduce the dimensionality and thereby the memory footprint of the cache. This technique bypasses model retraining and task-specific customization, positioning itself as a lightweight, plug-and-play solution for LLM deployment in resource-scarce environments.

Methodology

The compression method is coordinated through a progressive strategy which varies the degree of compression across network layers. The rationale here is that errors introduced by compression can propagate through the network with greater amplification from shallower to deeper layers. To address this, the authors propose varying the compression based on layer sensitivity, which is assessed using cumulative condition numbers of the weight matrices. This enables a nuanced adjustment based on the specific layer, balancing memory savings with performance retention.

The authors theoretically support their method with error bounds on layer compression and error propagation, suggesting that earlier (shallower) layers are more susceptible to noise amplification through subsequent transformations and nonlinearities. This theoretical underpinning emphasizes the efficacy of their progressive compression strategy in mitigating potential performance loss.

Results

The experiments showcase the method's capability using models such as LLaMA-2-13B and LLaMA-3-Instruct-8B/70B across tasks including commonsense reasoning, reading comprehension, text summarization, and mathematical reasoning. Their quantitative analyses demonstrate significant reductions in GPU memory usage—often around 55% to 60%—while maintaining comparable performance to full-cache models. In some instances, especially on certain datasets, the compressed models even showed enhanced task performance.

Implications and Future Directions

The proposed methodology implies a versatile framework for optimizing transformer models in constrained environments without extensive adjustments or fine-tuning. This capability is paramount as LLMs continue to extend in scale and applicability, necessitating efficient deployment mechanisms.

The framework can be extended to incorporate task-specific considerations, potentially allowing for dynamic adaptation and further refinement of the compression strategy. Moreover, as transformer architectures evolve, the low-rank approach could synergize with emerging configurations, maintaining this method's relevance. Additional exploration into the interaction of complex activation functions and compression can also yield insights into further minimizing encountered errors.

In conclusion, this work significantly contributes to the LLM deployment strategies by providing a theoretically sound, adaptable, and efficient KV cache compression mechanism. Through LoRC, the authors offer a method that transcends traditionally rigid approaches, aligning technological advances with practical deployment needs.