Residual Quantization with Implicit Neural Codebooks (2401.14732v2)

Abstract: Vector quantization is a fundamental operation for data compression and vector search. To obtain high accuracy, multi-codebook methods represent each vector using codewords across several codebooks. Residual quantization (RQ) is one such method, which iteratively quantizes the error of the previous step. While the error distribution is dependent on previously-selected codewords, this dependency is not accounted for in conventional RQ as it uses a fixed codebook per quantization step. In this paper, we propose QINCo, a neural RQ variant that constructs specialized codebooks per step that depend on the approximation of the vector from previous steps. Experiments show that QINCo outperforms state-of-the-art methods by a large margin on several datasets and code sizes. For example, QINCo achieves better nearest-neighbor search accuracy using 12-byte codes than the state-of-the-art UNQ using 16 bytes on the BigANN1M and Deep1M datasets.

Introduction to Residual Quantization with Neural Networks

Residual Quantization (RQ) is an iterative method used extensively in multi-codebook vector quantization for tasks like data compression and vector search. However, the effectiveness of this method is contingent upon the distribution of residual errors, which may vary significantly due to the selection of codewords in previous steps. Traditional RQ techniques sidestep this issue by utilizing a generic, fixed codebook at each quantization step, a strategy that fails to leverage the dependency between residuals and prior quantization choices.

QINCo: A Novel Approach in Neural Residual Quantization

Recent work introduces QINCo (Quantization with Implicit Neural Codebooks), a neural network adaptation of RQ which tailors codebooks to individual data points by predicting specialized codebooks based on previous quantization approximations. Unlike conventional methods, where a static set of codebooks is employed, QINCo dynamically adjusts each step, enabling a significant enhancement of quantization accuracy. This methodology has shown remarkable improvements over the current state-of-the-art across several datasets and code sizes.

Training Stability and Compatibility with Fast Search Techniques

In contrast to other neural Multi-Codebook Quantization (MCQ) methods where gradient-based optimization proves challenging, QINCo decodes in the original data space, negating the need for complex gradient propagation. This architectural choice simplifies training, ensures stability, and circumvents the widely encountered issue of codebook collapse seen in networks of this nature. Additionally, QINCo's design shares similarities with traditional RQ, supporting integration with inverted file indexes (IVF) and re-ranking for fast approximate decoding, which enhances its utility in efficient similarity search tasks.

Quantization Performance and Scalability

QINCo not only excels in its primary function but also demonstrates superior performance when scaled. Its ability to adapt and utilize implicit neural codebooks across multiple quantization steps optimizes the residual distribution and outperforms fixed codebook counterparts. The robustness of QINCo is underscored by its consistent performance increase with additional training data, highlighting its suitability for large-scale machine learning systems.

In conclusion, QINCo represents a significant advancement in the field of vector quantization, offering an innovative solution that aligns with the current and future demands of data compression and similarity search applications. Its neural network-based approach that customizes codebooks to data points ensures that as the complexity of datasets increases, so does the precision and efficiency of the quantization process.