Safe Embedding-Matrix Bit Width for Quality-Preserving Compression

Determine the lowest bit width to which the Gemma 3 1B embedding matrix can be compressed without degrading the quality of generated tokens, using direct evaluation or a proxy metric that does not require full-model quantization.

Background

The paper identifies the embedding matrix as a promising compression target because its weights are static and its large vocabulary dimension makes it a substantial part of the model. Lowering its precision could therefore reduce memory use and accelerate inference.

The authors explicitly state that the safety limit for embedding quantization has not been established and propose empirical evaluation of different bit widths through token-quality measurements or a suitable proxy metric.

References

However, this hypothesis requires empirical verification: up to which bit-width can the embedding matrix be safely compressed without affecting the quality of generated tokens? This value can be identified by quantizing the embedding matrix and measuring the tokenization quality either by direct evaluation or some proxy metric that does not require full model quantization.

— A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models  (2608.26926 - Safronov, 27 Aug 2026) in Section 2, subsection “Formula variant for Embedding”