Verification of Roofline Predictions for Layer-Wise Heterogeneous Quantization

Verify whether the roofline-based speedup prediction remains accurate for layer-wise heterogeneous quantization of Gemma 3, particularly when different layers use Q4 and Q8 precision and dequantization overheads may differ between the two formats.

Background

The paper analytically estimates a heterogeneous configuration in which the middle ten Gemma 3 layers remain at Q8 while the first and last eight layers use Q4. This configuration is predicted to provide greater speedup than uniform Q8 quantization while retaining higher precision in the most sensitive layers.

The prediction is unresolved in practice because the overhead of loading and dequantizing Q4 and Q8 weights may not be identical. The authors state that verification requires fine-grained computation-graph control through tools such as ExLlamaV3 or TensorRT-LLM, which were unavailable for Gemma 3 within the study.

References

However, it is possible that under layer-wise heterogeneous quantization the overhead of dequantizing weights when loading them into compute cores may differ for Q4 and Q8, potentially introducing an additional prediction error. Practical verification of this result requires tools with fine-grained control over the computation graph, such as ExLlamaV3 or TensorRT-LLM, which were not available for the Gemma 3 architecture within the scope of this study.

— A Layer Importance Metric for Quantization Accounting for the Speed-Quality Trade-off in Autoregressive Models  (2608.26926 - Safronov, 27 Aug 2026) in Section “Formalizing the Latency Parameters of FFN and Embedding (Speed Component),” subsection “FFN”