q4_k_m Quantization for Neural Networks

Updated 5 April 2026

q4_k_m Quantization is a mixed-precision blockwise scheme that compresses neural network weights to 4 bits per parameter, optimizing memory and speed.
It applies groupwise floating-point scaling and zero-point adjustments within fixed-size blocks to balance minimal accuracy loss with significant efficiency gains.
Empirical studies on LLMs and vision-language models show q4_k_m delivers substantial size reduction and faster inference, making it ideal for resource-constrained deployments.

q4_k_m Quantization refers to a family of mixed-precision, blockwise quantization schemes for neural network model weights, most prominently implemented in the llama.cpp toolkit and industry-standard GGUF formats. Q4_K_M compresses each block of weights to 4 bits per parameter (INT4), with “K” denoting quantization group/block size, and “_M” indicating certain layers may use alternate (e.g., 6-bit) precision for numerical stability and accuracy tradeoffs. Q4_K_M quantization has been validated in LLMs, vision-LLMs, and general deep learning workloads for achieving pronounced memory savings and fast inference at minimal to negligible accuracy loss (Yasuno, 12 Mar 2026, Yasuno, 24 Mar 2026, Zhao et al., 5 May 2025).

1. Mathematical Formulation and Algorithm

Q4_K_M quantizes weights by partitioning each weight tensor into fixed-size blocks (e.g., 64, 128, or 256), then applying uniform quantization within each block. Each block is encoded by a groupwise floating-point scale $s_g$ and zero-point $z_g$ (stored at FP16/FP32 precision).

Quantization Process:

For a block $W_g = \{w_i\}$ $W_{g} = {w_{i}}$ of $G$ $G$ consecutive weights:
- Compute block minimum $w^\mathrm{min}_g$ and maximum $w^\mathrm{max}_g$ .
- Compute scale $s_g = (w^\mathrm{max}_g - w^\mathrm{min}_g)/(2^4-1)$ .
- Compute zero-point $z_g = \operatorname{round}(-w^\mathrm{min}_g/s_g)$ .
- For each $w \in W_g$ :
$q = \operatorname{clip}\left(\operatorname{round}\left(\frac{w}{s_g}\right) + z_g,\, 0,\,15\right)$ - Store $z_g$ 0 (4 bits), $z_g$ 1, $z_g$ 2 for each block.

Dequantization during inference:

$z_g$ 3

The "_M" suffix signals "mixed" precision: while 4 bits is standard, some layers (e.g., attention norm, output) may use 6 bits to avoid outlier-induced underflow or quality loss (Yasuno, 12 Mar 2026, Zhao et al., 5 May 2025).

2. Integration with Neural Model Pipelines

In most deployments, Q4_K_M is a post-training quantization scheme applied after any further fine-tuning or adapter merging. A typical workflow in the GGUF/llama.cpp ecosystem involves:

Completing all model fine-tuning (e.g., QLoRA with LoRA adapters).
Merging adapters into the base weights to obtain a single-precision model.
Running a quantization tool (e.g., llama-quantize ... q4_k_m) to generate Q4_K_M-quantized files.
Loading and running the quantized model in inference frameworks.

No calibration dataset or special tuning is generally required. Double quantization and blockwise statistics are handled automatically by the toolkit (Yasuno, 12 Mar 2026, Zhao et al., 5 May 2025).

3. Quantitative Effects on Model Quality, Size, and Speed

Empirical studies across transformer architectures establish the trade-offs enabled by Q4_K_M:

Model/Task	Full-Precision Score	Q4_K_M Score	Δ Score	Speedup	Size Reduction
Swallow-8B (JP LLM)	2.820/3	2.830/3	+0.010	6.1×	16→4.9 GB
ELYZA-JP-8B	2.700/3	2.730/3	+0.030	3.3×	16→4.9 GB
LLaVA-1.5-7B	2.93/5	2.93/5	—	29%↑	14→4.1 GB
DeepSeek-R1 (671B)	83.48	82.70	−0.78	~1×	671→377 GB
DeepSeek-V3 (671B)	70.05	69.82	−0.23	~1×	671→377 GB

For architectures such as Llama-3 and LoRA-adapted Japanese LLMs, Q4_K_M can deliver a slight average-score improvement, attributed to regularization of overfitted LoRA weights (Yasuno, 12 Mar 2026).
For extremely large models (e.g., DeepSeek-671B), accuracy loss is sub-1% across most reasoning and knowledge benchmarks (Zhao et al., 5 May 2025).
In vision-language evaluation (LLaVA), Q4_K_M achieves 0.54 quality/sec—higher than FP16 and Q8_0; its mean qualitative score is 2.93/5, with unimodal competitors showing slightly higher mean and less variance (Yasuno, 24 Mar 2026).

4. Trade-Offs and Comparative Performance

Q4_K_M achieves the lowest memory footprint among fixed-bit, blockwise PTQ variants (~29%–44% of FP16 weight size), enabling deployment on 12–16 GB consumer GPUs (for 7–8B models) and on 8×80GB enterprise nodes for 671B models (Zhao et al., 5 May 2025, Yasuno, 12 Mar 2026, Yasuno, 24 Mar 2026).
Inference acceleration is realized primarily because blockwise integer kernels maximize tensor-core efficiency (4.5%–29% relative speedup over Q5_K_M/Q8_0) (Yasuno, 24 Mar 2026).
For transformer architectures with grouped-query attention (GQA, e.g., Qwen2.5), Q4_K_M can degrade performance substantially (−0.28 absolute), and higher precision (Q8_0) is recommended in such cases (Yasuno, 12 Mar 2026).
Q4_K_M is less robust to length/structure in output distributions, showing higher bimodality and negative correlation of output length and quality, especially in complex reasoning tasks (Yasuno, 24 Mar 2026).

5. Implementation Considerations and Hardware Deployment

Q4_K_M is natively supported in multiple toolkits (llama.cpp, GGUF), and does not require per-layer calibration, only per-block statistics aggregated from a tiny calibration set (Zhao et al., 5 May 2025).
The practical limit for reliable deployment is set by GPU VRAM (<64 GB: Q4_K_M not feasible for 671B models; use DQ3_K_M). For 7–8B models, deployment is trivial on 12–16 GB consumer hardware (Yasuno, 12 Mar 2026, Zhao et al., 5 May 2025).
Standard configuration maintains all transformer layers on GPU; only the vision projector or specialized heads in VLMs need to remain in higher precision (Yasuno, 24 Mar 2026).
Code example (Python, llama-cpp-python): $z_g$ 4
No specialized CUDA kernels are required beyond standard BLAS/int4 routines.

6. Limitations and Model/Task Suitability

Q4_K_M is not suitable for all architectures: for certain configurations (notably GQA and extremely small VRAM environments), higher-bit or dynamic schemes (e.g., DQ3_K_M) may be superior (Yasuno, 12 Mar 2026, Zhao et al., 5 May 2025).
Instabilities (bimodal quality, length-related hallucination) are observed in complex VLM tasks, making it less recommendable for deployment requiring consistent high-quality outputs (Yasuno, 24 Mar 2026).
Memory footprint, while heavily reduced, can still exceed single-device limits for ultra-large models. Dynamic quantization and per-layer adaptation are necessary for further compression.

7. Relation to Broader Quantization Theory

Q4_K_M instantiates a practical, post-training, blockwise affine quantization representative of modern integer-only quantization strategies, yielding minimal to negligible accuracy loss at scale (Yasuno, 12 Mar 2026, Yasuno, 24 Mar 2026, Zhao et al., 5 May 2025). As contrasted to learned quantization grids, dynamic k-means, or stochastic quantization, Q4_K_M prioritizes maximum regularity, inference efficiency, and hardware compatibility, making it central in productionizing LLMs for resource-constrained and high-throughput settings. When compared with mixed or adaptive approaches, Q4_K_M remains the default fixed-precision, high-throughput, low-error baseline against which more sophisticated or aggressive quantization methods are evaluated.