BitsAndBytes: 4-bit Quantization

• BitsAndBytes is an open-source quantization library that performs 4-bit, weight-only post-training quantization using NF4, reducing model memory without retraining.
• It integrates natively with Hugging Face Transformers and supports nested and double quantization for applications in machine translation, ASR, and mobile deployment.
• Optimized kernel designs in BitsAndBytes yield notable speedups and memory savings, though trade-offs in latency, robustness, and accuracy vary by task and hardware.

BitsAndBytes, often abbreviated as BNB or styled as bitsandbytes, is an open-source quantization library that integrates natively with Hugging Face Transformers to load pretrained models in 8-bit and 4-bit precision without retraining. In the recent literature, it appears primarily as a post-training quantization mechanism for weight-only compression, as a source of runtime kernels for low-precision inference on NVIDIA GPUs, and as an ecosystem component around which downstream optimizations, conversion pipelines, and post-quantization correction methods are built. Across machine translation, automatic speech recognition, mobile deployment, and automated program repair, BitsAndBytes is associated most strongly with 4-bit NormalFloat quantization (NF4), optional nested or double quantization, and calibration-free workflows; however, the empirical record is uneven, with quality, latency, robustness, and energy behavior varying sharply by task, model scale, and hardware regime (Marie et al., 28 Aug 2025, Söhler et al., 11 Nov 2025, Qi et al., 2 Apr 2026, Yadav et al., 6 Dec 2025, Vallecillos-Ruiz et al., 25 Jun 2026).

1. Quantization model and numerical formulation

In the studies considered here, BitsAndBytes is primarily used for weight-only post-training quantization. For multilingual machine translation, activations and the KV cache remain in higher precision while only the weights are quantized; for LLM inference, the library stores quantized weights together with per-group scaling metadata and dequantizes back to FP16 or BF16 for execution on current NVIDIA GPUs, which do not provide native 4-bit tensor core compute in the configurations discussed (Marie et al., 28 Aug 2025, Qi et al., 2 Apr 2026).

The central BitsAndBytes 4-bit format in these papers is NF4, a non-uniform codebook quantizer tailored to weight distributions that are approximately normal. In the machine translation study, NF4 uses codebook entries $\{c_k\}$ derived from normal-distribution quantiles, with scales $s_g$ estimated per group and no explicit zero-point for codebook mapping. For contrast, the same study gives the standard grouped uniform quantization formulation

$$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$

whereas BitsAndBytes NF4 uses the non-uniform mapping

$$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$

The same source notes that mean squared error,

$\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$

is a common analytic lens, but that BitsAndBytes does not explicitly optimize MSE or a downstream loss; it relies instead on the NF4 codebook and per-group scaling without per-language calibration (Marie et al., 28 Aug 2025).

At the storage level described for NF4 LLM inference, BitsAndBytes packs weights as 4-bit indices, two per byte, and stores a scale per group, with a block size of 64 elements in the optimized-kernel study. Dequantization then recovers runtime values through $D(q_i) = s_g \cdot C[q_i]$, where $C$ is the 16-entry NF4 codebook. Nested or double quantization further compresses the metadata by quantizing the per-block scales themselves; this feature is explicitly enabled in the multilingual MT study and described as optional in the ASR and mobile-deployment discussions (Qi et al., 2 Apr 2026, Marie et al., 28 Aug 2025, Söhler et al., 11 Nov 2025, Yadav et al., 6 Dec 2025).

2. Software integration and execution pathways

BitsAndBytes is used in these papers through the Hugging Face Transformers integration. The reported configurations center on load_in_4bit=True, bnb_4bit_quant_type="nf4", and a higher-precision compute dtype such as torch.float16 or torch.bfloat16; optional fields such as bnb_4bit_use_double_quant appear in the mobile and ASR material, although not every paper reports whether the option was enabled in the final measurements. In the multilingual MT study, the authors use the default Transformers integration and do not customize block size or per-channel versus per-group granularity beyond enabling nested quantization; in the Whisper-small study, the paper does not report the internal per-tensor or per-channel granularity used by BNB NF4, nor the block sizes (Marie et al., 28 Aug 2025, Söhler et al., 11 Nov 2025, Yadav et al., 6 Dec 2025).

Execution modality is not uniform across domains. In the Whisper-small ASR study, BitsAndBytes is evaluated on GPU only, with all reported BNB results using dynamic NF4; no BNB static quantization is evaluated there. The paper explicitly states that it did not test BitsAndBytes int8 or LLM.int8() mixed precision, although it cites LLM.int8() as related work. In the mobile-execution study, BitsAndBytes is used on a Google Colab NVIDIA T4 GPU as a Python-side PTQ step for Meta’s Llama 3.2 3B model, after which the model is converted to GGUF via llama.cpp tools and deployed on Android through Termux and Ollama. That paper emphasizes that BitsAndBytes is used for quantizing and evaluating in Python, whereas the final Android execution uses GGUF tooling rather than BitsAndBytes directly (Söhler et al., 11 Nov 2025, Yadav et al., 6 Dec 2025).

The same body of work also clarifies boundaries of library scope. In the APR study, BitsAndBytes appears only as 4-bit weight-only quantization (bnb_4); KV-cache quantization is instead provided by HQQ and Quanto through the Transformers quantized-cache mechanism. In the multilingual MT study, BitsAndBytes is 4-bit only in the evaluated setup, and no 2-bit BitsAndBytes configuration is reported (Vallecillos-Ruiz et al., 25 Jun 2026, Marie et al., 28 Aug 2025).

3. Multilingual machine translation

A large-scale evaluation of post-training quantization in multilingual machine translation provides the most extensive task-specific characterization of BitsAndBytes NF4. The study quantizes five LLMs ranging from 1.7B to 70B parameters—Qwen3-1.7B, Qwen3-8B, Llama-3.1-8B-Instruct, Qwen3-32B, and Llama-3.3-70B—and evaluates 55 languages in both directions to and from English, for 110 translation directions in total. Translations come from WMT24++ across four domains, COMET (wmt22-comet-da) is the primary metric, and BLEU and chrF are also computed with SacreBLEU. The paper explicitly warns that averaging scores across languages is methodologically unsound; comparisons are instead made per language pair or through rank-based assessment (Marie et al., 28 Aug 2025).

Within that evaluation, BitsAndBytes is calibration-free 4-bit NF4 with nested quantization enabled. Quality preservation is uneven. High-resource language pairs often exhibit modest degradation, typically on the order of approximately 0.3 to 1.0 COMET, while low-resource and typologically diverse languages degrade more sharply, especially Indic languages and Zulu. Representative English-to-target results include Qwen3-8B Japanese $85.7 \rightarrow 85.4$ ($-0.3$), French $80.8 \rightarrow 80.0$ ($s_g$0), Bengali $s_g$1 ($s_g$2), Malayalam $s_g$3 ($s_g$4), and Zulu $s_g$5 ($s_g$6). For the larger Llama-3.3-70B model, English-to-target Zulu falls from $s_g$7 to $s_g$8 under Qwen3-32B and from $s_g$9 to $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$0 under Llama-3.3-70B, showing that high absolute performance does not eliminate language-specific losses (Marie et al., 28 Aug 2025).

The reverse direction, translating into English, exposes stronger outliers. The most notable are on Llama-3.1-8B, where Bengali-to-English drops from $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$1 to $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$2 ($$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$3 COMET) and Malayalam-to-English from $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$4 to $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$5 ($$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$6); Zulu-to-English on the same model drops from $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$7 to $$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$8 ($$q_i = \mathrm{clip}\Big( \big\lfloor \tfrac{W_i}{s_g} \big\rceil + z_g,\; q_{\min},\; q_{\max} \Big), \quad \hat{W}_i = s_g\,(q_i - z_g),$$9). On Llama-3.3-70B, Bengali-to-English declines from $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$0 to $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$1 ($$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$2), Malayalam-to-English from $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$3 to $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$4 ($$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$5), and Zulu-to-English from $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$6 to $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$7 ($$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$8). The paper summarizes this pattern by stating that “algorithmic robustness is not invariant to scale”: BitsAndBytes is competitive around 8B, but among the tested PTQ methods it becomes the worst option at 70B in Tables 1 and 2, while GGUF typically yields the smallest loss and AWQ is steadier at larger scale (Marie et al., 28 Aug 2025).

Operationally, the study also provides guidance relevant to BitsAndBytes deployments even though the detailed decoding sweep is performed with GGUF rather than BNB. Increasing temperature from $$q = \arg\min_k |x/s_g - c_k|, \qquad \hat{x} = s_g \cdot c_q.$$9 to $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$0 reduces COMET by approximately $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$1 to $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$2 points across languages, whereas tightening top-$\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$3 from $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$4 to $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$5 changes scores by at most about $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$6 COMET, and the ranking of weight precisions remains stable across those settings. The paper therefore recommends deterministic decoding ($\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$7) for quantized MT and notes that language-matched calibration, which helps mainly at 2-bit for GGUF, is not applicable to BitsAndBytes NF4 because the method is calibration-free (Marie et al., 28 Aug 2025).

4. Automatic speech recognition

BitsAndBytes has also been evaluated in a cross-library PTQ study of Whisper-small, a 244M-parameter Transformer-based ASR model. In that setting, BNB is one of four PTQ libraries and is tested only on GPU using dynamic NF4. The experimental datasets are LibriSpeech test-clean and test-other, the baseline is FP32, and the reported runtime metric is Real-Time Factor (RTF), with experiments run on DeiC National HPC hardware consisting of Lenovo ThinkSystem SR675 V3 nodes with dual AMD EPYC 9454 CPUs, 768 GB DDR5-4800 memory, and four NVIDIA Hopper H100-SXM5 GPUs per node (Söhler et al., 11 Nov 2025).

The quantitative picture is compact but informative. Relative to the FP32 GPU baseline, BitsAndBytes NF4 yields WER_c = 3.54 versus 3.48, WER_o = 13.49 versus 11.88, CER_c = 1.05 versus 1.02, and CER_o = 4.05 versus 3.62. RTF changes from 0.006 for FP32 to 0.008 for BNB NF4, and model size reduction is reported as 70%. The paper interprets this as a minimal absolute WER increase on clean speech (+0.06) but a much larger degradation on noisy speech (+1.61), indicating reduced robustness in adverse acoustic conditions. A figure tracking WER increase on test-other relative to test-clean further shows that lower-bit formats, including BNB NF4, exhibit larger robustness losses under noise (Söhler et al., 11 Nov 2025).

Comparative results sharpen the interpretation. Optimum-Quanto dynamic int8 on GPU attains WER_c = 3.41, WER_o = 10.65, and the same RTF = 0.008, so it dominates BNB NF4 on accuracy while offering less compression (57% versus 70%). HQQ dynamic int3 reaches WER_c = 4.11, WER_o = 12.93, RTF = 0.019, and 71% size reduction; relative to this point, BNB NF4 is faster and slightly less compressed, with better clean-set accuracy but worse noisy-set robustness. The study’s practical conclusion is correspondingly narrow: choose BNB NF4 on GPU when memory constraints are severe and the deployment environment is mostly clean, but prefer int8 when robustness to noisy or diverse acoustic conditions is required (Söhler et al., 11 Nov 2025).

5. Kernel architecture and performance bottlenecks

Beyond its role as a quantization API, BitsAndBytes is also a systems target. The paper “Fast NF4 Dequantization Kernels for LLM Inference” identifies the NF4 dequantization path in BitsAndBytes as a memory-bound bottleneck. In that implementation, NF4 weights are stored as 4-bit codes together with per-group scaling metadata, and the runtime repeatedly dequantizes those weights to FP16 before GEMM execution. The optimization proposed there preserves the same API and tensor layouts, modifies the kDequantizeBlockwise path, and remains token-exact. Its main ideas are to stage the 16-entry NF4 LUT once per thread block from constant memory into shared memory and to replace a branch-heavy 4-level index-decoding tree with simple nibble extraction by shift and mask. The reported configuration uses 64 threads per block, 8 elements per thread, a 512-element tile, and only 64 bytes of shared memory per block for the LUT (Qi et al., 2 Apr 2026).

The measured gains are substantial in the reported A100-80GB setup with CUDA 12.6, PyTorch 2.1, and BitsAndBytes 0.47.0. Kernel-level speedups are 2.10× on Gemma 27B, 2.19× on Qwen3 32B, and 2.04× on Llama3.3 70B. End-to-end improvements reach 1.54×, with example latency changes of 39.96 → 26.00 seconds for Llama3.3 70B at batch 2, 23.13 → 17.49 for Gemma 27B at batch 64, and 25.10 → 19.43 for Qwen3 32B at batch 32. The paper attributes these gains to exploiting the approximately 12–15× latency advantage of shared memory over global memory access, reducing LUT global-memory traffic by 64× per block, cutting instruction count for index decoding by 71%, and eliminating warp divergence (Qi et al., 2 Apr 2026).

A broader comparative perspective is given by FLUTE, which targets lookup-table-quantized LLMs in the same low-bit, memory-bandwidth-limited regime. That work explicitly notes that BitsAndBytes already supports LUT-based NormalFloat quantization and fused dequantization-plus-GEMM kernels for 4-bit NF4, then proposes an alternative kernel design built around offline weight restructuring, pairwise vectorized LUT lookups, shared-memory LUT duplication to reduce bank conflicts, and Stream-K work decomposition. At batch sizes below 32 and group size 128, FLUTE reports kernel speedups of 2–4× over existing GEMM kernels and end-to-end throughput increases of 1.5–2×; it also extends the LUT/NF approach to 3-bit settings that BitsAndBytes kernels generally do not support in the described workflows (Guo et al., 2024).

6. Deployment patterns, limitations, and post-quantization extensions

BitsAndBytes is frequently used as the first stage in a larger deployment pipeline rather than as the terminal runtime format. In the mobile-execution study, Meta’s Llama 3.2 3B model is quantized with BitsAndBytes NF4 on a T4 GPU and then converted to GGUF with llama.cpp tools, ending in a q4_k_m artifact for Android inference through Termux and Ollama. The paper reports a model-size change from 6.00 GB in BF16 to 2.10 GB after BitsAndBytes 4-bit PTQ, corresponding to a 64.92% reduction, and then to 1.88 GB for the final GGUF q4_k_m, a 68.66% reduction. The same work reports MMLU dropping from 64.2% to 61.8%, WikiText-2 perplexity of 8.57 ± 0.06, and DailyMail summarization BLEU = 0.45, while describing the outputs as coherent and usable for general tasks. It also notes that ideal INT4 memory for 3B parameters would be about 1.50 GB, so the measured 1.88 GB reflects metadata, partially unquantized tensors, and GGUF block-quantization overhead (Yadav et al., 6 Dec 2025).

The APR study exposes a different deployment frontier. Across six models from 6.7B to 70B on HumanEval-Java and Defects4J, the BitsAndBytes bnb_4 configuration reduces in-memory model size by −64.48% to −73.78%, depending on model, but increases inference time by +126.87% to +354.42% and energy by approximately +126% to +353%. Effectiveness remains near baseline in some cases and shifts in others: on HumanEval-Java, bnb_4 changes repaired-problem totals from 74 to 76 for Llama-8B, 88 to 90 for Mixtral-8x7B, and 104 to 104 for DeepSeek-33B; on Defects4J, it changes totals from 35 to 39 for Mistral-7B, 60 to 64 for Mixtral-8x7B, and 81 to 83 for DeepSeek-33B, while decreasing performance for several other model-task pairs. The same paper stresses that similar totals can hide low solved-set overlap, formalizing that issue with Jaccard Index and Jaccard Consistency Rate, and reporting that 48% of all configurations evaluated are strictly dominated in Pareto analysis. For BitsAndBytes users, the implication is not that bnb_4 is uniformly preferable, but that its utility depends on whether the primary objective is memory reduction, behavioral consistency, or energy-latency efficiency (Vallecillos-Ruiz et al., 25 Jun 2026).

Recent work also treats BitsAndBytes as a substrate for post-quantization correction. GlowQ is designed to complement 4-bit quantized LLMs, including BitsAndBytes NF4 and FP4, by adding a low-rank correction term to selected modules:

$\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$8

where a shared right factor $\mathrm{MSE} = \mathbb{E}[(x-\hat{x})^2],$9 is cached once per input-sharing group and reused across multiple modules. This avoids restoring every layer independently, as in earlier low-rank correction methods. On the reported benchmarks, GlowQ reduces time to first token by 5.6% and increases throughput by 9.6% on average, while GlowQ-S reduces TTFB by 23.4% and increases throughput by 37.4%, maintaining accuracy within 0.2 percentage points on average. In this sense, BitsAndBytes serves not only as a quantizer but also as a compositional base for later accuracy-recovery mechanisms (An et al., 26 Mar 2026).

Taken together, these results define a specific technical profile. BitsAndBytes is a calibration-free, Hugging Face-centric low-precision library whose NF4 path is effective for weight memory reduction and often adequate for high-resource or clean-input regimes. Its limitations are equally clear: 2-bit support is absent in the multilingual MT setup, KV-cache quantization is outside the evaluated BNB scope in APR, noisy ASR conditions expose robustness losses, multilingual low-resource translation can incur multi-point COMET degradation, and low-bit execution may increase latency and energy when hardware utilization is unfavorable. The literature therefore places BitsAndBytes not as a universally dominant quantization method, but as a widely adopted and technically consequential component within a broader quantized-model stack (Marie et al., 28 Aug 2025, Söhler et al., 11 Nov 2025, Vallecillos-Ruiz et al., 25 Jun 2026).