Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs

Abstract: Proactive interference (PI) is a documented failure mode in LLMs in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate, mirroring a classical phenomenon in human working memory. Post-training quantization (PTQ) is now the default deployment path for open-weight models, yet its effect on this failure mode has not been tested. We evaluate three precision levels (FP16, INT8, INT4/NF4, via bitsandbytes) across three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct), holding the retrieval task fixed. INT4 quantization significantly reduces accuracy under high interference in every model (e.g., from 81.0% to 68.3% for Qwen), confirmed by paired McNemar's tests ($p \le 2.6 \times 10<sup>{-6}$) and a mixed-effects regression spanning all interference levels; INT8, often assumed safe, also carries a smaller but real penalty in two of three models. The effect is specific to semantically similar (word-type) distractors and reverses sign under a numeric control condition, and is mechanistically linked to a rise in same-key intrusion errors under INT4 (from 21.5% to 24.6% of trials, $p = 4.8 \times 10<sup>{-7}$). A follow-up ablation shows the effect originates in the quantized transformer backbone rather than the output projection layer. These results suggest that bitsandbytes 4-bit quantization can impose an additional cost on applications relying on long, updatable, semantically dense contexts, even when aggregate benchmark accuracy appears largely unaffected. We release our code and tokenizer-verified vocabulary construction method at https://github.com/ShayanShahrabi/compress-and-forget

• The paper shows that bitsandbytes INT4 quantization significantly worsens semantic proactive interference across three instruction-tuned LLMs, with accuracy falling from 81.0% to 68.3% for Qwen2.5-7B at high interference.
• Mixed-effects analysis finds that INT4 harms word-type retrieval as interference grows but reverses direction on numeric controls, indicating representational coarsening rather than generic noise.
• Error analysis and ablations localize the effect to the quantized transformer backbone, while paired testing shows INT8 causes smaller but measurable penalties in two of three models.

Overview

This paper investigates whether post-training quantization (PTQ) degrades a specific, well-documented behavioral failure mode of LLMs: proactive interference (PI), in which retrieval of a repeatedly overwritten value degrades as prior overwrites accumulate. The authors connect two previously disjoint literatures — the PI-LLM paradigm introduced by Wang and Sun (Wang et al., 9 Jun 2025) and the PTQ robustness literature exemplified by LLM.int8() and QLoRA (Xie et al., 2022, Dettmers et al., 2023) — and ask a question that aggregate benchmarking has not addressed: does weight compression disproportionately damage retrieval under semantic interference?

The experimental design is deliberately narrow. Three architecturally distinct instruction-tuned models (Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, Phi-3.5-mini-instruct) are evaluated at three precisions (FP16, INT8 via LLM.int8(), INT4 via NF4 with double quantization, all through bitsandbytes) on the PI-LLM key-rebinding task, with interference levels $k \in \{1, 2, 4, 8, 16, 32, 64, 96\}$. The task contrasts word-type attributes drawn from shared semantic categories (mood, favorite color, favorite animal, occupation) against a numeric-attribute control condition with minimal semantic confusability. All comparisons are trial-level paired: for each seed, the identical trial sequence is presented at every precision, verified with zero mismatches across 23,075 unique trials.

Principal findings

The central result is that INT4 quantization significantly reduces accuracy under high semantic interference in every model tested. At each model's most-powered interference level — Qwen at $k=64$, Mistral at $k=16$, Phi at $k=8$ — the paired McNemar's test yields odds ratios of 2.58 to 20.00 with $p \le 2.6\times10^{-6}$. The most striking case is Qwen2.5-7B, whose word-type accuracy falls from 81.0% to 68.3% between FP16 and INT4 at $k=64$, despite an aggregate accuracy gap of only 1.1 percentage points across all levels. This dissociation is the paper's most consequential claim: the effect is concentrated precisely where aggregate metrics average it away.

The specificity result rules out a generic noise-injection account. A mixed-effects logistic regression over all eight interference levels, three models, and five seeds finds that the INT4 $\times$ log-interference-level interaction is negative under word-type attributes ($\beta = -0.046$, $p = 5.7\times10^{-8}$) but positive under numeric attributes ($\beta = +0.053$, $k=64$0). The effect does not merely vanish on the control condition; it reverses sign. This survives stratifying by numeric token length and adding random slopes for interference by model and attribute, ruling out both tokenization confounds and unmodeled model-specific trends as explanations.

The error analysis provides direct behavioral evidence for the proposed mechanism. Virtually all incorrect responses (99.8–99.9% at every precision) are same-key intrusions — exact matches to earlier overwritten values — rather than unrelated answers. The intrusion rate rises monotonically with more aggressive quantization: from 21.5% of trials at FP16 to 24.6% at INT4 ($k=64$1), while intrusion recency remains stable across precision levels. This pattern supports the paper's interpretation that 4-bit weights coarsen the representational precision needed to keep semantically similar candidate values distinct, rather than injecting generic answer noise.

A follow-up ablation localizes the effect within the architecture. Because bitsandbytes leaves lm_head at full precision by default under the installed library version — a fact the authors diagnose transparently after discovering their skip-list argument had no effect — all main INT4 results reflect backbone-only quantization. Forcibly quantizing lm_head produces no detectable change in any model (all McNemar's $k=64$2) and accounts for at most 2.5% of the total FP16-to-INT4 gap. The penalty therefore originates in the quantized transformer backbone, consistent with the residual-stream representational account.

The revised INT8 picture

A methodological refinement materially changes one conclusion. Under the originally reported unpaired Fisher's exact test, INT8 appeared statistically indistinguishable from FP16 in all three models, yielding a clean "cliff" narrative. The appropriately matched McNemar's test recovers a real, smaller INT8 penalty in two of three models — Mistral ($k=64$3, OR 1.84) and Phi ($k=64$4, OR 2.50) — with Qwen remaining clean ($k=64$5). The practical picture is graded rather than binary: INT4 carries a large penalty everywhere, INT8 a smaller but nonzero penalty in most models tested. The authors' recommendation that INT8 should not be assumed categorically safe for interference-heavy workloads follows directly from this paired re-analysis, and the paper is candid about revising its own earlier framing.

The Interference Endurance Score (IES), a normalized area-under-the-curve summary adapted from Wang and Sun, corroborates the monotonic ordering FP16 $k=64$6 INT8 $k=64$7 INT4 in every model, with INT4 losses of 1.8%, 5.1%, and 5.2% for Qwen, Mistral, and Phi respectively. The authors correctly note IES's own limitation: because Qwen remains near ceiling through $k=64$8, averaging dilutes its dramatic single-point collapse at $k=64$9 to a 1.8% aggregate loss. They therefore report per-level and standardized analyses side by side rather than either alone — a reasonable epistemic posture, though the IES pooling convention across attributes with different vocabulary caps is flagged in the appendix as unverified.

Limitations

The paper's scope restrictions are acknowledged plainly. Quantization coverage is bitsandbytes-specific; calibration-based methods (GPTQ, AWQ) and formats such as EXL2 and HQQ are untested, so whether the effect reflects NF4 specifically or aggressive weight compression generally remains open. Activation and KV-cache quantization, common in production serving stacks, are likewise unexamined. The numeric control pool is not tokenizer-filtered to be single-token, unlike the word-type pools; the authors mitigate this with flat token-length composition checks and stratified analysis, but concede it is an asymmetry in rigor. Prompts are short (a few hundred tokens at most), leaving the behavior of the effect in long-context regimes an open question that prior work on quantized long-context retrieval (Mekala et al., 26 May 2025) suggests could resolve in either direction. The proposed mechanism rests on two behavioral evidence lines — intrusion analysis and backbone localization — neither of which constitutes a representational-level test of residual-stream precision or attention dynamics. Only greedy decoding is tested, and the model pool spans only 3.8–7B parameters. The ablation dataset's pairing was also not re-verified with the exhaustive zero-mismatch check applied to the main dataset, which the authors flag explicitly.

One additional caveat deserves emphasis: the effect is established for bitsandbytes-style PTQ at inference time and must not be conflated with training-time phenomena such as quantization acting as a regularizer during continual learning (Zhang et al., 22 Dec 2025); the mechanisms are distinct despite superficially similar framing.

Conclusion

This paper demonstrates that bitsandbytes 4-bit quantization selectively amplifies proactive interference in LLMs — a statistically robust effect across three architectures, confirmed by paired tests, mixed-effects regression, error-type analysis, and component ablation, yet invisible to aggregate benchmarking. Its specificity (opposite-signed effects under semantic versus numeric interference) argues against generic noise accounts and toward a representational-coarsening mechanism localized in the transformer backbone. The principal open questions are whether the effect extends beyond the NF4 scheme to calibration-based quantizers, and how it interacts with long-context regimes and internal representations probed directly.