Long-context scaling of quantization-amplified proactive interference

Determine whether the proactive-interference penalty caused by bitsandbytes INT4 quantization strengthens or attenuates when the PI-LLM retrieval paradigm is extended from the few-hundred-token prompts tested in the study to contexts of 8,000 or more tokens.

Background

The study evaluates Qwen2.5-7B-Instruct, Mistral-7B-Instruct-v0.3, and Phi-3.5-mini-instruct under FP16, INT8, and bitsandbytes INT4/NF4 quantization using a proactive-interference retrieval task in which a subject’s attribute is repeatedly overwritten and the model must report only the latest value. The reported prompts contain at most a few hundred tokens, and the principal finding is that INT4 quantization disproportionately harms retrieval under semantically confusable interference while producing an opposite-signed effect under the numeric control condition.

Whether this quantization-related failure becomes more severe or instead diminishes in genuinely long-context settings remains unresolved. The question is practically important because quantized models are widely used for applications involving long, frequently updated conversational or document-based state, and prior work on quantized long-context retrieval suggests that the direction and magnitude of the effect cannot be inferred from the short-context experiments alone.

References

Our prompts are also short-to-medium (at most a few hundred tokens), so whether the effect strengthens or attenuates in the 8k+-token long-context regime is an open, practically important question -- one that prior work on quantized long-context retrieval \citep{mekala2025quantization} suggests could go either way and would be worth testing directly with our paradigm.

— Compress and Forget: bitsandbytes Quantization Amplifies Proactive Interference in LLMs  (2608.18578 - Shahrabi-Farahani et al., 19 Aug 2026) in Section 6, “Limitations and Future Work”