Uncertain side effects of localized knowledge editing in LLMs
Characterize and quantify the side effects induced by knowledge editing techniques that modify localized components of Large Language Models, given the lack of clarity about where knowledge is stored and how edits propagate within model internals.
References
However, the side effects are unclear as the underlying LLM mechanisms still need to be clarified [79].
Developing more principled approaches that provide tokenization-robust knowledge modification without substantially expanding collateral effects remains an open problem.
The magnitude of the naive collapse itself varies across architectures ($17.39$ points on GPT-2 XL versus $4.75$ on Qwen), and isolating the contributions of layer depth, key dimension, and $\lambda$ to this difference is left to future work.