Do LLMs genuinely understand the historical evolution of software libraries?

Determine whether large language models used for repository-level Python migration (e.g., flagship models such as Claude Sonnet 4 and GPT-5) genuinely understand the precise historical evolution and version-specific changes of third-party libraries, or whether their evolution-aware rationales are post-hoc and not grounded in accurate version histories; specifically, assess if their explanations and edits correctly reflect documented version transitions (for example, pysnmp 7.x replacing asyncore with asyncio) across a wide range of libraries.

Background

In the case studies, the authors observe that advanced models often produce evolution-aware reasoning, describing historical changes in libraries (e.g., pysnmp replacing asyncore with asyncio in 7.x) and updating deprecated APIs appropriately. This appears impressive given the scarcity of large-scale, structured migration resources.

Despite these observations, the authors explicitly note uncertainty about whether the models truly possess accurate knowledge of detailed version histories across many libraries. They further emphasize that prior work indicates even strong proprietary models struggle to identify the specific version in which API changes occurred, motivating a focused investigation of whether the produced rationales reflect genuine historical understanding or post-hoc justification.