Generalizable Characterization of LLM Behavior

Characterize which properties of large language model behavior in software engineering are generalizable across model generations rather than transient capability gaps.

Background

The paper notes that empirical results for LLM-driven refactoring depend strongly on the model generation being evaluated. Although benchmark performance improves rapidly, it remains difficult to distinguish enduring behavioral characteristics from limitations that may disappear as models improve. This unresolved methodological issue affects the validity and transferability of evaluations of autonomous refactoring systems.