Causes of Unexplained Step Changes in Offline Model Responses

Determine the causes of the persistent step changes observed in the longitudinal embeddings of offline large language model responses—such as the shifts around September 3 and September 24, 2024—that could not be attributed to publicly announced model updates.

Background

In the longitudinal analysis, the authors observe distinct step changes in response embeddings over time. While some can be linked to known checkpoint updates (e.g., the October 2 GPT‑4o update), others—particularly around September 3 and September 24—do not align with any known updates, despite the models being queried at temperature zero and, in these comparisons, without internet access.

The authors explicitly state that the causes of these unexplained step changes are not clear, indicating a need for future work to identify underlying drivers such as unannounced updates, infrastructure changes, guardrail adjustments, or other system-level factors.