Hidden Breakthroughs in Language Model Training

Abstract: Loss curves are smooth during most of model training, so visible discontinuities stand out as possible conceptual breakthroughs. Studying these breakthroughs enables a deeper understanding of learning dynamics, but only when they are properly identified. This paper argues that similar breakthroughs occur frequently throughout training but they are obscured by a loss metric that collapses all variation into a single scalar. To find these hidden transitions, we introduce POLCA, a method for decomposing changes in loss along arbitrary bases of the low-rank training subspace. We use our method to identify clusters of samples that share similar changes in loss during training, disaggregating the overall loss into that of smaller groups of conceptually similar data. We validate our method on synthetic arithmetic and natural language tasks, showing that POLCA recovers clusters that represent interpretable breakthroughs in the model's capabilities. We demonstrate the promise of these hidden phase transitions as a tool for unsupervised interpretability.

• The paper introduces POLCA to uncover hidden breakthroughs by attributing loss changes to individual data examples and specific parameter update directions.
• It constructs an interpretable basis using the top eigenvectors of the Hessian matrix to capture high-curvature directions in the model’s weight space.
• The method provides granular insights into training dynamics by revealing sudden improvements and temporary degradations in performance on sub-tasks.

The paper "Hidden Breakthroughs in LLM Training" (2506.15872) introduces Projection Oriented Loss Change Allocation (POLCA), a method designed to uncover "hidden breakthroughs" or specific learning events that occur during the training of LLMs. These breakthroughs are often obscured when looking at the aggregate loss curve, which tends to be smooth. POLCA enables a more granular analysis by disaggregating loss changes to individual data examples and decomposing these changes along specific directions in the model's weight space.

The core idea is that while the overall loss might decrease smoothly, individual examples or groups of examples might experience sudden improvements (or even temporary degradations) in performance on specific sub-tasks or concepts. Furthermore, these changes might be tied to parameter updates along particular directions in the high-dimensional weight space.

POLCA Methodology

The POLCA method involves three main steps:

1. Finding an Interpretable Basis:
   • The method requires an orthogonal basis for the low-rank training subspace. The paper proposes constructing this basis iteratively using the top $k$ eigenvectors of the Hessian matrix, projected onto the nullspace of the previously found basis vectors. This is detailed in Algorithm 1.
   • Algorithm 1: Finding the decomposed optimization basis

     Input: Training set X, Model checkpoints {θ_t}_(t=1)^T. B = ∅ (empty matrix) For t = 1 to T: Π_⊥ = I - B (B^T B)^(-1) B^T (Projection onto nullspace of B) H_t = ∇_θ^2 L(X, θ_t) (Hessian at checkpoint θ_t) B_new = top k eigenvectors of Π_⊥ H_t B = [B, B_new] (Append new eigenvectors to basis) Return B

   • To compute Hessian eigenvectors efficiently without explicit Hessian construction, methods like Hessian-vector products (e.g., using libraries like hessian-eigenthings or tools like CoLA) are employed. The basis aims to capture directions of high curvature, which are hypothesized to be conceptually meaningful.
2. Decomposing Loss with POLCA:
   • POLCA modifies Loss Change Allocation (LCA). While LCA attributes loss changes between checkpoints to individual parameter movements along axis-aligned directions for the entire dataset, POLCA extends this by:
     • Allowing arbitrary orthonormal basis vectors $b$ (from the basis found in step 1) instead of just axis-aligned ones.
     • Focusing on the loss change for each individual example $x$, not just the aggregate dataset loss.
     • Using a second-order