---
title: Developmental Interpretability in ML
url: https://www.emergentmind.com/topics/developmental-interpretability
type: topic
---

# Developmental Interpretability in ML

Searching arXiv for the cited papers and adjacent work on developmental interpretability.
Developmental interpretability studies machine learning systems not only as finished artifacts, but as evolving developmental systems. In this framing, the central questions are how representations, circuits, strategies, and meanings arise, compete, stabilize, or disappear during training, post-training, or iterative inference, and how those trajectories bear on capability formation, robustness, and construct validity. The perspective has been articulated most explicitly for large language models, where it is distinguished from behavioral analysis and from mechanistic interpretability of a final checkpoint, but related work also applies it to developmental architectures such as Neural Cellular Automata, to information-theoretic analyses of self-organized biological patterning, to semantically aligned time-series models, and to interpretable reinforcement learning inspired by cognitive development [2508.15841] [2505.08778] [2602.02239].

## 1. Conceptual scope and defining commitments

Developmental interpretability is defined by its temporal perspective. Rather than asking only whether a model exhibits a capability at the end of training, it asks when the capability appears, what has to be in place beforehand, whether its trajectory is staged or nonlinear, and whether apparent competence is stable under controlled perturbations. A recurring claim is that developmental order itself can be evidence about computation: if a model shows a capability only after simpler competencies have emerged, that suggests a more structured internal organization; if performance can be reversed by small lexical or prompt changes, that suggests a heuristic or superficial association rather than a stable underlying representation [2606.28524].

A useful taxonomy separates **behavioral analysis**, **mechanistic interpretability**, and **developmental interpretability**. Behavioral analysis treats the model as a black box and asks what it can do; mechanistic interpretability attempts to reverse-engineer the computations of the final model; developmental interpretability studies the training trajectory across checkpoints and asks how those computations emerge over time. The field’s central thesis is that many important behaviors are **nonlinear, staged, and path-dependent**, so the model’s “ontogeny” matters as much as its final state [2508.15841].

This perspective is broader than a single methodology. In some work, development is training-stage emergence of linguistic and social-reasoning behavior. In other work, it is the temporal preservation of semantic meaning in a latent dynamical system, or the growth of a solution through local iterative dynamics, or the self-organization of reproducible fate patterns under noise. A plausible implication is that “developmental interpretability” names a family of temporally oriented interpretability programs rather than a single formalism.

## 2. Methodological families

Several methodological families recur across the literature. Some instrument training directly; some track geometry or information content over time; some use controlled perturbation and stress-testing; some treat iterative inference itself as a developmental process.

| Method family | What is tracked | Representative use |
|---|---|---|
| Probing and output diagnostics | Linguistic labels, semantic roles, token-class behavior | TRACE tracks feature acquisition over training [2507.03668] |
| Geometry and curvature | Intrinsic dimensionality, Hessian curvature, loss-landscape degeneracy, essential dynamics | Stagewise developmental milestones in transformers [2402.02364] |
| Information-theoretic structure | Entropy, regularity, variation, disentanglement | “Representations as language” over training [2406.02449] |

TRACE is explicitly built for **training-time instrumentation**. It injects lightweight hooks into a standard transformer training loop and logs hidden states, gradients, curvature information, and outputs at regular intervals, giving high temporal resolution. The reported signals include **early syntactic emergence**, **delayed semantic acquisition**, **representational compression**, oscillations in intrinsic dimensionality, and curvature spikes that are interpreted as structural transition markers. The toolkit supports feature probing, intrinsic dimensionality estimation using **TwoNN** and **PCA-based estimators**, **Lanczos-based Hessian approximations**, and output diagnostics, with ABSynth supplying semantic frames, role annotations, POS tags, complexity metadata, and controlled lexical distributions [2507.03668].

A complementary line of work links developmental milestones to the geometry of optimization. “Loss Landscape Degeneracy Drives Stagewise Development in Transformers” argues that training passes through discrete stages marked by changes in **loss-landscape degeneracy**, quantified by the **local learning coefficient (LLC)**, together with changes in function-space trajectory captured by **essential dynamics (ED)**. The core utility of this approach is that stage boundaries can be detected even when loss is smooth or ambiguous, and the resulting milestones line up with changes in internal circuits, embedding structure, attention behavior, and input/output performance [2402.02364].

Another methodological program treats hidden representations as a language in their own right. “Representations as Language” discretizes vectors into bins and defines dimension-wise entropy-based measures for **information**, **variation**, **regularity**, and **disentanglement**. The entropy estimator is
$$
H_{dw}(V) = \frac{1}{|D|}\sum_{d}^{D}\sum_{n}^{N} -p(V_{dn})\log(p(V_{dn})),
$$
with a Miller-Meadow correction. This framework is used to ask when structure arises, how compression changes across phases of training, and which representational organizations predict better out-of-distribution generalization [2406.02449].

The broader review literature places these methods on a ladder from **correlational** to **causal** to **mechanistic** analysis: representational probing, activation patching and causal tracing, circuit analysis, Sparse Autoencoders, and circuit probing. Developmental interpretability does not replace these methods; it reorients them toward trajectories rather than endpoints [2508.15841].

## 3. Developmental trajectories in language models

The most developed body of work concerns transformers and LLMs. One major result is that capabilities often emerge in a staged order rather than as smooth monotonic growth. In controlled synthetic settings, TRACE reports that **core roles** such as AGENT, ACTION, and PATIENT stabilize early, whereas **adjunct or peripheral roles** like LOCATION and DESTINATION emerge more slowly and fluctuate more. Intrinsic dimensionality shows an early drop, then rebound and stabilization, suggesting compression, later expansion, and then settling of the learned representation [2507.03668].

A parallel account comes from stagewise analyses of transformer optimization. In a language model, five stages were identified: **LM1 (0–900 steps)**, learning **bigram statistics**; **LM2 (900–6.5k)**, learning **\(n\)-grams** and making positional embeddings useful; **LM3 (6.5k–8.5k)**, where **previous-token heads** begin forming; **LM4 (8.5k–17k)**, where **induction heads** develop and the model acquires **in-context learning**; and **LM5 (17k–50k)**, a later consolidation stage. In an in-context linear regression transformer, another five-stage pattern was reported, including early task-prior learning, emergence of in-context learning, then overfitting and structural collapse. The interpretive claim is that emergent abilities such as in-context learning are better understood as stage transitions than as mysterious discontinuities [2402.02364].

Review work generalizes this point. It describes **formation and composition of computational circuits**, **biphasic knowledge acquisition**, and the possibility that **In-Context Learning (ICL)** is often **transient** and can later be replaced by **In-Weights Learning (IWL)**. It also frames emergent abilities as **phase transitions** associated with critical thresholds in pre-training loss, and cites abrupt transitions in in-context learning around roughly **2.5–5 billion tokens** in certain transformer studies, with scores jumping from below **0.15 nats** to about **0.4 nats** [2508.15841].

A more focused developmental study examines social reasoning. “Developmental Trajectories of Situation Modeling and Mentalizing in Transformer Language Models” tracks false-belief-task behavior across checkpoints in Olmo2 and Pythia. The completion score is the log-odds
$$
\log_2\left(\frac{p(\text{start})}{p(\text{end})}\right),
$$
with accuracy determined by whether the sign matches the correct belief state. The paper reports that above-chance FBT performance depends jointly on model size and sufficient training volume, emerges relatively late in pretraining, and is most improved by post-training interventions in the **False Belief, Implicit** condition. Crucially, **situation modeling performance generally precedes and exceeds FBT performance**, which the authors interpret as evidence that belief attribution builds on a more basic model of “what happened” [2606.28524].

The same paper emphasizes fragility. Non-factive verbs such as “thinks” can increase false-belief attributions even in **True Belief** settings, and Olmo2 13B’s answers about an **Antagonist** agent are influenced both by the Target agent’s knowledge state and by the presence of non-factive verbs. This indicates contamination across agents and weakens any simple claim that the model has a clean, unified theory-of-mind-like representation [2606.28524].

Developmental regularity can itself be a target of interpretation. “Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research” proposes five axes of correspondence—**functional**, **positional**, **developmental**, **relational**, and **configurational**—for asking when mechanistic claims should generalize across model instances. In a study of **1-back attention heads** across random seeds of the Pythia models, selective previous-token attention tends to emerge around **\(10^3\) training steps**, corresponding to roughly **2B tokens**. A generalized additive model on maximum 1-back attention achieved \(R^2 = 0.95\), supporting the claim that developmental trajectories are highly consistent even when positional correspondence is much weaker. Larger models showed **earlier onset**, **steeper rise**, and **higher maximum peak** [2509.22831].

The information-theoretic view yields a related two-phase account. In semantic parsing transformers, Phase 1 is an **In-Distribution Learning** phase, where token and POS regularity and disentanglement rise as the model rapidly learns the training task. Phase 2 is a **Robustness to Noise** phase, beginning after training accuracy has largely saturated, in which the representational space compresses, bigram regularity increases, token regularity falls, and out-of-distribution generalization begins to improve only after a long delay. This suggests that generalization is linked not merely to fitting the task but to later reorganization of the representational code [2406.02449].

## 4. Developmental architectures and self-organized systems

Developmental interpretability is not confined to transformers. “ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus” frames ARC solving as a developmental process in which a system evolves over time from a simple initial state into a task-specific structure. The method trains a fresh NCA per problem using the few provided examples, explicitly as **test-time training / program synthesis**: the system “writes” a task-specific NCA by gradient descent, then runs it on the test input. The interpretive claim is that NCAs are suited to this because they operate through **local, iterative, emergent dynamics**, so intermediate states can be viewed as developmental stages rather than as opaque hidden activations [2505.08778].

The architectural idea with the clearest interpretability relevance is **EngramNCA’s dual-state cell design**. Each cell has a **public state** and a **private state**. EngramNCA is described as an ensemble of **GeneCA**, which grows primitive morphologies from a seed cell and creates immutable private memory encodings, and **GenePropCA**, which propagates and activates those primitives and modulates private memory without changing visible state. This explicit separation between visible pattern and latent developmental information is presented as making the model more interpretable than a standard NCA, because behavior can be conceptually attributed to morphological growth, memory storage, and memory propagation or regulation [2505.08778].

An information-theoretic counterpart appears in the study of self-organized developmental systems. “Information content and optimization of self-organized developmental systems” introduces a utility
$$
U = S_\mathrm{pat} - S_\mathrm{rep},
$$
where \(S_\mathrm{rep}\) is the reproducibility entropy of the developmental ensemble and \(S_\mathrm{pat}\) is the pooled patterning entropy. The paper then decomposes
$$
U=\mathrm{PI}+\mathrm{CI},
$$
into **positional information (PI)** and **correlational information (CI)**. The interpretive importance of this decomposition is that developmental information is not restricted to a local mapping between position and fate; it can also reside in nonlocal correlations among cells. A perfectly alternating pattern with random spatial shifts can therefore have low PI but high CI [2312.05895].

This framework is normative as well as descriptive. By optimizing \(U\), the paper evaluates which circuit designs produce robust, reproducible patterns under intrinsic and extrinsic noise, and demonstrates the framework on lateral inhibition, cell-type proportioning and sorting, reaction-diffusion systems, and an expander module that acts as a global sensor of amplitude variation [2312.05895].

A cognitively oriented variant appears in interpretable reinforcement learning inspired by Piaget. The **AO2 (Abstraction via Options of Options)** framework represents knowledge as a **tree of schemas/options** whose abstraction level is “determined by the system itself.” Learning alternates between **assimilation**, updating weights of an activated schema, and **accommodation**, reforming the schema tree when the current schema is insufficient. The result is an explicitly traceable developmental process in which structure is constructed rather than hard-coded, decisions follow readable schema/action decomposition, and abstraction is self-organized rather than externally fixed [2102.00572].

## 5. Semantic alignment and domain-specific evaluation

A distinct strand argues that interpretability in dynamic systems should be formulated as preservation of meaning over time. “Interpretability in Deep Time Series Models Demands Semantic Alignment” distinguishes **structural opacity** from **semantic opacity** and argues that deep time-series interpretability should pursue **semantic alignment**: predictions should be expressed in terms of variables meaningful to the end user, mediated by spatial and temporal mechanisms that admit user-dependent constraints. The central formal requirements are
$$
P(U_t = C^U_t \mid X_{\le t}) = 1 \quad \text{a.s.}, \, \forall t
$$
for encoder alignment to instantaneous concepts, and
$$
P(Z_{t+1} = C^Z_{t+1} \mid X_{\le t}) = 1 \quad \text{a.s.}, \, \forall t
$$
for propagation alignment to future dynamic concepts. Mechanism alignment is expressed as
$$
P(V \mid V') \in \mathcal{M}^{(h)}_{V \mid V'}.
$$
The developmental insight is that interpretability is not enough at one step; meaning must remain aligned under temporal evolution, because otherwise latent variables can silently repurpose themselves over time [2602.02239].

This concern with temporally extended meaning has an educational analogue. “Interpretability Framework for LLMs in Undergraduate Calculus” argues that final-answer accuracy is insufficient because a model may be **syntactically fluent yet conceptually flawed**, sensitive to prompt phrasing, or pedagogically misaligned. The proposed framework combines **reasoning flow analysis** with **sensitivity ablation / input salience analysis**. Solutions are decomposed into reasoning steps annotated by **mathematical operation**, **conceptual tag**, and **cognitive complexity**, then organized into a directed reasoning graph; prompt elements are removed or perturbed and outputs compared using **TF-IDF weighted cosine similarity** and **Normalized edit distance (Levenshtein distance)** [2510.17910].

The framework operationalizes **Robustness**, **Impact Factor**, **Step Count**, **Complexity**, **Reasoning Pattern Trace**, and **Phrase Sensitivity**, with phrase sensitivity defined as the maximum divergence across all ablations and robustness as the average semantic similarity across ablated responses. For a gradient/directional derivative problem, the reported reasoning pattern is **substitution → differentiation → differentiation → solving → differentiation → evaluation → solving**, with 7 total steps and complexity score 23. The broader claim is that interpretability can be a developmental lens on reasoning sequences: not only whether the model is correct, but whether its reasoning path develops in a way aligned with valid mathematical progression [2510.17910].

Taken together, these works suggest a broader notion of developmental interpretability in which temporal preservation of semantic content and temporally legible reasoning trajectories are as important as tracing internal circuits.

## 6. Limitations, controversies, and open directions

The literature is explicit that developmental interpretability is not yet a complete theory. One limitation concerns the status of measurement. Probing can show that information is decodable, but not that it is causally used; this motivates the turn toward activation patching, causal tracing, circuit analysis, and stress-testing, but the review literature notes that automated and scalable mechanistic discovery remains difficult [2508.15841].

Another limitation is theoretical grounding. In the loss-landscape work, the LLC is formally defined at local minima, yet it is applied at intermediate checkpoints that are not necessarily minima; the authors describe the approach as empirically successful but not fully justified theoretically. The same paper also notes that the relationship between the Bayesian, asymptotic singular-learning-theory story and real SGD dynamics is unclear, that PCA-based ED can mislead, and that later stages are not always well characterized [2402.02364].

A separate issue is the gap between developmental visibility and mechanistic explanation. ARC-NCA argues that developmental NCA dynamics are interpretable because solutions emerge gradually, hidden/public state separation provides a memory-bearing substrate, and qualitative examples reveal local mechanisms such as growth, filling, propagation, and correction. At the same time, the paper explicitly states that it does **not** provide deep mechanistic interpretability analyses in the style of probing, circuit analysis, or formal attribution; its interpretability is mainly **developmental and qualitative** [2505.08778].

Construct validity remains a central controversy. In social-reasoning work, apparently successful benchmark behavior can be driven by lexical cues, and trivial perturbations can reverse apparent theory-of-mind performance. Review work on emergent abilities also notes a controversy over whether emergence is partly a measurement artifact caused by nonlinear metrics, even though it argues that sharp transitions remain real under more continuous measures [2606.28524] [2508.15841].

Generalization of interpretability findings is another unresolved problem. The proposed axes of **functional**, **positional**, **developmental**, **relational**, and **configurational** correspondence make explicit that a mechanism can generalize in one sense but not another. The empirical result that developmental regularity can be strong while positional regularity is weak suggests that comparative interpretability will require a vocabulary richer than “same circuit” or “different circuit” [2509.22831].

Finally, semantically aligned approaches emphasize that perfect alignment is difficult to verify “in the wild,” especially when user concepts are only partially observed or require a translation map \(\tau : \mathcal{Z} \to \mathcal{C}\). The open agenda therefore includes standardized benchmarks and metrics for interpretability quality and faithfulness, interpretability across the full training lifecycle including pretraining, instruction tuning, and RLHF, extension to multimodal and agentic systems, and development of human-in-the-loop workflows. A plausible implication is that the field’s long-term success will depend on integrating developmental, mechanistic, information-theoretic, and semantically aligned views rather than treating them as competing paradigms [2602.02239] [2508.15841].

Source: https://www.emergentmind.com/topics/developmental-interpretability