Developmental Interpretability in ML

• Developmental interpretability is a framework that examines how machine learning systems evolve, tracing the emergence and stabilization of capabilities during training.
• It integrates methods such as probing, geometry, and information-theoretic analysis to monitor dynamic changes in representations and performance.
• Applications span language models, neural cellular automata, and reinforcement learning, emphasizing temporal trajectories over static endpoint analysis.

Searching arXiv for the cited papers and adjacent work on developmental interpretability. Developmental interpretability studies machine learning systems not only as finished artifacts, but as evolving developmental systems. In this framing, the central questions are how representations, circuits, strategies, and meanings arise, compete, stabilize, or disappear during training, post-training, or iterative inference, and how those trajectories bear on capability formation, robustness, and construct validity. The perspective has been articulated most explicitly for LLMs, where it is distinguished from behavioral analysis and from mechanistic interpretability of a final checkpoint, but related work also applies it to developmental architectures such as Neural Cellular Automata, to information-theoretic analyses of self-organized biological patterning, to semantically aligned time-series models, and to interpretable reinforcement learning inspired by cognitive development (Kendiukhov, 19 Aug 2025, Guichard et al., 13 May 2025, Felice et al., 2 Feb 2026).

1. Conceptual scope and defining commitments

Developmental interpretability is defined by its temporal perspective. Rather than asking only whether a model exhibits a capability at the end of training, it asks when the capability appears, what has to be in place beforehand, whether its trajectory is staged or nonlinear, and whether apparent competence is stable under controlled perturbations. A recurring claim is that developmental order itself can be evidence about computation: if a model shows a capability only after simpler competencies have emerged, that suggests a more structured internal organization; if performance can be reversed by small lexical or prompt changes, that suggests a heuristic or superficial association rather than a stable underlying representation (Rivière et al., 26 Jun 2026).

A useful taxonomy separates behavioral analysis, mechanistic interpretability, and developmental interpretability. Behavioral analysis treats the model as a black box and asks what it can do; mechanistic interpretability attempts to reverse-engineer the computations of the final model; developmental interpretability studies the training trajectory across checkpoints and asks how those computations emerge over time. The field’s central thesis is that many important behaviors are nonlinear, staged, and path-dependent, so the model’s “ontogeny” matters as much as its final state (Kendiukhov, 19 Aug 2025).

This perspective is broader than a single methodology. In some work, development is training-stage emergence of linguistic and social-reasoning behavior. In other work, it is the temporal preservation of semantic meaning in a latent dynamical system, or the growth of a solution through local iterative dynamics, or the self-organization of reproducible fate patterns under noise. A plausible implication is that “developmental interpretability” names a family of temporally oriented interpretability programs rather than a single formalism.

2. Methodological families

Several methodological families recur across the literature. Some instrument training directly; some track geometry or information content over time; some use controlled perturbation and stress-testing; some treat iterative inference itself as a developmental process.

Method family	What is tracked	Representative use
Probing and output diagnostics	Linguistic labels, semantic roles, token-class behavior	TRACE tracks feature acquisition over training (Aljaafari et al., 4 Jul 2025)
Geometry and curvature	Intrinsic dimensionality, Hessian curvature, loss-landscape degeneracy, essential dynamics	Stagewise developmental milestones in transformers (Hoogland et al., 2024)
Information-theoretic structure	Entropy, regularity, variation, disentanglement	“Representations as language” over training (Conklin et al., 2024)

TRACE is explicitly built for training-time instrumentation. It injects lightweight hooks into a standard transformer training loop and logs hidden states, gradients, curvature information, and outputs at regular intervals, giving high temporal resolution. The reported signals include early syntactic emergence, delayed semantic acquisition, representational compression, oscillations in intrinsic dimensionality, and curvature spikes that are interpreted as structural transition markers. The toolkit supports feature probing, intrinsic dimensionality estimation using TwoNN and PCA-based estimators, Lanczos-based Hessian approximations, and output diagnostics, with ABSynth supplying semantic frames, role annotations, POS tags, complexity metadata, and controlled lexical distributions (Aljaafari et al., 4 Jul 2025).

A complementary line of work links developmental milestones to the geometry of optimization. “Loss Landscape Degeneracy Drives Stagewise Development in Transformers” argues that training passes through discrete stages marked by changes in loss-landscape degeneracy, quantified by the local learning coefficient (LLC), together with changes in function-space trajectory captured by essential dynamics (ED). The core utility of this approach is that stage boundaries can be detected even when loss is smooth or ambiguous, and the resulting milestones line up with changes in internal circuits, embedding structure, attention behavior, and input/output performance (Hoogland et al., 2024).

Another methodological program treats hidden representations as a language in their own right. “Representations as Language” discretizes vectors into bins and defines dimension-wise entropy-based measures for information, variation, regularity, and disentanglement. The entropy estimator is

$$H_{dw}(V) = \frac{1}{|D|}\sum_{d}^{D}\sum_{n}^{N} -p(V_{dn})\log(p(V_{dn})),$$

with a Miller-Meadow correction. This framework is used to ask when structure arises, how compression changes across phases of training, and which representational organizations predict better out-of-distribution generalization (Conklin et al., 2024).

The broader review literature places these methods on a ladder from correlational to causal to mechanistic analysis: representational probing, activation patching and causal tracing, circuit analysis, Sparse Autoencoders, and circuit probing. Developmental interpretability does not replace these methods; it reorients them toward trajectories rather than endpoints (Kendiukhov, 19 Aug 2025).

3. Developmental trajectories in LLMs

The most developed body of work concerns transformers and LLMs. One major result is that capabilities often emerge in a staged order rather than as smooth monotonic growth. In controlled synthetic settings, TRACE reports that core roles such as AGENT, ACTION, and PATIENT stabilize early, whereas adjunct or peripheral roles like LOCATION and DESTINATION emerge more slowly and fluctuate more. Intrinsic dimensionality shows an early drop, then rebound and stabilization, suggesting compression, later expansion, and then settling of the learned representation (Aljaafari et al., 4 Jul 2025).

A parallel account comes from stagewise analyses of transformer optimization. In a LLM, five stages were identified: LM1 (0–900 steps), learning bigram statistics; LM2 (900–6.5k), learning $n$-grams and making positional embeddings useful; LM3 (6.5k–8.5k), where previous-token heads begin forming; LM4 (8.5k–17k), where induction heads develop and the model acquires in-context learning; and LM5 (17k–50k), a later consolidation stage. In an in-context linear regression transformer, another five-stage pattern was reported, including early task-prior learning, emergence of in-context learning, then overfitting and structural collapse. The interpretive claim is that emergent abilities such as in-context learning are better understood as stage transitions than as mysterious discontinuities (Hoogland et al., 2024).

Review work generalizes this point. It describes formation and composition of computational circuits, biphasic knowledge acquisition, and the possibility that In-Context Learning (ICL) is often transient and can later be replaced by In-Weights Learning (IWL). It also frames emergent abilities as phase transitions associated with critical thresholds in pre-training loss, and cites abrupt transitions in in-context learning around roughly 2.5–5 billion tokens in certain transformer studies, with scores jumping from below 0.15 nats to about 0.4 nats (Kendiukhov, 19 Aug 2025).

A more focused developmental study examines social reasoning. “Developmental Trajectories of Situation Modeling and Mentalizing in Transformer LLMs” tracks false-belief-task behavior across checkpoints in Olmo2 and Pythia. The completion score is the log-odds

$$\log_2\left(\frac{p(\text{start})}{p(\text{end})}\right),$$

with accuracy determined by whether the sign matches the correct belief state. The paper reports that above-chance FBT performance depends jointly on model size and sufficient training volume, emerges relatively late in pretraining, and is most improved by post-training interventions in the False Belief, Implicit condition. Crucially, situation modeling performance generally precedes and exceeds FBT performance, which the authors interpret as evidence that belief attribution builds on a more basic model of “what happened” (Rivière et al., 26 Jun 2026).

The same paper emphasizes fragility. Non-factive verbs such as “thinks” can increase false-belief attributions even in True Belief settings, and Olmo2 13B’s answers about an Antagonist agent are influenced both by the Target agent’s knowledge state and by the presence of non-factive verbs. This indicates contamination across agents and weakens any simple claim that the model has a clean, unified theory-of-mind-like representation (Rivière et al., 26 Jun 2026).

Developmental regularity can itself be a target of interpretation. “Toward a Theory of Generalizability in LLM Mechanistic Interpretability Research” proposes five axes of correspondence—functional, positional, developmental, relational, and configurational—for asking when mechanistic claims should generalize across model instances. In a study of 1-back attention heads across random seeds of the Pythia models, selective previous-token attention tends to emerge around $10^3$ training steps, corresponding to roughly 2B tokens. A generalized additive model on maximum 1-back attention achieved $R^2 = 0.95$, supporting the claim that developmental trajectories are highly consistent even when positional correspondence is much weaker. Larger models showed earlier onset, steeper rise, and higher maximum peak (Trott, 26 Sep 2025).

The information-theoretic view yields a related two-phase account. In semantic parsing transformers, Phase 1 is an In-Distribution Learning phase, where token and POS regularity and disentanglement rise as the model rapidly learns the training task. Phase 2 is a Robustness to Noise phase, beginning after training accuracy has largely saturated, in which the representational space compresses, bigram regularity increases, token regularity falls, and out-of-distribution generalization begins to improve only after a long delay. This suggests that generalization is linked not merely to fitting the task but to later reorganization of the representational code (Conklin et al., 2024).

4. Developmental architectures and self-organized systems

Developmental interpretability is not confined to transformers. “ARC-NCA: Towards Developmental Solutions to the Abstraction and Reasoning Corpus” frames ARC solving as a developmental process in which a system evolves over time from a simple initial state into a task-specific structure. The method trains a fresh NCA per problem using the few provided examples, explicitly as test-time training / program synthesis: the system “writes” a task-specific NCA by gradient descent, then runs it on the test input. The interpretive claim is that NCAs are suited to this because they operate through local, iterative, emergent dynamics, so intermediate states can be viewed as developmental stages rather than as opaque hidden activations (Guichard et al., 13 May 2025).

The architectural idea with the clearest interpretability relevance is EngramNCA’s dual-state cell design. Each cell has a public state and a private state. EngramNCA is described as an ensemble of GeneCA, which grows primitive morphologies from a seed cell and creates immutable private memory encodings, and GenePropCA, which propagates and activates those primitives and modulates private memory without changing visible state. This explicit separation between visible pattern and latent developmental information is presented as making the model more interpretable than a standard NCA, because behavior can be conceptually attributed to morphological growth, memory storage, and memory propagation or regulation (Guichard et al., 13 May 2025).

An information-theoretic counterpart appears in the study of self-organized developmental systems. “Information content and optimization of self-organized developmental systems” introduces a utility

$U = S_\mathrm{pat} - S_\mathrm{rep},$

where $S_\mathrm{rep}$ is the reproducibility entropy of the developmental ensemble and $S_\mathrm{pat}$ is the pooled patterning entropy. The paper then decomposes

$U=\mathrm{PI}+\mathrm{CI},$

into positional information (PI) and correlational information (CI). The interpretive importance of this decomposition is that developmental information is not restricted to a local mapping between position and fate; it can also reside in nonlocal correlations among cells. A perfectly alternating pattern with random spatial shifts can therefore have low PI but high CI (Brückner et al., 2023).

This framework is normative as well as descriptive. By optimizing $U$, the paper evaluates which circuit designs produce robust, reproducible patterns under intrinsic and extrinsic noise, and demonstrates the framework on lateral inhibition, cell-type proportioning and sorting, reaction-diffusion systems, and an expander module that acts as a global sensor of amplitude variation (Brückner et al., 2023).

A cognitively oriented variant appears in interpretable reinforcement learning inspired by Piaget. The AO2 (Abstraction via Options of Options) framework represents knowledge as a tree of schemas/options whose abstraction level is “determined by the system itself.” Learning alternates between assimilation, updating weights of an activated schema, and accommodation, reforming the schema tree when the current schema is insufficient. The result is an explicitly traceable developmental process in which structure is constructed rather than hard-coded, decisions follow readable schema/action decomposition, and abstraction is self-organized rather than externally fixed (Hakimzadeh et al., 2021).

5. Semantic alignment and domain-specific evaluation

A distinct strand argues that interpretability in dynamic systems should be formulated as preservation of meaning over time. “Interpretability in Deep Time Series Models Demands Semantic Alignment” distinguishes structural opacity from semantic opacity and argues that deep time-series interpretability should pursue semantic alignment: predictions should be expressed in terms of variables meaningful to the end user, mediated by spatial and temporal mechanisms that admit user-dependent constraints. The central formal requirements are

$n$0

for encoder alignment to instantaneous concepts, and

$n$1

for propagation alignment to future dynamic concepts. Mechanism alignment is expressed as

$n$2

The developmental insight is that interpretability is not enough at one step; meaning must remain aligned under temporal evolution, because otherwise latent variables can silently repurpose themselves over time (Felice et al., 2 Feb 2026).

This concern with temporally extended meaning has an educational analogue. “Interpretability Framework for LLMs in Undergraduate Calculus” argues that final-answer accuracy is insufficient because a model may be syntactically fluent yet conceptually flawed, sensitive to prompt phrasing, or pedagogically misaligned. The proposed framework combines reasoning flow analysis with sensitivity ablation / input salience analysis. Solutions are decomposed into reasoning steps annotated by mathematical operation, conceptual tag, and cognitive complexity, then organized into a directed reasoning graph; prompt elements are removed or perturbed and outputs compared using TF-IDF weighted cosine similarity and Normalized edit distance (Levenshtein distance) (Dakshit et al., 19 Oct 2025).

The framework operationalizes Robustness, Impact Factor, Step Count, Complexity, Reasoning Pattern Trace, and Phrase Sensitivity, with phrase sensitivity defined as the maximum divergence across all ablations and robustness as the average semantic similarity across ablated responses. For a gradient/directional derivative problem, the reported reasoning pattern is substitution → differentiation → differentiation → solving → differentiation → evaluation → solving, with 7 total steps and complexity score 23. The broader claim is that interpretability can be a developmental lens on reasoning sequences: not only whether the model is correct, but whether its reasoning path develops in a way aligned with valid mathematical progression (Dakshit et al., 19 Oct 2025).

Taken together, these works suggest a broader notion of developmental interpretability in which temporal preservation of semantic content and temporally legible reasoning trajectories are as important as tracing internal circuits.

6. Limitations, controversies, and open directions

The literature is explicit that developmental interpretability is not yet a complete theory. One limitation concerns the status of measurement. Probing can show that information is decodable, but not that it is causally used; this motivates the turn toward activation patching, causal tracing, circuit analysis, and stress-testing, but the review literature notes that automated and scalable mechanistic discovery remains difficult (Kendiukhov, 19 Aug 2025).

Another limitation is theoretical grounding. In the loss-landscape work, the LLC is formally defined at local minima, yet it is applied at intermediate checkpoints that are not necessarily minima; the authors describe the approach as empirically successful but not fully justified theoretically. The same paper also notes that the relationship between the Bayesian, asymptotic singular-learning-theory story and real SGD dynamics is unclear, that PCA-based ED can mislead, and that later stages are not always well characterized (Hoogland et al., 2024).

A separate issue is the gap between developmental visibility and mechanistic explanation. ARC-NCA argues that developmental NCA dynamics are interpretable because solutions emerge gradually, hidden/public state separation provides a memory-bearing substrate, and qualitative examples reveal local mechanisms such as growth, filling, propagation, and correction. At the same time, the paper explicitly states that it does not provide deep mechanistic interpretability analyses in the style of probing, circuit analysis, or formal attribution; its interpretability is mainly developmental and qualitative (Guichard et al., 13 May 2025).

Construct validity remains a central controversy. In social-reasoning work, apparently successful benchmark behavior can be driven by lexical cues, and trivial perturbations can reverse apparent theory-of-mind performance. Review work on emergent abilities also notes a controversy over whether emergence is partly a measurement artifact caused by nonlinear metrics, even though it argues that sharp transitions remain real under more continuous measures (Rivière et al., 26 Jun 2026, Kendiukhov, 19 Aug 2025).

Generalization of interpretability findings is another unresolved problem. The proposed axes of functional, positional, developmental, relational, and configurational correspondence make explicit that a mechanism can generalize in one sense but not another. The empirical result that developmental regularity can be strong while positional regularity is weak suggests that comparative interpretability will require a vocabulary richer than “same circuit” or “different circuit” (Trott, 26 Sep 2025).

Finally, semantically aligned approaches emphasize that perfect alignment is difficult to verify “in the wild,” especially when user concepts are only partially observed or require a translation map $n$3. The open agenda therefore includes standardized benchmarks and metrics for interpretability quality and faithfulness, interpretability across the full training lifecycle including pretraining, instruction tuning, and RLHF, extension to multimodal and agentic systems, and development of human-in-the-loop workflows. A plausible implication is that the field’s long-term success will depend on integrating developmental, mechanistic, information-theoretic, and semantically aligned views rather than treating them as competing paradigms (Felice et al., 2 Feb 2026, Kendiukhov, 19 Aug 2025).