The two clocks and the innovation window: When and how generative models learn rules

Abstract: Generative models trained on finite data face a fundamental tension: their score-matching or next-token objective converges to the empirical training distribution rather than the population distribution we seek to learn. Using rule-valid synthetic tasks, we trace this tension across two training timescales: $τ{\mathrm{rule}}$, the step at which generations first become rule-valid, and $τ{\mathrm{mem}}$, the step at which models begin reproducing training samples. Focusing on parity and extending to other binary rules and combinatorial puzzles, we characterize how these two clocks, $τ{\mathrm{rule}}$ and $τ{\mathrm{mem}}$, depend on key aspects of the learning setup. Specifically, we show that $τ{\mathrm{rule}}$ increases with rule complexity and decreases with model capacity, while $τ{\mathrm{mem}}$ is approximately invariant to the rule and scales nearly linearly with dataset size $N$. We define the \emph{innovation window} as the interval $[τ{\mathrm{rule}}, τ{\mathrm{mem}}]$. This window widens with increasing $N$ and narrows with rule complexity, and may vanish entirely when $τ{\mathrm{rule}} \geq τ{\mathrm{mem}}$. The same two-clock structure arises in both diffusion (DiT) and autoregressive (GPT) models, with architecture-dependent offsets. Dissecting the learned score of DiT models reveals a corresponding evolution of the optimization landscapes, where rule-valid samples' basins expand substantially around $τ{\mathrm{rule}}$, while training samples' basins begin to dominate around $τ{\mathrm{mem}}$. Together, these results yield a unified and predictive account of when and how generative models exhibit genuine innovation.

• The paper introduces two distinct training clocks, Trule and Tmem, that delineate the phases of rule learning and memorization in generative models.
• It demonstrates that the innovation window, where models generate rule-compliant yet non-memorized outputs, narrows with increasing rule complexity.
• Empirical evaluations using diffusion and autoregressive transformers validate the scaling behaviors and highlight architectural influences on generalization versus memorization.

Dynamics of Rule Learning and Memorization in Generative Models

Introduction

The study "The two clocks and the innovation window: When and how generative models learn rules" (2605.10019) systematically analyzes how modern generative models (diffusion and autoregressive transformers) acquire rules from finite data. The central concept is the existence of two distinct timescales during training: one for rule acquisition (Trule) and one for memorization of the training set (Tmem). The interval [Trule, Tmem], termed the "innovation window," is where models can generate rule-conforming, non-memorized (i.e., novel) samples. This work carefully delineates the factors shaping these clocks, their manifestations across different architectures and tasks, and the underlying training dynamics.

Experimental Testbed and Evaluation

The experimental setting focuses on synthetic data with explicitly programmable rules, providing ground-truth knowledge of the valid set and the training subset. The core testbed is parity constraints on binary images divided into groups, where the group size (G) determines rule complexity. The study tracks rule accuracy, novelty, and exact memorization at the group and sample levels. Generative architectures considered include EDM-preconditioned diffusion transformers (DiT) and GPT-style autoregressive transformers.

This synthetic regime allows precise quantification of when a model first generates samples conforming to the rule (Trule) and when it begins to exactly replicate training samples (Tmem). Importantly, these two points are rarely coincident, allowing for dynamic separation of generalization and memorization.

Two Clocks: Emergence and Scaling

The emergence of two clocks during training is a core contribution:

• Trule (Rule-Learning Onset): Defined by the first production of high-accuracy, rule-valid generations. Trule increases rapidly with rule complexity (G), shows weak dependence on dataset size (N), and decreases with increased model capacity.
• Tmem (Memorization Onset): Defined when a significant fraction of generations match training data. Tmem is nearly invariant to rule complexity and scales almost linearly with N, with only moderate dependence on model architecture.

The innovation window—the interval [Trule, Tmem]—delimits where generative models create rule-valid, novel samples. For simple rules (small G), this window is large; for complex rules (large G), the window narrows or closes, as rule learning is delayed beyond memorization onset.

Numerical Results

For diffusion transformers trained on D=36, N=4096 samples, the empirical observations include:

• Trule progresses from 10^3 steps for G=2 to >10^5 for G=6, showing exponential scaling with rule complexity.
• Tmem scales nearly linearly with dataset size: Tmem ≈ 35 N^1.14 for DiT-mini, robust across G and capacity.
• For high G, models cannot achieve rule learning before training set memorization, closing the innovation window and resulting in performance dominated by rote memorization.

Mechanistic Dissection

Training Dynamics and Attractor Landscapes

Tracking the evolution of sample states during training, the authors identify:

• A sharp, synchronized transition to rule-validity at Trule, where the attractor basins in the denoising landscape for rule-valid samples expand suddenly.
• A more gradual, stochastic shift at Tmem as attractors for individual training points dominate, shrinking the novelty pool.

Visualization of the diffusion model's learned vector field reveals:

• Before Trule, all Boolean hypercube vertices are attractors.
• At Trule, rule-valid vertices expand their basins.
• At Tmem, basins for training samples swallow neighboring rule-valid, but previously unmemorized, regions—a process corresponding to the onset of memorization.

This behavior is mirrored in loss dynamics: the DSM loss on held-out valid samples diverges from that on training samples at Tmem, but not at Trule, consistent across architectures.

Architectural and Task Generality

The two-clock phenomenon generalizes beyond DiT to autoregressive GPT-style transformers. While the timescales differ—GPT learns and memorizes faster and the innovation window is typically narrower—the qualitative dynamics persist. Notably, for GPT, the transitions between rule learning and memorization are more abrupt, and learning occurs preferentially at deterministic token positions defined by the rule structure.

Beyond parity, similar scaling and phase separations are observed for other constraint types—exact-K and row-K constraints and combinatorial tasks such as Latin squares and Sudoku. Rules involving high-degree parity (multiplicative interactions) remain notably hard, whereas count-based constraints are acquired rapidly and show large innovation windows.

Theoretical Implications

The analysis supports the conclusion that learning high-order, global interactions (e.g., large-G parity) is fundamentally challenging for both diffusion and autoregressive transformers. This aligns with known results in computational learning theory and recent analyses of spectral and architectural biases, which indicate an intrinsic preference for low-degree (low-frequency) components in neural learning dynamics. The tractability of counting-type rules, despite their global nature, underlines that it is the type (degree) of coupling, not simply its globality, that presents the main barrier.

The attractor basin analysis offers a geometric, dynamic perspective on memorization and generalization—clarifying how, over the course of training, novelty is "devoured" as energy landscapes shift from global rules to specific training examples.

Practical Implications and Future Directions

The findings have direct relevance for designing generative models when strict rule adherence or structured generalization is required, such as in scientific modeling and combinatorial design. The vanishing of the innovation window for complex rules indicates that without architectural or training modifications, current models are not equipped to internalize certain abstract or combinatorial rules from data.

Several avenues for improvement are proposed:

• Architectural Interventions: Introducing global broadcast mechanisms or structured multiplicative interactions could enable models to better internalize complex, high-order rules.
• Auxiliary Training Objectives: Tasks that force the model to directly predict or detect parity-like dependencies can scaffold representations more aligned with global constraints.
• Curricular Approaches: Gradually increasing rule complexity (e.g., curriculum learning over G) may permit models to build up to more difficult abstract reasoning tasks.

Conclusion

This work offers a comprehensive, mechanistic account of when and how modern generative models generalize, memorize, and exhibit genuine innovation. By demonstrating the existence and scaling of two fundamental training clocks, it delineates the conditions under which models move from rule learning to memorization. The resultant "innovation window" provides both a diagnostic and a target for future model and algorithmic development. For high-order rules, the closure of this window exposes a central limitation of current generative architectures, marking a clear direction for advancing the theoretical and practical capabilities of AI systems in structured and rule-driven regimes.