Circuit Synchronization Precedes Generalization: Causal Evidence from Fourier Structure in Grokking Transformers

Abstract: Grokking -- where a transformer on modular arithmetic suddenly transitions from near-chance to near-perfect validation accuracy -- is attributed to a Fourier circuit, but its timing, causal structure, and controllability remain poorly understood. We introduce the Frequency Synchronization Degree (FSD), a normalised, permutation-tested metric for Fourier circuit synchronisation requiring no prior circuit knowledge. Across nine modular addition configurations (primes p in {53, 71, 97, 113, 131}, three seeds), FSD synchronises 500-3,000 steps before grokking (mean lead +1,722 steps; all nine positive, sign-test p~0.004), and precedes a restricted-logit loss baseline (Nanda et al.'s excluded loss) in all nine cases, making it the earliest available predictor. We provide direct causal evidence that the inter-phase gap is a regularisation phenomenon: forking training at the FSD-ceiling step and varying weight decay lambda produces strictly monotone earlier grokking, with Delta_t proportional to 1/lambda. This law replicates across three primes (p in {53,97,131}; R^2=1.00 and R^2=0.99 for two clean cases), captured as Delta_t ~ C/lambda, consistent with (1/lambda)*log(||W_mem||/tau). Architecture ablations show an attention-only model groks with a strong FSD precursor; an MLP-only model never groks; a single-layer model's FSD lags, confirming the precursor is a multi-block circuit property.

• The paper demonstrates that Fourier-based circuit synchronization precedes transformer generalization, validated by the novel FSD metric across multiple modular addition tasks.
• It establishes a causal link between circuit formation and weight decay, revealing a two-phase grokking dynamic with early circuit synchronization leading to memorization pruning.
• The study highlights the necessity of attention mechanisms over MLP-only architectures, with scaling laws confirming an inverse relationship between weight decay and generalization timing.

Circuit Synchronization Precedes Generalization in Grokking Transformers

Overview

The paper "Circuit Synchronization Precedes Generalization: Causal Evidence from Fourier Structure in Grokking Transformers" (2606.12966) presents a rigorous mechanistic analysis of the grokking phenomenon observed in transformers trained on modular arithmetic tasks. Grokking describes a two-phase learning process where a model first memorizes the training data before abruptly transitioning to generalization, demonstrating near-perfect validation accuracy. This work formalizes and empirically validates the causal progression of circuit synchronization—specifically Fourier-based algorithmic circuits—relative to the generalization phase. The author introduces the Frequency Synchronization Degree (FSD) as a circuit-agnostic, statistically robust metric that predicts grokking well before traditional progress measures and establishes a controlled causal link between circuit formation and weight decay-driven regularization.

FSD as a Leading Indicator of Grokking

FSD is defined as a normalized, permutation-tested metric quantifying the synchronization of MLP neurons around dominant Fourier frequencies. Across nine modular addition configurations, FSD reliably reaches post-grokking levels 500–3,000 steps prior to the leap in validation accuracy, with a mean lead of +1,722 steps (exact binomial test, $p \approx 0.004$). This precursor window is robust across multiple primes and seeds, providing statistical significance and strong predictive utility.

Figure 1: FSD lead (orange) vs. restricted-logit loss lead (blue) across all nine addition experiments; FSD consistently synchronizes before grokking, outperforming ExLoss.

The FSD metric establishes its superiority over mechanistically-informed progress measures such as Nanda et al.’s restricted-logit loss (excluded loss). In every configuration, FSD synchronizes before ExLoss, and for cases requiring multi-frequency representations (e.g., $p=113$), ExLoss fails to predict generalization at all. This confirms FSD’s operation-agnostic detection capability and its role as the earliest available circuit formation predictor.

Two-Phase Grokking Dynamics

Empirical trajectories reveal a clear separation between circuit formation (Phase 1) and circuit liberation (Phase 2), where memorized weights are progressively pruned by regularization.

Figure 2: Two-phase grokking for addition mod~97; Fourier rank collapse precedes grokking by 1,500 steps, defining the circuit-liberation period.

During Phase 1, the Fourier rank of MLP block 0 neurons collapses monotonically from 6 to 1, indicating convergent compression to single-sinusoid encoding. Phase 2 maintains rank-1 structure while the Frobenius weight norm decays continuously; validation accuracy remains low until memorization is completely pruned.

Figure 3: Memorization trajectory for addition mod~97; the generalization gap is sustained throughout Phase 2 and collapses instantaneously at grokking, confirming active memorization pruning.

Direct causal validation is achieved through weight-decay interventions at the FSD ceiling. Training forks at circuit-complete checkpoints and varies the weight decay $\lambda$. The model generalizes strictly earlier with increasing $\lambda$ and follows a precise inverse-$\lambda$ scaling law: $\Delta t = C/\lambda$.

Figure 4: Weight-decay intervention; validation accuracy accelerates monotonically with increasing $\lambda$, confirming the causal role of regularization.

Architecture Ablation and Operation Contrast

The necessity and sufficiency of architectural components are clarified via ablation and operation analysis.

• Architecture ablation: Attention-only 2-layer transformers exhibit the strongest FSD precursor and grok with early synchronization, while MLP-only models never grok, confirming the necessity of attention for the Fourier algorithm.
• Operation contrast: For modular addition and subtraction (algebraically analogous), FSD synchronizes before generalization. For multiplication, FSD lags grokking substantially, confirming its selectivity for the Fourier-structured solution.

  Figure 5: Architecture ablation; attention-only models grok with maximal FSD precursor, whereas 1-layer models reveal lagging FSD.

Cross-Prime Scaling Laws

Weight-decay interventions replicate cleanly across primes ($p=53, 97, 131$). The empirical $\Delta t = C_p/\lambda$ scaling law holds for each, with $R^2 > 0.99$ in most cases, affirming the generality of the regularization-driven liberation phase.

Figure 6: Cross-prime Phase-2 timing; monotonic acceleration with $p=113$0 and inverse-$p=113$1 law fits across primes.

Symbolic Extraction and Causal Hierarchy

FourierKAN symbolic regression confirms that post-grokking neuron activations are expressible as single-sinusoid superpositions, consistently matching analytical DFT decompositions with high $p=113$2 and neuron agreement, and correctly identifying multi-frequency circuits.

Zero-ablation reveals a clear causal hierarchy: block 1 attention is the primary computational pathway, ablation of which eliminates generalization. Block 0 MLP is an early organizational precursor, predictive but not load-bearing at the time of generalization.

Implications and Future Directions

The findings substantiate a mechanistic two-phase account for grokking in algorithmic tasks: circuit formation precedes and enables generalization, and the pruning of memorization can be causally controlled via regularization rate. The FSD metric, as a circuit-agnostic leading indicator, offers practical utility for early diagnosis and targeted training interventions. Scaling laws elucidate the dependency of grokking timing on memorization and regularization parameters, suggesting strategies for accelerating generalization through dynamic weight decay scheduling.

Theoretical implications extend to mechanistic interpretability, phase transition theory in representation learning, and spectral bias research. Practically, this work enables more efficient and controllable training protocols for algorithmic models and invites extensions to larger primes, deeper architectures, and broader tasks—including permutation groups, finite field arithmetic, and those covered in Omnigrok [liu2023omnigrok].

Conclusion

This work delivers quantitative, causal evidence that circuit synchronization as measured by FSD precedes and predicts the onset of generalization in grokking transformers, independently of traditional progress measures. The two-phase grokking theory, supported by cross-prime scaling laws and weight-decay interventions, delineates computation-complete circuit formation followed by regularization-driven memorization removal. Attention layers, not block 0 MLP, carry the load at grokking, and the operation-specific nature of the precursor clarifies the diversity of algorithmic solutions. The methodology and scaling relationships have immediate applications for mechanistic interpretability and controlled generalization in AI systems.