---
title: 'Grokfast Algorithm: Accelerating Neural Grokking'
url: https://www.emergentmind.com/topics/grokfast-algorithm
type: topic
---

# Grokfast Algorithm: Accelerating Neural Grokking

The Grokfast Algorithm, also known as Egalitarian Gradient Descent (EGD), is a family of optimization modifications designed to dramatically accelerate the “grokking” phenomenon in overparameterized neural networks. Grokking is characterized by an extended delay between the achievement of perfect training accuracy and the abrupt onset of high generalization accuracy, often resulting in significant computational cost and practical inefficiency. The Grokfast methodology addresses this by spectrally decomposing the optimization process and amplifying the slow, generalization-inducing gradient components, or by preconditioning gradients so that all principal directions evolve at equal rates. Robust empirical evidence demonstrates consistent order-of-magnitude speedups in a variety of tasks, including modular arithmetic, parity problems, image recognition, language modeling, and graph learning [2405.20233][2510.04930].

## 1. Grokking Phenomenon and Plateau Dynamics

Grokking refers to the sharply delayed transition from overfitting to generalization during training. Typically, a model will achieve near-zero training loss rapidly, while validation/test accuracy remains at chance for a protracted plateau, sometimes spanning thousands of training iterations. This delay is especially pronounced in algorithmic tasks such as modular arithmetic and sparse parity, where the model eventually "discovers" a correct hypothesis and test accuracy spikes suddenly [2405.20233][2510.04930].

An analysis of gradient descent dynamics reveals that this plateau arises from highly asymmetric convergence speeds along different spectral modes of the gradient covariance. Fast directions—corresponding to large singular values—are quickly optimized, leading to memorization. Slow directions, governed by small singular values, are responsible for generalization but progress orders of magnitude more slowly, producing the empirical plateau [2510.04930].

## 2. Spectral Decomposition of Gradient Trajectories

The Grokfast approach conceptualizes the sequence of parameter gradients $\{g_t\}$ as a discrete-time random signal. By applying spectral decomposition, the gradient signal is separated into:

- **High-frequency (fast) components**: Drive rapid fitting and memorization.
- **Low-frequency (slow) components**: Underpin the slower emergence of generalization.

Formally, for each parameter coordinate, $g_t = g_t^{(\text{slow})} + g_t^{(\text{fast})}$, where $g_t^{(\text{slow})} = (h * g)_t$ for some low-pass filter $h$. Boosting $g_t^{(\text{slow})}$ accelerates movement along generalization-relevant directions, effectively shrinking the grokking plateau [2405.20233].

Alternatively, a matrix-level spectral normalization (as in EGD) aligns all singular directions of the gradient update, ensuring uniform progress across modes [2510.04930].

## 3. Algorithmic Modifications and Implementation

Two main strategies have been established for operationalizing Grokfast:

### A. Spectral Gradient Amplification

The canonical Grokfast update modifies the parameter gradients as:
$$
\hat{g}_t = g_t + \lambda \, g_t^{(\text{slow})}
$$
where $\lambda > 0$ is an amplification factor. Two filtering mechanisms are considered:

- **Moving Average (MA)**: A windowed average of past $w$ gradients per parameter.
- **Exponential Moving Average (EMA)**: A momentum-style recursion $\mu_t = \alpha \mu_{t-1} + (1-\alpha) g_t$, with the final update $\hat{g}_t = g_t + \lambda \mu_t$.

The EMA variant is preferred for its $O(\text{model size})$ memory usage and ease of integration. It requires only a single additional buffer per parameter [2405.20233].

### B. Egalitarian Gradient Descent (EGD)

EGD whitens the empirical gradient such that every principal direction evolves at identical speed. For a gradient matrix $G_t$ at time $t$:
$$
\tilde G_t = (G_t G_t^\top)^{-1/2} G_t
$$
When $G_t$ has SVD $U_t S_t V_t^\top$, the transformation yields $U_t V_t^\top$, making all singular values unity so that the update moves equally along all axes. Practically, this can be implemented with exact SVD (per layer) or approximate covariance tracking for scalability [2510.04930].

#### Pseudocode Summary (EMA variant)

```python
# Before training
mu = { name: torch.zeros_like(param) for each param }
λ, α = <chosen amplification>, <chosen momentum>
for each step t:
    loss.backward()  # g_t in param.grad
    for name, param in model.named_parameters():
        if param.grad is None: continue
        mu[name] = α*mu[name] + (1-α)*param.grad.data
        param.grad.data = param.grad.data + λ * mu[name]
    optimizer.step()
    model.zero_grad()
```
[2405.20233]

## 4. Empirical Results and Performance Gains

Extensive experiments validate Grokfast and EGD across tasks:

| Task                  | Model          | Baseline Grok Delay | Grokfast/EGD Delay | Speedup       |
|-----------------------|---------------|---------------------|--------------------|---------------|
| modular × mod 97      | 2-layer TF    | ~40,000 steps       | ~800 steps         | ×50           |
| MNIST subsampled      | 3-layer MLP   | ~44,000 steps       | ~2,000 steps       | ×22           |
| Sparse Parity (n,k)   | 2-layer MLP   | 100–200 epochs      | 3–8 epochs         | ×20–100       |

EGD and Grokfast consistently reduce plateau lengths by 20×–100× without loss of final accuracy. For instance, in modular addition, vanilla SGD plateaus for several hundred epochs before validation accuracy rises, while EGD triggers grokking within 5–10 epochs [2405.20233][2510.04930].

Fast convergence is also reported on language (IMDb-LSTM), graph (QM9-GCNN), and other domains, with early increases in validation accuracy relative to standard training.

## 5. Hyperparameters, Trade-offs, and Integration

Key parameters for Grokfast-EMA are the amplification $\lambda$ ($0.1 \leq \lambda \leq 5$) and EMA momentum $\alpha$ ($0.8 \leq \alpha \leq 0.99$). For the MA variant, a window size $w$ in [50, 200] is typical. EGD is hyperparameter-free, relying only on standard optimizer step size.

**Integration**:
- Both approaches are compatible with weight decay (WD ≈ 0.005–0.02 enhances effect), Adam, momentum, and dropout.
- A two-stage regime—conventional training up to overfitting, followed by Grokfast—can further shorten time-to-generalization.
- The unfiltered high-frequency component should not be dropped, as both memorization and generalization modes are necessary.

**Trade-offs**:
- MA filtering introduces $O(w \times \text{model size})$ memory; EMA and EGD are $O(\text{model size})$ for practical low-rank approximations.
- Excessively large $\lambda$ or inappropriate filter bandwidth ($\alpha \to 1$) destabilizes training.
- Full SVD in EGD presents computational overhead in wide layers; mitigations include diagonal/block approximations, low-rank covariance, or randomized SVD.

## 6. Theoretical Guarantees and Limitations

For linear/quadratic objectives, Grokfast and EGD collapse the spectral condition number dependency. In classic SGD, convergence along the slowest mode requires $O(\kappa \log(1/\epsilon))$ steps ($\kappa$ is the condition number); under EGD, all directions contract as $(1-\eta)^k$, making plateau length $O(\log(1/\epsilon))$, independent of spectrum [2510.04930].

The methodology extends to nonlinear networks under mild smoothness assumptions, layer-wise. In practical deep learning, EGD’s efficacy is robust for layers with well-defined principal axes but may require block-wise or approximate updates in very large models. Computational cost may limit exact EGD on transformer-scale layers, although low-rank decompositions offer mitigation.

## 7. Extensions, Applications, and Practitioner Guidance

Grokfast and EGD are applicable to diverse architectures and tasks, demonstrating broad empirical generality. Further developments include streaming principal component analysis for efficient spectral tracking, integration with adaptive gradient methods, and adaptation for transformer models with structural approximations.

**Practitioner advice**:
- Implement Grokfast by adding a one-line gradient modification after backward pass.
- Calibrate $\lambda$ and $\alpha$ via small exploratory runs, monitoring early validation accuracy gains.
- Combine with moderate weight decay for maximal effect.
- Monitor for destabilization if filter/hyperparameters depart from recommended regimes.
- Employ SVD approximations (e.g., low-rank, block-diagonal) for scalable EGD in large-scale settings.

Grokfast and EGD make empirically elusive sudden generalization practical within everyday training budgets, offering significant speed and resource savings for research and production environments [2405.20233][2510.04930].

Source: https://www.emergentmind.com/topics/grokfast-algorithm