Driven Hunt Method Optimization

• Driven Hunt Method is an optimization approach that leverages a predator–prey framework to guide parameters along low-curvature ravine trajectories.
• It introduces a two-agent dynamic where interaction forces between the prey (model) and predator enable directed movement, reducing slow Brownian diffusion.
• Empirical results demonstrate significant acceleration in generalization, with up to 100× fewer gradient calls required on grokking tasks.

Driven Hunt Method is an optimization procedure introduced in the predator–prey framework of "Predator-Prey Model: Driven Hunt for Accelerated Grokking" (Lopatin et al., 10 Sep 2025). It is defined for training a parametric model with parameters $\theta$ in a high-dimensional Euclidean space to minimize a loss $L(\theta)$, especially in overparameterized regimes where the loss landscape exhibits ravine structure: narrow valleys with large curvature across the valley walls and weak curvature along the valley bottom. In that setting, standard optimizers can reach a zero-risk manifold quickly but then move only diffusively along it, producing delayed generalization or grokking. The Driven Hunt Method introduces two interacting agents—a prey and a predator—whose interaction adds a direct force along the line connecting them; in ravine regimes this force tends to align with the low-curvature manifold direction and thereby converts slow diffusion into directed motion along the ravine (Lopatin et al., 10 Sep 2025).

1. Ravine structure and the optimization problem

The method is formulated for minimizing a differentiable loss $L(\theta)$ over $\theta \in \mathbb{R}^d$. Its central motivation is the observation that, in overparameterized systems, the relevant part of the loss landscape often has a ravine structure: a low-loss manifold $M \subset \mathbb{R}^d$ with high-curvature directions orthogonal to $M$ and low-curvature directions tangent to $M$. Near a point $\theta^\star \in M$, the Hessian $H(\theta^\star)$ is described as having a subset of large eigenvalues $\{\lambda_\perp\}$ aligned with normal directions to $L(\theta)$0 and small eigenvalues $L(\theta)$1 along tangent directions to $L(\theta)$2, with a decomposition

$L(\theta)$3

such that

$L(\theta)$4

and

$L(\theta)$5

with $L(\theta)$6, and typically $L(\theta)$7 near zero along an interpolating zero-risk manifold in overparameterized systems (Lopatin et al., 10 Sep 2025).

Within this geometry, SGD and Adam are described as falling quickly into the ravine because of strong gradients across the walls, but subsequently diffusing along the flat bottom. The expected distance traveled along the ravine under approximately Brownian motion scales like $L(\theta)$8, whereas directed motion scales like $L(\theta)$9. The paper identifies this disparity as the mechanism behind delayed generalization in grokking-prone regimes. In this terminology, "driven hunt" refers specifically to the ravine case in which the predator drives the prey along the ravine rather than allowing training to remain in a random walk along the zero-risk manifold.

2. Two-agent dynamics and exact update rules

The Driven Hunt Method introduces two parameter vectors. The prey $L(\theta)$0 is the main trainable model, and the predator $L(\theta)$1 is a second agent that chases the prey. Their geometry is defined by the distance

$L(\theta)$2

and, when $L(\theta)$3, the unit direction from predator to prey

$L(\theta)$4

The interaction is governed by a distance-dependent potential $L(\theta)$5 with strength parameter $L(\theta)$6 and radius parameter $L(\theta)$7. The prey is repelled along $L(\theta)$8 with magnitude $L(\theta)$9, while the predator is driven along the same direction with magnitude $\theta \in \mathbb{R}^d$0. The baseline optimizer is Adam/AdamW-style. Using gradient $\theta \in \mathbb{R}^d$1 and current parameters $\theta \in \mathbb{R}^d$2, the update denoted $\theta \in \mathbb{R}^d$3 performs the steps

$\theta \in \mathbb{R}^d$4

$\theta \in \mathbb{R}^d$5

$\theta \in \mathbb{R}^d$6

with bias corrections

$\theta \in \mathbb{R}^d$7

and parameter update

$\theta \in \mathbb{R}^d$8

The implementation note is explicit that in PP-conn, Adam-style weight decay is folded into the gradient inside UpdateAdam, although for AdamW the decay can be decoupled (Lopatin et al., 10 Sep 2025).

Two variants are specified. In the classic predator–prey variant, the prey first takes a baseline step

$\theta \in \mathbb{R}^d$9

and the predator either takes its own optimizer step,

$M \subset \mathbb{R}^d$0

when GradPred=True, or remains unchanged,

$M \subset \mathbb{R}^d$1

when GradPred=False. The interaction then applies

$M \subset \mathbb{R}^d$2

In the connected-momenta variant, PP-conn, the prey and predator share optimizer moments $M \subset \mathbb{R}^d$3. A pretraining stage of $M \subset \mathbb{R}^d$4 steps first evolves a single model $M \subset \mathbb{R}^d$5: $M \subset \mathbb{R}^d$6 Afterward, $M \subset \mathbb{R}^d$7 and $M \subset \mathbb{R}^d$8. For $M \subset \mathbb{R}^d$9, the prey update is

$M$0

while the predator uses either $M$1 or $M$2, depending on GradPred, and then

$M$3

The interaction is again

$M$4

A fixed-distance regime is identified: when $M$5, the predator–prey distance stabilizes at

$M$6

Empirically, $M$7 tends to $M$8 and $M$9 (Lopatin et al., 10 Sep 2025).

3. Mechanism in ravines: from diffusion to directed motion

The geometric mechanism is described in terms of the decomposition of motion into normal and tangent components relative to the zero-risk manifold $M$0. After the memorization phase, both prey and, optionally, predator are near the ravine bottom. In this regime, the optimizer’s gradient acts mainly in normal directions, correcting deviations across the ravine walls, while the tangent directions remain weakly curved and correspondingly weakly driven by the loss. Standard training therefore exhibits stochastic wandering along $M$1.

Driven hunt modifies this regime by adding the interaction displacement

$M$2

Near $M$3, the direction $M$4 is described as having a dominant tangent component $M$5, since both agents lie close to the manifold. The tangent displacement per step is therefore approximately

$M$6

while normal oscillations remain controlled by the baseline optimizer. The resulting division of labor is explicit: across high-curvature directions, the optimizer keeps both agents near $M$7; along low-curvature directions, predator chase and prey repulsion induce directed motion with effective speed proportional to $M$8 for the prey and $M$9 for the predator (Lopatin et al., 10 Sep 2025).

This construction distinguishes Driven Hunt from ordinary momentum or ravine-acceleration heuristics. The method does not estimate curvature. It uses the second agent’s position to define a direction of motion and then applies a controlled driving force along that direction. The paper states that, once memorization has occurred, this agent-defined direction empirically aligns with ravine directions. It also reports that the interaction term often exceeds the optimizer step norm while still accelerating grokking without destabilizing test accuracy. A further theoretical interpretation offered in the paper is entropy-based irreversibility: after the driven motion reaches high-entropy neighborhoods around generalizing solutions, the dynamics remain captured in wide valleys, making the transition effectively irreversible in the same qualitative sense discussed for baseline grokking.

4. Algorithmic variants, hyperparameters, and computational overhead

The recommended implementation is PP-conn. Its procedure is: initialize $\theta^\star \in M$0, set shared moments $\theta^\star \in M$1, $\theta^\star \in M$2, and predator $\theta^\star \in M$3; pretrain for $\theta^\star \in M$4 steps with Adam/AdamW to reach the ravine; set prey $\theta^\star \in M$5 and predator $\theta^\star \in M$6; then run the predator–prey loop until a target test-accuracy threshold is met. During this loop, one computes the prey gradient, updates the prey with UpdateAdam, updates the predator with either its own gradient or zero gradient, applies the interaction displacement, and periodically evaluates train and test accuracy together with diagnostics such as $\theta^\star \in M$7, $\theta^\star \in M$8, $\theta^\star \in M$9, and $H(\theta^\star)$0 (Lopatin et al., 10 Sep 2025).

The interaction hyperparameters are $H(\theta^\star)$1, $H(\theta^\star)$2, $H(\theta^\star)$3, $H(\theta^\star)$4, and GradPred. Their roles are specified directly. Larger $H(\theta^\star)$5 or smaller $H(\theta^\star)$6 increase $H(\theta^\star)$7. The paper reports that $H(\theta^\star)$8 worked well in experiments, and that $H(\theta^\star)$9 was used for both tasks. For the optimizer, the baseline AdamW defaults are

$\{\lambda_\perp\}$0

In PP-conn, EMA (use_m=True) was used for MNIST and disabled for ModuloOperation in some runs.

The computational overhead is stated in terms of gradient calls. Baseline AdamW uses one gradient evaluation per step. PP-classic also uses one gradient evaluation per step when GradPred=False, but two when GradPred=True. PP-conn uses one gradient evaluation per step because the predator can run with $\{\lambda_\perp\}$1 and shared moments. Additional operations consist only of computing $\{\lambda_\perp\}$2 and $\{\lambda_\perp\}$3 and adding the interaction displacements; these are described as negligible relative to forward and backward passes. A practical limitation is memory: PP-classic doubles the parameter footprint, while PP-conn alleviates this by sharing optimizer state but still maintains two parameter copies. Sensitivity to hyperparameters is also acknowledged, and automated ravine-entry detection together with adaptive schedules for $\{\lambda_\perp\}$4, $\{\lambda_\perp\}$5, and $\{\lambda_\perp\}$6 is listed as an open engineering question (Lopatin et al., 10 Sep 2025).

5. Empirical results on grokking tasks

The empirical evaluation is organized around grokking, defined as delayed generalization: training accuracy rapidly reaches near-perfect values on the training set, while test accuracy improves only after a long delay. Two principal tasks are reported. The first is modular arithmetic with a decoder-only transformer on division modulo $\{\lambda_\perp\}$7, learning

$\{\lambda_\perp\}$8

from pairs $\{\lambda_\perp\}$9. The second is MNIST classification with an MLP in a restricted setup chosen to induce grokking: $L(\theta)$00 images, MSE loss, no softmax, and increased initial weight norms. For the AdamW baseline, memorization occurs quickly, but time-to-generalization is approximately $L(\theta)$01 epochs for modulo division and approximately $L(\theta)$02 epochs for MNIST. Batch sizes are $L(\theta)$03 for the modulo task and $L(\theta)$04 for MNIST; epoch sizes are $L(\theta)$05 and $L(\theta)$06 steps respectively; generalization thresholds are $L(\theta)$07 test accuracy for modulo division and $L(\theta)$08 for MNIST (Lopatin et al., 10 Sep 2025).

For PP-classic with GradPred=False, the number of gradient calls per epoch is the same as in the baseline, yet the reported acceleration is approximately $L(\theta)$09 fewer gradient calls to reach the test-accuracy threshold on modulo division and approximately $L(\theta)$10 fewer on MNIST. The paper notes that the predator may lose accuracy and leave the zero-risk manifold while the prey maintains or improves accuracy. It also reports that early iterations can slightly push the prey away from the ravine if the initial prey–predator direction is orthogonal, but that the prey returns due to loss gradients.

For PP-conn with GradPred=False, the reported acceleration is approximately $L(\theta)$11 on both tasks, while maintaining a single gradient evaluation per step. In the $L(\theta)$12 regime, the interaction potential tends to $L(\theta)$13 and the predator–prey distance stabilizes at

$L(\theta)$14

Accuracy is reported to remain stable or improve after reaching generalization, and additional epochs do not decrease test accuracy. Diagnostic curves further show that $L(\theta)$15 often dominates the optimizer step norm during acceleration, and that the prey covers distances comparable to baseline training much faster under driven hunt.

The paper also reports baseline grokking-time dependencies for modular addition with $L(\theta)$16. Let $L(\theta)$17 denote the fraction of the maximal sample

$L(\theta)$18

Then time-to-grok decreases exponentially,

$L(\theta)$19

observed as a linear decrease of $L(\theta)$20 versus $L(\theta)$21, averaged over $L(\theta)$22 runs per point. It also reports that $L(\theta)$23 grows linearly with initial weight norm, again averaged over $L(\theta)$24 runs per point. For PPM in the PP-conn form, an explicit dependence of $L(\theta)$25 on sample size was not determined (Lopatin et al., 10 Sep 2025).

6. Relation to existing optimizers, deployment practice, and open problems

Driven Hunt is presented in relation to several established optimizer families. Unlike gradient descent, momentum, and Adam, it does not rely solely on gradients of $L(\theta)$26, because it adds an interaction term that enforces movement along the prey–predator line independently of the gradient. Unlike Nesterov-style or classical ravine methods, it does not use extrapolation or explicit curvature estimation. Unlike natural gradient, it does not reweight gradients by Fisher information. Unlike evolutionary strategies, it does not use population-based exploration. The paper characterizes it instead as closer to a simple MARL-style interaction embedded into optimization: the prey is repelled, the predator chases, and the resulting two-agent dynamics produce a geometry-informed drift (Lopatin et al., 10 Sep 2025).

The practical guidance is correspondingly specific. The method is intended for regimes known to exhibit grokking or slow progress along flat valleys, such as modular arithmetic and heavily overparameterized models trained on small datasets. The recommended deployment recipe is to train with AdamW for a small number of steps $L(\theta)$27—for example $L(\theta)$28—until train accuracy is high or the optimizer step norm drops, indicating ravine entry; then instantiate prey at the pretrained model and predator at $L(\theta)$29; use PP-conn with shared optimizer state; choose $L(\theta)$30 and $L(\theta)$31 so that $L(\theta)$32, with $L(\theta)$33; and select $L(\theta)$34 so that the steady distance

$L(\theta)$35

is reasonable. Reported starting values are $L(\theta)$36, $L(\theta)$37 for transformers on modular arithmetic and $L(\theta)$38, $L(\theta)$39 for the MNIST MLP.

The same section of the paper also makes clear that the predator need not be an independently learning model. GradPred=False often suffices, and in the fastest reported runs the predator uses zero gradient while still providing the driving geometry. This addresses a likely misconception that both agents must optimize the task loss symmetrically. Another likely misconception is that the method guarantees acceleration in arbitrary landscapes; the paper instead limits its claims to ravine-like regimes and identifies sensitivity to $L(\theta)$40, $L(\theta)$41, and $L(\theta)$42, as well as open problems including explicit $L(\theta)$43 laws for PPM, adaptive hyperparameter selection from curvature proxies or ravine detection, formal convergence guarantees in nonconvex ravine landscapes, and extensions to multi-agent variants and information-sharing schemes.

Reproducibility details are explicit. The codebase is reported at https://github.com/ilyalopatin26/PpmGrok.git. The modulo-division transformer uses $L(\theta)$44, $L(\theta)$45, $L(\theta)$46 layers, and $L(\theta)$47, with Xavier initialization for linear layers and normal initialization for embeddings with standard deviation $L(\theta)$48. The modular-addition scaling experiment uses $L(\theta)$49, $L(\theta)$50, $L(\theta)$51 layer, and $L(\theta)$52, with embeddings of standard deviation $L(\theta)$53 and a generalization threshold reduced to $L(\theta)$54 because of increased stochasticity with small data. The MNIST model is a single-hidden-layer MLP of width $L(\theta)$55 with ReLU, uniform default initialization, MSE loss, no softmax, and initial weights multiplied by a coefficient to increase norm. Taken together, these details situate the Driven Hunt Method as a two-agent optimization procedure for ravine-dominated grokking dynamics, with the central operational claim that it converts slow ravine diffusion into directed progress and thereby reduces the gradient calls needed to reach generalization by large factors on the demonstrated tasks (Lopatin et al., 10 Sep 2025).