Anchored Weight Decay in ES Fine-Tuning

• AWD is a regularization technique that adds an anchored penalty to ES fine-tuning to constrain optimization toward the initial model parameters.
• It improves prior-task stability and mitigates performance drift seen with standard ES methods while reducing computational cost compared to larger populations.
• AWD integrates a weight-decay step post-ES update using L1 or L2 penalties, offering a practical trade-off between sparsity and smooth shrinkage.

Searching arXiv for the specified paper and closely related context. Anchored Weight Decay (AWD) is a parameter-space regularization technique introduced for Evolution Strategies (ES) fine-tuning of LLMs, in which optimization is explicitly constrained toward the initial model parameters $\theta_0$. In "Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies" (Schweighofer et al., 28 May 2026), AWD is proposed in response to prior-task degradation observed during ES fine-tuning. The paper argues that this phenomenon is better characterized as performance drift rather than irreversible forgetting, and that it is not unique to ES because it can also arise under reinforcement-learning fine-tuning. AWD addresses this drift by adding an anchor penalty to the ES objective, with the stated effect of stabilizing prior-task performance while preserving target-task performance at much lower computational cost than large ES populations.

1. Formal definition

In standard ES, the objective is to maximize the expected reward under Gaussian weight perturbations:

$$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$

AWD augments this objective with a penalty on deviation from the initial weights:

$$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$

where $l(x)=\tfrac12 x^2$ for an $\ell_2$ penalty or $l(x)=|x|$ for an $\ell_1$ penalty, and $\lambda>0$ controls the anchor strength.

The defining feature of AWD is that the decay target is not the origin but the initialization $\theta_0$. In this sense, AWD is a regularizer over weight displacement rather than over weight magnitude. The formulation is explicitly intended to constrain optimization toward the starting model while allowing task-improving movement when supported by reward signal.

2. Mechanistic motivation

The paper’s theoretical intuition attributes prior-task drift to ES training dynamics, particularly random-walk behavior in weakly constrained directions of the weight space (Schweighofer et al., 28 May 2026). ES updates are described as decomposing into a signal aligned with the target-task gradient and an isotropic noise component. In directions where the target reward provides little constraint, the noise accumulates over iterations and drives the parameters away from $\theta_0$.

Under a simple isotropic model, the expected squared deviation is given as

$$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$0

where $$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$1 is the number of iterations, $$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$2 is the number of parameters, and $$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$3 is the population size. In the paper’s summary, this implies that drift scales proportionally to $$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$4.

AWD is motivated as a direct countermeasure to this effect. By adding $$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$5 or $$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$6 to the objective, it penalizes large deviations from the initial model. The paper states that this bias acts like a proximal or trust-region step, damping random-walk in unconstrained directions. A plausible implication is that AWD primarily targets instability in low-curvature subspaces rather than suppressing all learning uniformly.

3. Optimization rule and algorithmic integration

In implementation, AWD is applied as a weight-decay step after the standard ES update. The update rule is

$$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$7

$$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$8

$$\max_\theta \; J_{\rm ES}(\theta) \;=\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr].$$9

where $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$0 denotes normalized reward.

Operationally, the procedure consists of sampling perturbations $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$1, evaluating rewards $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$2, normalizing rewards, applying the standard ES update, and then applying the anchored decay. The summary presents normalization as, for example, a z-score over the sampled rewards. The required inputs are the initial weights $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$3, reward function $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$4, iteration count $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$5, population size $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$6, noise scale $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$7, ES step $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$8, anchor strength $$\max_\theta\; \mathbb E_{\epsilon \sim \mathcal N(0,I)}\bigl[R(\theta + \sigma\,\epsilon)\bigr] \;-\; \lambda \sum_{i=1}^d l\bigl(\theta_i-\theta_{0,i}\bigr),$$9, and penalty $l(x)=\tfrac12 x^2$0.

This placement of the regularization after the ES step is central to the method’s practical identity. AWD is therefore not a separate optimizer; it is a modification of ES fine-tuning that preserves the original inference-only structure while adding an explicit pull toward the initialization.

4. Penalty choices and tuning regime

The summary distinguishes two penalty families. With $l(x)=\tfrac12 x^2$1, AWD induces smooth shrinkage; with $l(x)=\tfrac12 x^2$2, it induces sparsity in weight differences. Both are reported to give comparable prior-task stability, while $l(x)=\tfrac12 x^2$3 yields a smoother target-task trade-off (Schweighofer et al., 28 May 2026).

The reported hyperparameter guidance is specific. For population size, the experiments use $l(x)=\tfrac12 x^2$4 as a baseline and also evaluate $l(x)=\tfrac12 x^2$5 and $l(x)=\tfrac12 x^2$6. For the ES step and noise scale, the practitioner-oriented default is $l(x)=\tfrac12 x^2$7 and $l(x)=\tfrac12 x^2$8, with the proviso that the signal-to-noise ratio must remain sufficient. For anchor strength, the summary gives $l(x)=\tfrac12 x^2$9 for $\ell_2$0 and $\ell_2$1 for $\ell_2$2 on Qwen-2.5 3B on Countdown.

The recommended tuning recipe is to start with high $\ell_2$3 for strong anchoring and then decrease it until target-task accuracy matches unregularized ES. The practical recommendation is to use $\ell_2$4 anchoring by default, with the same high-to-low sweep over $\ell_2$5. This suggests that the target-task baseline, rather than a fixed regularization scale, serves as the operative reference point for calibration.

5. Experimental setting and quantitative behavior

The reported target tasks use 200 train examples each: Countdown for arithmetic reasoning, GSM8K for multi-step math word problems, and ProofWriter for logical inference (Schweighofer et al., 28 May 2026). Prior tasks use 500–2,000 evaluation examples each and include HellaSwag, PIQA, ARC-Challenge, and MMLU-Pro, plus the other two target tasks when not trained. The metrics are target-task accuracy under greedy decoding, accuracy on each prior task as a measure of forgetting, and KL-divergence between the base and fine-tuned model distributions as a measure of output shift.

For Countdown on Qwen-2.5 3B, the reported results are as follows:

Method	Target-task change	Prior-task average change
ES (30)	+45.3 pp	$\ell_2$6 pp
ES (128)	+48.6 pp	$\ell_2$7 pp
ES+AWD ($\ell_2$8, $\ell_2$9)	+44.3 pp	$l(x)=|x|$0 pp
ES+AWD ($l(x)=|x|$1, $l(x)=|x|$2)	+43.8 pp	$l(x)=|x|$3 pp

The summary’s interpretation is that AWD at $l(x)=|x|$4 matches the prior-task stability of ES at $l(x)=|x|$5, at much lower cost. It also reports characteristic KL-divergence patterns: ES(30) induces large KL shifts on prior tasks, whereas ES(128) or AWD closes this gap. On the target task, ES variants maintain low KL divergence, while GRPO exhibits much larger shifts.

These results are presented as support for two broader claims. First, prior-task degradation during ES training is often transient enough to be described as drift rather than irreversible forgetting. Second, stabilization can be achieved without the compute burden implied by large populations.

6. Computational profile, deployment guidance, and scope

The summary contrasts forward-only ES with RL in the form of GRPO (Schweighofer et al., 28 May 2026). ES relies solely on forward passes and is described as scaling trivially across GPUs or nodes via seeds and rewards, whereas GRPO requires forward and backward passes together with cross-device gradient synchronization. Within ES itself, increasing population size is computationally expensive: doubling $l(x)=|x|$6 doubles GPU inference cost per iteration, and ES(256) is stated to require approximately $l(x)=|x|$7 the compute of ES(30).

Against this background, AWD is positioned as a low-overhead regularizer. Its additional cost consists of the weight-decay step and streaming $l(x)=|x|$8 from CPU to GPU once per iteration, with approximately $l(x)=|x|$9–$\ell_1$0 runtime overhead in practice. The accompanying implementation advice is to keep a pinned-RAM copy of $\ell_1$1 and stream it layer-by-layer during updates to avoid GPU memory bloat. The recommended monitoring practice is to log prior-task accuracies and KL divergences during training to detect drift early.

The scope of AWD is also delimited in the summary. It only constrains parameter drift globally; for key submodules, task-weighted anchor penalties may be considered if certain layers encode more prior knowledge. In continual-learning deployments, the summary recommends combining AWD with occasional small replay buffers or importance-weighted regularization when domain knowledge about critical tasks is available. It also explicitly cautions that AWD stabilizes task performance but does not automatically preserve all behavioral priors, and that unintended alignment or safety shifts should still be monitored in downstream evaluation.

A common misconception addressed by the underlying paper is that ES uniquely causes irreversible prior-task forgetting. The paper’s position is narrower and more technical: prior-task performance often recovers during ES training, similar drift can also arise in RL methods, and AWD is effective because it constrains the random-walk component of optimization rather than eliminating target-task adaptation. In that framing, AWD is best understood as a practical regularization mechanism for continual fine-tuning regimes in which retention of prior capabilities matters alongside target-task gains.