Anti-Regularization (AR) Overview

• Anti-regularization (AR) is a technique that reverses traditional regularization by adding a sign-reversed reward to boost model expressivity in low-data regimes.
• It employs a power-law decay for the anti-regularization term while integrating safeguards like gradient clipping, parameter projection, and spectral safety to ensure stable optimization.
• Empirical results show AR reduces underfitting in regression and improves calibration in classification, making it effective for small-sample and challenging settings.

Anti-regularization (AR) is a modification of empirical risk minimization in which a sign-reversed reward term is added to the loss to intentionally increase model expressivity in the small-sample regime, and then attenuated with a power-law decay as the sample size grows (Kim et al., 24 Aug 2025). In the formulation introduced in "Convergence and Generalization of Anti-Regularization for Parametric Models" (Kim et al., 24 Aug 2025), AR is designed to reduce underfitting while preserving generalization, with explicit spectral safety and trust-region conditions and a lightweight stability safeguard combining a projection operator with gradient clipping. The method is analyzed for linear smoothers and in the Neural Tangent Kernel (NTK) regime, and is presented as a simple intervention that integrates into standard empirical risk minimization pipelines while fading away when unnecessary.

1. Formal definition and scheduling

Let $S=\{(x_i,y_i)\}_{i=1}^n$ be the training set and $\theta\in\mathbb{R}^p$ the model parameters. The per-example loss is defined as

$$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$

AR introduces a reward, or negative regularizer,

$$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$

and defines the AR-augmented empirical risk as

$$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$

A central feature is that $\lambda$ is not held fixed. Instead, it decays with sample size according to

$$\lambda(n) \;=\; \lambda_0 \;\Bigl(\frac{n_0}{n}\Bigr)^{\alpha}, \quad \alpha>0.$$

Recommended defaults are $\alpha=1$ for regression and $\alpha=\tfrac12$ for classification (Kim et al., 24 Aug 2025).

This construction differs from traditional regularization in a specific way. Traditional regularizers $(+\lambda\|\theta\|^2)$ suppress complexity uniformly and remain fixed regardless of data size. AR instead rewards complexity initially, deliberately boosting expressivity to fight underfitting in small-data regimes, then fades out via $\theta\in\mathbb{R}^p$0. A plausible implication is that AR is intended not as a permanent inductive bias but as a transient intervention whose magnitude is coupled to sample availability.

2. Optimization procedure and stability safeguards

The training procedure is a standard gradient-based loop with three additions: updating $\theta\in\mathbb{R}^p$1 according to the current sample size, clipping the gradient, and projecting parameters back into a trust region (Kim et al., 24 Aug 2025). The algorithm is specified as follows:

$$\lambda(n) \;=\; \lambda_0 \;\Bigl(\frac{n_0}{n}\Bigr)^{\alpha}, \quad \alpha>0.$$0

The projection operator uses a ball of radius $\theta\in\mathbb{R}^p$2:

$\theta\in\mathbb{R}^p$3

Gradient clipping at threshold $\theta\in\mathbb{R}^p$4 is

$\theta\in\mathbb{R}^p$5

These safeguards are motivated by the main failure mode of negative regularization: when $\theta\in\mathbb{R}^p$6 is too large, the reward term can destabilize optimization or even destroy boundedness. The paper identifies projection and clipping as a lightweight stability safeguard that ensures stable intervention under stated assumptions, and ablation results indicate that disabling these safeguards causes catastrophic divergence in regression and modest instability in classification (Kim et al., 24 Aug 2025).

3. Spectral safety, convexity, and convergence

In linear regression,

$\theta\in\mathbb{R}^p$7

the principal theoretical condition is spectral safety. If

$\theta\in\mathbb{R}^p$8

then the Hessian satisfies

$\theta\in\mathbb{R}^p$9

so $$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$0 is $$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$1-strongly convex with a unique global minimizer

$$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$2

Under the same condition, gradient descent with step-size

$$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$3

converges linearly to $$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$4 (Kim et al., 24 Aug 2025).

The same logic extends to the NTK regime. When a sufficiently wide MLP linearizes under gradient descent, training in function space is equivalent to kernel ridge on the NTK $$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$5:

$$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$6

Again, the spectral safety condition

$$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$7

guarantees existence and amplifies small-eigenvalue directions (Kim et al., 24 Aug 2025).

These results isolate the regime in which anti-regularization remains analytically controlled. The formal role of $$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$8 is not merely to enlarge parameter norms; it selectively amplifies directions that would otherwise be suppressed by the spectrum of the empirical covariance or kernel operator. This suggests that AR is most naturally interpreted as a controlled relaxation of regularization rather than an unconstrained drive toward high-complexity solutions.

4. Boundedness, classification, and risk–variance balancing

For classification, the paper emphasizes that cross-entropy minus a quadratic reward is unbounded below on separable data. Two remedies are given. First, minimizing $$\ell(\theta;x,y) \;=\; \begin{cases} \tfrac12\,(y - f_\theta(x))^2 & \text{(regression)}\ H\big(p_y,\;q_\theta(x)\big) & \text{(classification, cross-entropy)}. \end{cases}$$9 over the compact set $$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$0 ensures a minimum exists. Second, one may replace $$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$1 with a bounded, Lipschitz margin reward $$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$2, in which case

$$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$3

so the objective is bounded below (Kim et al., 24 Aug 2025).

A separate line of analysis studies the balance between empirical risk and variance in linear smoothers via the optimism identity

$$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$4

For small $$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$5,

$$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$6

and

$$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$7

Balancing first-order terms yields $$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$8, so $$R(\theta) \;=\; \tfrac12\,\|\theta\|_2^2 \quad\text{or more generally}\quad R(\theta) =\tfrac12\,\theta^\top W\,\theta,\;W\succeq0,$$9 is the natural choice in regression. In classification, the complexity penalty scales like $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$0, so $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$1 (Kim et al., 24 Aug 2025).

This analysis provides the rationale for the power-law schedule. Rather than treating the decay exponent as an arbitrary tuning parameter, the paper ties it to asymptotic control of variance inflation. The recommended defaults therefore emerge from a balance between improved empirical fit and the increase in effective complexity induced by the sign-reversed reward.

5. Empirical findings and ablation structure

The reported regression experiments are on UCI Concrete and Airfoil Self-Noise. At small–medium splits (1–25%), AR reduces RMSE and raises $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$2 markedly; at larger splits it reverts to the baseline (no harm). With respect to optimization, SGD + Momentum diverges without safety, whereas projection + clipping restores stability. Adam is described as naturally more stable but still benefiting from safety. The output-scale ratio $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$3 remains near 1 when AR is safe, and projection/clipping rates stay within recommended bounds (Kim et al., 24 Aug 2025).

The classification experiments are on MNIST and CIFAR-10. On easy MNIST, AR is neutral, with baseline $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$4 chosen except at 0.5%. On harder CIFAR-10, AR consistently reduces Expected Calibration Error (ECE) by 2–4 points while leaving accuracy unchanged (Kim et al., 24 Aug 2025).

The ablation studies identify three critical axes. First, the decay exponent matters: $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$5 around 1 is robust for regression, and around 0.5 for classification; performance is flat in a wide band, which the paper treats as confirmation of the theoretical defaults. Second, the stability safeguards are critical: disabling them causes catastrophic divergence in regression and modest instability in classification. Third, a degrees-of-freedom targeting schedule that fixes $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$6 works, but is more complex than the simple power-law schedule (Kim et al., 24 Aug 2025).

These findings constrain several potential misconceptions. AR is not presented as uniformly beneficial across tasks; it is neutral on an easy benchmark and primarily advantageous in harder or smaller-data settings. Nor is it a substitute for numerical safeguards: the empirical results explicitly associate the reward term with divergence when projection and clipping are removed.

6. Implementation, hyperparameters, and relation to standard regularization

AR is intended to integrate directly into standard empirical risk minimization. No architectural changes are required: one replaces

$$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$7

and adds a few lines for $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$8, gradient clipping, and parameter projection. The method works with any gradient-based optimizer, including SGD and Adam. The paper notes that AR may be decoupled from weight-decay hooks such as AdamW to avoid interference (Kim et al., 24 Aug 2025).

The computational overhead is described as negligible per step for gradient clipping and projection. Spectral safety, meaning estimation of $$\hat F_{\lambda}(\theta) \;=\; \underbrace{\frac1n\sum_{i=1}^n \ell(\theta;x_i,y_i)}_{\text{empirical risk}} \;-\; \underbrace{\lambda\,R(\theta)}_{\text{anti-regularization (reward)}}, \quad \lambda\ge0.$$9, can be done via a few power-method iterations or cheaply approximated in large models. Suggested initial strengths are

$\lambda$0

The recommended decay exponents are $\lambda$1 for regression and $\lambda$2 for classification, with the instruction to increase $\lambda$3 if early instability is observed. The trust-region radius $\lambda$4 is chosen to be generous, for example 5–10, but finite. The gradient-clip threshold $\lambda$5 is set to typical gradient norms, and the clipping ratio $\lambda$6 is monitored in the ranges $\lambda$7 for regression and $\lambda$8 for classification (Kim et al., 24 Aug 2025).

The comparison with traditional regularization is explicit. Traditional regularizers suppress complexity uniformly and remain fixed regardless of data size. AR instead rewards complexity initially and then fades out with data. The paper therefore positions AR as especially attractive for SMEs or low-resource settings where underfitting is as much a concern as overfitting (Kim et al., 24 Aug 2025). A plausible implication is that its main domain is not large-data asymptotics, where $\lambda$9, but the transitional regime in which sample scarcity and limited resources make underfitting operationally significant.