Hadamard-Based Weight Smoothing in Sparse Learning

• Hadamard-based weight smoothing is a framework that transforms non-smooth, sparsity-regularized problems into equivalent smooth ones using overparametrization with element-wise products.
• It introduces surrogate smooth penalties through auxiliary variables, allowing standard gradient-based optimizers to efficiently navigate sparse learning tasks.
• Empirical results demonstrate significant sparsity and model compression in high-dimensional regression and neural network training with minimal performance trade-offs.

Hadamard-based weight smoothing is a general framework for converting non-smooth, sparsity-regularized optimization problems into equivalent smooth problems via overparametrization with Hadamard (element-wise) products or powers. By introducing smooth surrogate variables and reparametrizations, this approach enables the use of standard gradient-based optimization algorithms for sparse learning tasks, while preserving the underlying minima structure of non-smooth objectives (Kolb et al., 2023).

1. Base Formulation and Motivation

Sparse regularization, as found in penalties such as the $\ell_1$ norm or group norms, is fundamental in high-dimensional regression and model compression. These penalties give rise to optimization problems of the form

$P(\psi, w) = L(\psi, w) + \lambda R(w),$

where $L$ is a smooth loss (typically empirical risk), and $R$ is a non-smooth, sparsity-promoting regularizer (e.g., $R(w) = \|w\|_1$ or $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$ for grouped weights). The non-differentiability and often non-convexity of $R$ (especially for $\ell_q$ with $q < 1$) impede the application of standard SGD, leading to oscillatory dynamics near zero, poor sparsity, and slow convergence.

2. Hadamard Overparametrization: Construction and Variants

To address non-smooth regularization, auxiliary variables $\xi$ are introduced with a smooth surjection $P(\psi, w) = L(\psi, w) + \lambda R(w),$0. The simplest case is the depth-2 Hadamard product parametrization (HPP):

$P(\psi, w) = L(\psi, w) + \lambda R(w),$1

More generally, the approach encompasses various structured parametrizations:

Parametrization	Formula	Induced Regularization
HPP$P(\psi, w) = L(\psi, w) + \lambda R(w),$2	$P(\psi, w) = L(\psi, w) + \lambda R(w),$3	$P(\psi, w) = L(\psi, w) + \lambda R(w),$4
GHPP (group)	$P(\psi, w) = L(\psi, w) + \lambda R(w),$5	group-$P(\psi, w) = L(\psi, w) + \lambda R(w),$6
GHPP$P(\psi, w) = L(\psi, w) + \lambda R(w),$7 (mixed group)	$P(\psi, w) = L(\psi, w) + \lambda R(w),$8	$P(\psi, w) = L(\psi, w) + \lambda R(w),$9
HPowP$L$0	$L$1	$L$2

Here, $L$3 is always smooth and surjective, implying every $L$4 has infinitely many preimages $L$5 such that $L$6.

3. Surrogate Smooth Penalties and Variational Equivalence

A key component is the smooth surrogate penalty $L$7 in the auxiliary space, usually a weighted $L$8 norm. For HPP, $L$9. By an elementary AM-GM argument,

$R$0

meaning the surrogate penalty achieves the sparsity penalty exactly on the fiber $R$1. More generally,

$R$2

with equality for optimal $R$3. This construction generates closed-form surrogates for:

• $R$4 via HPP$R$5
• group $R$6 via GHPP
• mixed $R$7 via deeper group products
• general $R$8 for real $R$9 via HPowP

4. Equivalence of Minima and Theoretical Guarantees

The surrogate objective is defined as:

$R(w) = \|w\|_1$0

Under two mild conditions—local openness of $R(w) = \|w\|_1$1 at optimal $R(w) = \|w\|_1$2 and upper-hemicontinuity of the minimizer map $R(w) = \|w\|_1$3—the set of local and global minima of $R(w) = \|w\|_1$4 and $R(w) = \|w\|_1$5 coincide:

• Any local minimum $R(w) = \|w\|_1$6 of $R(w) = \|w\|_1$7 corresponds to local minima $R(w) = \|w\|_1$8 of $R(w) = \|w\|_1$9 with $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$0.
• Conversely, local minima of $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$1 project to local minima of $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$2 via $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$3.
• Global minima of $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$4 and $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$5 match, since $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$6 majorizes $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$7 and achieves equality at constrained minima.

All parametrizations and surrogates are $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$8 except possibly at coordinate-wise zero; the regularizer handles these singularities. No spurious local minima are introduced.

5. Gradient-Based Optimization Algorithms

Hadamard-based weight smoothing enables the direct use of SGD or Adam for sparse regularized objectives. For $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$9 via HPP, gradient steps proceed as follows:

• Let $R$0.
• Compute $R$1 at $R$2.
• Update:
  • $R$3
  • $R$4
  • $R$5
  • $R$6

Deeper or group variants use appropriate Jacobian factors and distribute the $R$7 terms. Initialization can use small random values or AM-GM matched factors. This approach requires only substituting the $R$8 parameters with auxiliary variables and $R$9 regularization, without the need for proximal operators or custom solvers.

6. Empirical Evaluation and Use Cases

Several empirical studies demonstrate the practicality of Hadamard-based weight smoothing:

• High-Dimensional Regression: On synthetic problems $\ell_q$0, HPP-SGD closely matches Lasso regularization paths, produces exact sparsity, and converges reliably. Direct SGD on nonsmooth $\ell_q$1 is inferior, failing to produce zeros due to oscillation.
• Sparse Neural Network Training: Fully-connected LeNet-300-100 models trained with HPP$\ell_q$2 on MNIST, using $\ell_q$3 regularization on $\ell_q$4, yield weights $\ell_q$5 with pronounced $\ell_q$6 sparsity. After one-shot pruning, up to 99% parameter reduction is achievable with minimal loss increase. Deeper factorizations (HPP$\ell_q$7 for $\ell_q$8) further enhance sparsity.
• Filter-Sparse CNNs: Applying group Hadamard powers to convolution filter groups achieves up to 90% filter removal after training from scratch, with negligible accuracy degradation.

7. Comparison with Existing Methodologies

Hadamard-based weight smoothing unifies various sparsity-inducing formulations:

• Universality: Accommodates $\ell_q$9, $q < 1$0 ($q < 1$1), $q < 1$2, and more within a single SGD-compatible framework.
• Theoretical Guarantees: Ensures equivalence of all minima between original and surrogate problems, with no spurious solutions.
• Practical Performance: Matches specialized solvers (e.g., glmnet, SGL) in high-dimensional regression; delivers substantial sparsity in neural networks using standard SGD.
• Implementation Simplicity: Replaces $q < 1$3 with surrogate variables; relies only on standard $q < 1$4 regularization and smooth optimization.
• Computational Overhead: Increases parameter count moderately; shallow models incur minimal extra compute. For deep/large models, parameter sharing and twin initializations can mitigate costs.

Hadamard-based weight smoothing offers a plug-and-play technique for achieving exact sparse regularization via smooth objectives, applicable across a diverse range of models and compatible with modern deep learning toolkits and optimizers (Kolb et al., 2023).