---
title: Hadamard-Based Weight Smoothing in Sparse Learning
url: https://www.emergentmind.com/topics/hadamard-based-weight-smoothing
type: topic
---

# Hadamard-Based Weight Smoothing in Sparse Learning

Hadamard-based weight smoothing is a general framework for converting non-smooth, sparsity-regularized optimization problems into equivalent smooth problems via overparametrization with Hadamard (element-wise) products or powers. By introducing smooth surrogate variables and reparametrizations, this approach enables the use of standard gradient-based optimization algorithms for sparse learning tasks, while preserving the underlying minima structure of non-smooth objectives [2307.03571].

## 1. Base Formulation and Motivation

Sparse regularization, as found in penalties such as the $\ell_1$ norm or group norms, is fundamental in high-dimensional regression and model compression. These penalties give rise to optimization problems of the form
$$
P(\psi, w) = L(\psi, w) + \lambda R(w),
$$
where $L$ is a smooth loss (typically empirical risk), and $R$ is a non-smooth, sparsity-promoting regularizer (e.g., $R(w) = \|w\|_1$ or $\sum_j \sqrt{|G_j|} \|w_{G_j}\|_2$ for grouped weights). The non-differentiability and often non-convexity of $R$ (especially for $\ell_q$ with $q < 1$) impede the application of standard SGD, leading to oscillatory dynamics near zero, poor sparsity, and slow convergence.

## 2. Hadamard Overparametrization: Construction and Variants

To address non-smooth regularization, auxiliary variables $\xi$ are introduced with a smooth surjection $K: \xi \mapsto w$. The simplest case is the depth-2 Hadamard product parametrization (HPP):
$$
w = u \odot v, \quad u, v \in \mathbb{R}^d.
$$
More generally, the approach encompasses various structured parametrizations:

| Parametrization | Formula | Induced Regularization |
|-----------------|---------|-----------------------|
| HPP$_k$         | $w = u_1 \odot \cdots \odot u_k$ | $\ell_{2/k}$ |
| GHPP (group)    | $w_{G_j} = u_{G_j} \nu_j$ | group-$\ell_{2,1}$ |
| GHPP$_{k_1, k_1+k_2}$ (mixed group) | $w_{G_j} = (\odot_{t=1}^{k_1}\mu_{jt})(\odot_{r=1}^{k_2}\nu_{jr})$ | $\ell_{2/k_1, 2/(k_1+k_2)}$ |
| HPowP$_k$       | $w = u \odot |v|^{k-1}$ | $\ell_{2/k}$ |

Here, $K$ is always smooth and surjective, implying every $w$ has infinitely many preimages $\xi$ such that $K(\xi) = w$.

## 3. Surrogate Smooth Penalties and Variational Equivalence

A key component is the smooth surrogate penalty $S(\xi)$ in the auxiliary space, usually a weighted $\ell_2$ norm. For HPP, $S(u,v) = \sum_j (u_j^2 + v_j^2)$. By an elementary AM-GM argument,
$$
\min_{u \odot v = \beta} (u^2 + v^2) = 2 |\beta|,
$$
meaning the surrogate penalty achieves the sparsity penalty exactly on the fiber $K^{-1}(\beta)$. More generally,
$$
R(w) = \min_{\xi: K(\xi) = w} S(\xi), \quad S(\xi) \geq R(K(\xi)),
$$
with equality for optimal $\xi$. This construction generates closed-form surrogates for:
- $\ell_{2/k}$ via HPP$_k$
- group $\ell_{2,1}$ via GHPP
- mixed $\ell_{2,2/k}$ via deeper group products
- general $\ell_{2/k}$ for real $k > 1$ via HPowP

## 4. Equivalence of Minima and Theoretical Guarantees

The surrogate objective is defined as:
$$
Q(\psi, \xi) = L(\psi, K(\xi)) + \lambda S(\xi).
$$
Under two mild conditions—local openness of $K$ at optimal $\xi$ and upper-hemicontinuity of the minimizer map $S(\xi): K(\xi) = w$—the set of local and global minima of $Q$ and $P$ coincide:
- Any local minimum $(\psi^*, w^*)$ of $P$ corresponds to local minima $(\psi^*, \xi^*)$ of $Q$ with $K(\xi^*) = w^*$.
- Conversely, local minima of $Q$ project to local minima of $P$ via $K$.
- Global minima of $P$ and $Q$ match, since $S$ majorizes $R$ and achieves equality at constrained minima.

All parametrizations and surrogates are $C^\infty$ except possibly at coordinate-wise zero; the regularizer handles these singularities. No spurious local minima are introduced.

## 5. Gradient-Based Optimization Algorithms

Hadamard-based weight smoothing enables the direct use of SGD or Adam for sparse regularized objectives. For $\ell_1$ via HPP, gradient steps proceed as follows:
- Let $w^{(t)} = u^{(t)} \odot v^{(t)}$.
- Compute $g = \partial L / \partial w$ at $w^{(t)}$.
- Update:
  - $\nabla_u Q = g \odot v^{(t)} + 2\lambda u^{(t)}$
  - $\nabla_v Q = g \odot u^{(t)} + 2\lambda v^{(t)}$
  - $u^{(t+1)} = u^{(t)} - \eta \nabla_u Q$
  - $v^{(t+1)} = v^{(t)} - \eta \nabla_v Q$

Deeper or group variants use appropriate Jacobian factors and distribute the $\lambda S$ terms. Initialization can use small random values or AM-GM matched factors. This approach requires only substituting the $w$ parameters with auxiliary variables and $\ell_2$ regularization, without the need for proximal operators or custom solvers.

## 6. Empirical Evaluation and Use Cases

Several empirical studies demonstrate the practicality of Hadamard-based weight smoothing:
- **High-Dimensional Regression:** On synthetic problems $(n=500, d=1000, s=10)$, HPP-SGD closely matches Lasso regularization paths, produces exact sparsity, and converges reliably. Direct SGD on nonsmooth $\ell_1$ is inferior, failing to produce zeros due to oscillation.
- **Sparse Neural Network Training:** Fully-connected LeNet-300-100 models trained with HPP$_k$ on MNIST, using $\ell_2$ regularization on $(u, v)$, yield weights $w$ with pronounced $\ell_1$ sparsity. After one-shot pruning, up to 99% parameter reduction is achievable with minimal loss increase. Deeper factorizations (HPP$_k$ for $k>2$) further enhance sparsity.
- **Filter-Sparse CNNs:** Applying group Hadamard powers to convolution filter groups achieves up to 90% filter removal after training from scratch, with negligible accuracy degradation.

## 7. Comparison with Existing Methodologies

Hadamard-based weight smoothing unifies various sparsity-inducing formulations:
- **Universality:** Accommodates $\ell_1$, $\ell_p$ ($p<1$), $\ell_{2,1}$, and more within a single SGD-compatible framework.
- **Theoretical Guarantees:** Ensures equivalence of all minima between original and surrogate problems, with no spurious solutions.
- **Practical Performance:** Matches specialized solvers (e.g., glmnet, SGL) in high-dimensional regression; delivers substantial sparsity in neural networks using standard SGD.
- **Implementation Simplicity:** Replaces $w$ with surrogate variables; relies only on standard $\ell_2$ regularization and smooth optimization.
- **Computational Overhead:** Increases parameter count moderately; shallow models incur minimal extra compute. For deep/large models, parameter sharing and twin initializations can mitigate costs.

Hadamard-based weight smoothing offers a plug-and-play technique for achieving exact sparse regularization via smooth objectives, applicable across a diverse range of models and compatible with modern deep learning toolkits and optimizers [2307.03571].

Source: https://www.emergentmind.com/topics/hadamard-based-weight-smoothing