ULPENS: Smooth, Tunable Sparsity Penalty

• ULPENS is a smooth, non-convex sparsity-inducing penalty that interpolates between the l1 norm and a selective suppression function, enabling effective sparse optimization.
• Its formulation via ultra-discretization of smoothed absolute value and minimum functions provides continuous differentiability and ordered adaptive weighting for gradient-based methods.
• ULPENS demonstrates superior performance in sparse recovery, image deblurring, and structured sparsity tasks by reducing bias and preserving significant signal components.

ULPENS designates a class of non-convex, non-separable, smooth sparsity-inducing penalty functions for sparse optimization, constructed via an ultra-discretization formula (Akaishi et al., 24 Sep 2025). The penalty enables continuous interpolation between the traditional $\ell_1$ norm and a non-convex selective suppressing function by tuning its inherent parameters. ULPENS offers ordered adaptive weighting of signal components, efficiently suppressing smaller magnitudes while better preserving large coefficients, and remains differentiable—allowing the use of gradient-based optimization algorithms and quasi-Newton methods.

1. Mathematical Construction and Ultra-discretization Formula

ULPENS is derived from two smooth approximations:

• Absolute value smoothing: For $t \in \mathbb{R}$,

$$h_\mathrm{abs}^\lambda(t) = \lambda \, \ln \left( \exp(t/\lambda) + \exp(-t/\lambda) \right)$$

with $$\lim_{\lambda \to 0} h_\mathrm{abs}^\lambda(t) = |t|$$. Its derivative is $\tanh(t/\lambda)$.

• Minimum smoothing: For $x \in \mathbb{R}^N$,

$$h_\mathrm{min}^\mu(x) = -\mu \ln \left( \sum_{n} \exp(-x_n/\mu) \right)$$

with $$\lim_{\mu \to 0} h_\mathrm{min}^\mu(x) = \min_n x_n$$.

ULPENS penalty:

Given $x \in \mathbb{R}^N$,

$$\psi(x) = -N\mu \ln \left( \sum_{n} \left[ \exp(x_n/\lambda) + \exp(-x_n/\lambda) \right]^{- \lambda / \mu} \right)$$

By adjusting $\lambda$ and $\mu$, ULPENS interpolates between the $\ell_1$ norm (convex, separable) and a selective suppressing function (non-convex, non-separable). Specifically:

• Large $\mu$: $\psi(x)$ approximates the $\ell_1$ norm.
• Small $\mu$: $\psi(x)\approx N \min_n |x_n|$.

2. Smoothness, Differentiability, and Gradient Structure

ULPENS is continuously differentiable due to its smooth construction. The gradient with respect to $x$ is given by

$$[\nabla \psi(x)]_n = N \, \varphi_n^{(x)} \, \tanh(x_n/\lambda)$$

where

$$\varphi_n^{(x)} = \frac{ \left[ \exp(x_n/\lambda) + \exp(-x_n/\lambda) \right]^{- \lambda/\mu} } { \sum_{k} \left[ \exp(x_k/\lambda) + \exp(-x_k/\lambda) \right]^{- \lambda/\mu} }$$

Ordered weighting: If $|x_n| \leq |x_m|$, then $\varphi_n^{(x)} \geq \varphi_m^{(x)}$. Thus, small-magnitude components are penalized more harshly, leading to selective suppression.

An upper bound for the Lipschitz constant of the gradient is derived, granting rigorous control over algorithmic step size in gradient optimization.

3. Interpolation properties and Control via Parameters

ULPENS offers a smooth, tunable transition between regimes:

• $\mu \to \infty$, $\lambda \to 0$: $\nabla \psi(x) \to \mathrm{sign}(x)$, i.e., the gradient of the $\ell_1$ norm.
• $\mu \to 0$, $\lambda$ finite: Penalty approaches selective suppression, strongly penalizing smaller values and retaining larger components.

An additional scale-normalized parameter $\nu$ can be introduced to set $\mu$ proportional to the maximum magnitude of $x$, yielding operational invariance to signal scale and stabilizing performance across datasets.

4. Practical Performance in Sparse Optimization

ULPENS is evaluated through numerical experiments against the $\ell_1$ norm, generalized minimax concave (GMC) penalty, and ordered weighted $\ell_1$ (OWL) penalty:

• Sparse signal recovery: ULPENS overcomes the known underestimation bias in $\ell_1$ minimization, better preserving high-amplitude coefficients.
• Optimization efficiency: Smoothness enables acceleration via Nesterov's momentum and full utilization of quasi-Newton methods, which are not compatible with non-smooth penalties.
• Structured sparsity: Substituting $\ell_1$ in group penalties (e.g., $\ell_{2,1}$) with ULPENS or groupwise ULPENS improves sparse recovery in multi-channel and dictionary learning settings.
• Image deblurring: ULPENS regularization preserves image edges and details more faithfully than total variation or $\ell_1$ alternatives.

Lower normalized mean square error values and faster convergence rates are observed when using ULPENS, particularly in demanding large-scale or real-time optimization environments.

5. Applications and Methodological Implications

ULPENS is relevant across domains requiring sparsity with minimization of large penalty-induced bias:

• Compressed sensing: Preserves true signal amplitudes while encouraging sparse representations.
• Denoising and source localization: Selectively eliminates low-level noise without suppressing salient components.
• Image processing: Enhances edge preservation and feature recovery compared to conventional convex regularizers.
• Structured sparsity: Generalizes readily to group-structured variants for use in multi-dimensional signal processing and statistical learning.

Smoothness and Lipschitz continuity of the gradient promote compatibility with efficient second-order algorithms, optimizing for both speed and solution quality.

6. Theoretical and Operational Significance

The ultra-discretization basis of ULPENS offers a principled means for continuous, parameterized decomposition between convex and non-convex penalty regimes. Ordered weighting focuses suppression on low-amplitude variables, and non-separability allows joint treatment of signal components for enhanced selectivity.

The balance of smoothness, adaptivity, and non-convexity, together with demonstrable effectiveness in various scenarios, positions ULPENS as a robust alternative to classical sparsity-inducing methods, and a viable regularizer for diverse optimization problems where conventional $\ell_1$ norms are insufficient. The penalty’s properties facilitate new directions in sparsity modeling, efficient computation, and improved reconstruction fidelity in high-dimensional statistical inference.