Sparse Top-K via Convex Regularization

• Sparse Top-K via convex regularization is a framework that approximates NP-hard top-K selection by replacing combinatorial constraints with convex surrogates.
• It leverages methods such as ℓ₁/Group LASSO, p-norm smoothing, and dynamic programming to enable differentiable and efficient sparse optimization.
• The approach offers robust theoretical guarantees and improved empirical performance in applications like dictionary learning, regression, and neural network sparsification.

Sparse Top-K via Convex Regularization refers to a family of frameworks and algorithmic primitives for enforcing or approximating strict Top-K sparsity—a constraint forcing only the K largest-magnitude, or most relevant, entries (or groups) in a vector to be nonzero—intractable in its combinatorial form, by leveraging convex (or strongly convex) relaxations and variational surrogates. This paradigm spans dictionary learning, neural network sparsification, regression, and subset selection, providing efficient, stable, and often differentiable surrogates for the discontinuous Top-K selection operator. Core approaches include convex proxy penalties (e.g., ℓ₁), isotonic convex analysis, rank-one convexification, smoothed dynamic programming surrogates, and nonconvex but efficiently prox-friendly regularizers.

1. Combinatorial Formulation and Convex Surrogates of Top-K Sparsity

The strict Top-K constraint, defined as $\|v\|_0 \le K$ for a vector $v$, is NP-hard to incorporate directly in optimization due to its nonconvex, discontinuous support selection. The exact operator arises as the solution to the integer program

$$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$

which selects the indices of the $k$ largest coordinates of $x$ (Sander et al., 2023).

This discrete selection admits a convex relaxation via the permutahedron $P(1_k) = \{ y \in [0,1]^n : \sum_i y_i = k \}$, allowing the hard maximization to be replaced by a linear program over this capped simplex: $$\mathrm{TopKMask}(x) = \arg \max_{y \in P(1_k)} \langle x, y \rangle$$ Enforcement in practical machine learning pipelines instead uses convex proxies—primarily, entrywise ℓ₁, group-ℓ₁, or regularized support functions—or sparse-masked flows enabling gradient propagation.

2. Convex Regularization Approaches to Induce Top-K Behavior

2.1 ℓ₁ and Group LASSO-based Sequential Selection

Axiotis and Yasuda establish that soft-thresholding with a well-chosen ℓ₁ or group-ℓ₁ penalty can sequentially mimic greedy coordinate or group selection: tuning the regularization parameter to the point where exactly one (group) variable enters yields the feature/group with the largest gradient norm; iterating this routine recapitulates the exact support of the Top-K by Orthogonal Matching Pursuit (OMP), assuming restricted strong convexity and smoothness (Axiotis et al., 2023).

2.2 p-Norm Smoothing and Isotonic Convex Analysis

Blondel et al. (Sander et al., 2023) extend convex regularization using p-norm smoothing on the capped simplex constraint, resulting in the differentiable operator

$$y^*_τ(x) = \arg\min_{y \in P(1_k)} \langle -x, y\rangle + \frac{1}{\tau p}\|y\|_p^p$$

The regularized solution interpolates between a fully dense distribution (τ small) and hard Top-K sparsity (τ large, $p \searrow 1$), and enables closed-form Jacobian computation for differentiable backpropagation.

2.3 Smoothed Dynamic Programming

Mensch and Blondel, and subsequent generalizations (Vivier-Ardisson et al., 29 Jan 2026), interpret Top-K as the final value of a dynamic program (akin to knapsack), and then regularize each max operation in the recursion with a convex $\Omega$. The solution becomes a differentiable, exactly K-sparse surrogate when using suitably sparse-inducing $v$0 (e.g., Gini or Tsallis), and retains permutation equivariance only if Shannon entropy is used for $v$1.

2.4 Rank-One Convexification

In regression and subset selection, the nonconvex cardinal constraint is relaxed by Atamtürk and Gómez via semidefinite constraints corresponding to the convex hull of rank-one quadratic forms with indicator variables (Atamturk et al., 2019). This yields:

• SDP-implementable formulations with conic constraints,
• Stronger relaxations than separable penalties (e.g., MC+, perspective),
• Nonseparable, unbiased penalties $v$2 that vanish on vectors of support $v$3.

2.5 Sparse Group $v$4-max Regularization

Tao et al. (Tao et al., 2024) propose

$v$5

for each group, penalizing only those entries not in the top $v$6 in magnitude. Its convex surrogate, $v$7, is used to define a flexible, prox-friendy penalty interpolating between hard Top-K and ℓ₁.

3. Algorithmic Realizations: Proximal, Unrolled, and DP-based Schemes

Sparse Top-K convex regularization induces highly structured algorithmic schemata. In dictionary learning, Lin et al. (Lin et al., 13 Nov 2025) couple strict Top-K LISTA unrolling (nonconvex, hard mask but backprop-flow-permissive) and a convex FISTA-style encoder (LISTAConv, using ℓ₁/soft-threshold) in an alternating minimization setting, with block-coordinate updates and projected gradient or closed-form ridge steps for dictionary and classifier parameters.

For smooth p-norm convex surrogates (Sander et al., 2023), efficient solutions proceed via reduction to isotonic regression, solved optimally with $v$8 complexity: a sorting step, followed by the Pool Adjacent Violators (PAV) or Dykstra algorithm, which is vectorizable and compatible with hardware acceleration.

The dynamic programming view (Vivier-Ardisson et al., 29 Jan 2026) yields $v$9 sequential complexity, and exposes natural forward-backward recursion for value and gradient passes, supporting deterministic soft Top-K as well as ancestral sampling for hard (discrete) masks.

Sparse group k-max regularization is compatible with iterative soft-thresholding, using the groupwise operator $$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$0, and enjoys a simple fixed-point implementation with per-iteration complexity dominated by the residual update and top-$$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$1 finding per group (Tao et al., 2024).

4. Theoretical Guarantees and Convergence

Convergence of convex-regularized Top-K procedures is established in various senses:

• Under PALM (proximal alternating linearized minimization) conditions, block-alternating updates for the convex variant of dictionary learning provably descend to stationary points (Lin et al., 13 Nov 2025).
• Sequential Group LASSO selection is shown to exactly recover the greedy OMP support under restricted strong convexity and smoothness, with standard geometric convergence rates and bicriteria approximation guarantees for the minimum loss achievable with k-sparse vectors (Axiotis et al., 2023).
• Rank-one convexification is proven to deliver relaxations with integrality gaps of ≤0.4% on real data, and the regularizer is unbiased on supports of size ≤k (Atamturk et al., 2019).
• The sparse group k-max penalty satisfies local optimality and first-order stationarity under explicit conditions on the gap between top-k and residual magnitudes (Tao et al., 2024).

5. Applications and Empirical Performance

Sparse Top-K convex regularization is broadly applicable:

Application Domain	Key Method/Penalty	Empirical Outcomes / Notable Metrics
Dictionary learning	LC-KSVD2 + LISTA/LISTAConv	CIFAR-10: 95.60% (Top-K), 94.65% (convex); TinyImageNet: 88.54% (Top-K), 88.40% (convex); GPU <4GB, fast convergence (Lin et al., 13 Nov 2025)
Sparse regression	Rank-one convexification	≤0.4% gap to optimal; better test error and support recovery than Lasso/Elastic-Net; SOCP extensions scalable to p≈500 (Atamturk et al., 2019)
Neural pruning	Differentiable convex Top-K	2-3x faster, lower test error than hard mask at 90% sparsity for MLPs; improved accuracy on ViT and MoE routers (Sander et al., 2023)
Group feature selection	Sparse group k-max	Correct positive ratio (CPR) up to 96.7%; superior RMSE and feature support across group-sparsity configurations (Tao et al., 2024)

In neural network sparsification, smooth convex Top-K surrogates provide stable training and improved accuracy for heavily pruned MLPs and sparse expert routing in MoEs (Sander et al., 2023). In regression, rank-one convexification shows empirical superiority over MC+ and elastic-net, both in terms of support recovery and out-of-sample accuracy, due to nonseparable adaptivity (Atamturk et al., 2019). In structured group-sparse modeling, the group k-max approach yields sharper within-group and inter-group zeros compared to standard Lasso or Sparse Group Lasso, with lower bias and improved signal recovery (Tao et al., 2024).

6. Trade-offs: Sparsity Control, Differentiability, and Expressiveness

Sparse Top-K via convex regularization presents a spectrum between exact combinatorial selection (maximum support control, non-differentiability) and fully convex, smooth penalties (easy optimization and gradient flow, less precise support control).

• Strict Top-K operators (as in hard-masked LISTA) guarantee support size but are non-differentiable in $$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$2.
• ℓ₁/Group LASSO-based sequential surrogates guarantee support under fine-tuned regularization and provide recovery guarantees under RSC/smoothness, but may not force exactly $$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$3 nonzeros in the presence of ties or near-equal coefficients.
• Isotonic p-norm surrogates interpolate smoothly between support sizes, with exact combinatorial selection recovered as $$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$4, $$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$5 (Sander et al., 2023).
• Dynamic-programming-based surrogates allow fine-grained control over the differentiability-sparsity trade-off via regularizer choice, with Shannon entropy yielding dense outputs and, e.g., Gini or Tsallis functions enforcing sparser selections outside a band of width set by $$\mathrm{TopKMask}(x) = \arg \max_{y \in \{0,1\}^n, \sum_i y_i=k}\langle x, y \rangle$$6 (Vivier-Ardisson et al., 29 Jan 2026).
• Rank-one convexification directly targets unbiasedness and approximation tightness, at the cost of requiring SDP or SOCP machinery.
• Sparse group k-max regularization bridges strictly combinatorial support with smooth shrinkage for less significant entries.

A plausible implication is that the optimal method for a given application depends on the requirements for gradient-based optimization, precise cardinality constraints, and computational scalability.

7. Connections, Extensions, and Open Problems

Sparse Top-K via convex regularization unifies a broad spectrum of sparse modeling literature:

• Connects classical dictionary learning and sparse regression with deep, gradient-based optimization, using both unrolled architectures and convex surrogates (Lin et al., 13 Nov 2025).
• Demonstrates that iterative ℓ₁ or group-wise selection, when properly parameterized, can achieve the same support selection as greedy combinatorial matching (Axiotis et al., 2023).
• Extends differentiable sparse masking to large-scale neural systems, with efficient hardware-accelerated (PAV, Dykstra) and DP-based algorithms (Sander et al., 2023Vivier-Ardisson et al., 29 Jan 2026).
• Motivates further theoretical exploration of nonseparable, unbiased regularizers beyond the classical perspective, as exemplified in rank-one convexification (Atamturk et al., 2019).
• Enriches the design space of sparsity-inducing penalties through group-aware, locally nonconvex, but shrinkage-friendly penalties such as sparse group k-max (Tao et al., 2024).

Open problems include the characterization of optimal regularizer shapes under combinatorial constraints, scalability of rank-one or SDP-based methods to high dimensions in real-time applications, and integration of strongly sparse, group-structured regularizers into automatic differentiation pipelines for deep learning at scale.