Analytical Sparsity Control Methods

• Analytical sparsity control objectives are mathematical frameworks that impose sparsity on control solutions while ensuring system performance and structural compliance.
• They employ techniques such as ℓ0 constraints, ℓ1 relaxations, and combinatorial penalties to efficiently enforce nonzero limits in high-dimensional problems.
• These methods find applications in control systems, machine learning, and PDE-constrained optimization, providing formal guarantees and optimal trade-offs between sparsity and performance.

An analytical sparsity control objective refers to a mathematically precise framework for inducing, quantifying, and optimizing sparsity in decision variables—typically control laws, feedback matrices, actuation schedules, or model parameters—so as to simultaneously achieve performance goals and enforce explicit structural constraints in complex systems. Rigorous analytical objectives of this type are central in the design, synthesis, and verification of controllers, estimators, or learning architectures where parsimony, communication overhead, or hardware constraints are decisive. These objectives arise in a wide range of fields, including large-scale control, machine learning, PDE-constrained optimization, and combinatorial decision-making, and are usually expressed via nonconvex functionals, regularization terms, combinatorial penalties, or hard constraints that target solutions with a specified number of nonzeros, minimal active support, or maximal “hands-off” intervals.

1. Formalization of Analytical Sparsity Objectives

Analytical sparsity control objectives are formulated by incorporating structural terms or hard constraints into an optimization problem to promote solutions with the desired sparsity level. The archetypal forms include:

• $\ell_0$ Pseudo-norm Constraints: $\|w\|_0 \leq k$, enforcing at most $k$ nonzero entries.
• Combinatorial Cardinality Penalties: $\lambda \cdot \mathrm{card}(K)$ or mixed objectives such as $\min f(x) + \lambda \|x\|_0$.
• Sparsity-Promoting Regularization: Convex (e.g., $\ell_1$ norm, group lasso) and nonconvex ($\ell_p$, $p \in (0,1)$, indicator, or block-based norms).
• Constraint-Driven Formulations: Direct constraints on expected density or fraction of active variables (e.g., $\mathbb{E}_z[\|z\|_0]/n \leq \epsilon$ in neural network pruning).
• Group/Support-Preserving Constraints: Combination of hard sparsity with convex structure ($w \in T$), e.g., sector or group constraints in portfolio optimization or signal processing.

An explicit example from (Vazelhes et al., 10 Jun 2025):

$\|w\|_0 \leq k$0

Here, $\|w\|_0 \leq k$1 is the loss or risk, $\|w\|_0 \leq k$2 is the sparsity level, and $\|w\|_0 \leq k$3 is a convex, support-preserving set.

2. Analytical Methodologies and Trade-offs

Contemporary analytical sparsity control balances combinatorial nonconvexity and numerical tractability via the following methodologies:

• Two-Step Projection (2SP): As in (Vazelhes et al., 10 Jun 2025), enforce exact sparsity ($\|w\|_0 \leq k$4) via hard-thresholding to $\|w\|_0 \leq k$5 largest entries, followed by Euclidean projection onto $\|w\|_0 \leq k$6 (convex additional constraint), i.e.,

$\|w\|_0 \leq k$7

This structure decouples sparsity from convex side constraints and avoids expensive joint projection onto the intersection.

• Homotopy and Reweighted $\|w\|_0 \leq k$8 Techniques: Relax cardinality constraints to $\|w\|_0 \leq k$9 penalizations, solving a path of regularized problems (as in (Dörfler et al., 2013)) to reveal sparsity/performance trade-offs, typically using ADMM or iterative thresholding.
• Analytical Subdifferentiation: For nonconvex, non-Lipschitz sparsity functionals (e.g., $k$0, $k$1), provide exact Fréchet, limiting, and singular subdifferential characterizations (Mehlitz et al., 2021), which are essential for deriving first-order optimality conditions in infinite-dimensional spaces.
• Support-Preserving Structure Exploitation: In high-dimensional control with sparsity patterns exploited at algorithmic and theoretical levels (e.g., soft-thresholding or semiparametric least squares in partially controllable systems (Efroni et al., 2021)).
• Analytically Tunable Parameters: Introduction of explicit parameters controlling the trade-off (e.g., $k$2 in (Vazelhes et al., 10 Jun 2025) for sparsity/optimality, shape controller $k$3 in (Deng et al., 2020) for sparsity of ternary weights).

3. Analytical Guarantees and Theoretical Results

The field establishes quantitative guarantees that characterize the trade-off between the degree of sparsity, feasibility with respect to side constraints, and the sub-optimality in objective value:

• Global Convergence Guarantees: Under standard restricted strong convexity/smoothness assumptions, methods such as two-step projection for IHT provide bounds of the form

$k$4

where $k$5 is a global minimizer and $k$6 quantifies relaxation in sparsity (Vazelhes et al., 10 Jun 2025).

• Three-Point Lemmas in Nonconvex Settings: Extensions of the classical three-point inequality are constructed for hard-thresholding plus convex projection, serving as the analytical backbone for global convergence proofs even under nonconvex, combinatorial sparsity constraints (Vazelhes et al., 10 Jun 2025).
• Penalty/Constraint Equivalence: Exact equivalence between nonconvex $k$7 objectives and convex relaxations (e.g., $k$8), as in maximum hands-off control (Nagahara, 2014), under specific controllability and system regularity conditions.
• Performance-Sparsity Frontiers: Analytical expressions delineating the trade-off surface between closed-loop performance and the number of retained nonzero feedback links or actuators, e.g., via regularization path or homotopy methods (Dörfler et al., 2013, Guo et al., 2022).
• Exact Subdifferential Calculi: Providing formulas for generalized derivatives that can be inserted into optimality systems (variational inequalities), allowing analysis and synthesis in systems with $k$9 or nonconvex $\lambda \cdot \mathrm{card}(K)$0 functionals (Mehlitz et al., 2021).

4. Representative Algorithms and Implementation

Classical and recent algorithms designed to address the analytical sparsity control objective include:

Method	Sparsity Mechanism	Key Properties
Iterative Hard-Thresholding (IHT) with 2SP	Hard $\lambda \cdot \mathrm{card}(K)$1 enforcement + projection	Global convergence, modular decoupling (Vazelhes et al., 10 Jun 2025)
Homotopy/ADMM for $\lambda \cdot \mathrm{card}(K)$2 regularization	Relaxed sparsity via $\lambda \cdot \mathrm{card}(K)$3 penalty	Progressive system pruning, path from dense to sparse (Dörfler et al., 2013)
Reweighted IRLS/Newton-CG for $\lambda \cdot \mathrm{card}(K)$4 norm	Sparse actuator support via IRLS	Shared support under uncertainty (Li et al., 2018)
Proximal Alternating Linearized Minimization (PALM)	Combined cardinality and performance constraints	Mixes robust $\lambda \cdot \mathrm{card}(K)$5 control with strict sparsity (Lian et al., 2019)
Support-Preserving Estimation	Soft-thresholding / semiparametric LS	Extracts minimal relevant model in high dimensions (Efroni et al., 2021)

Global guarantees require careful tuning of algorithmic hyperparameters (e.g., sparsity relaxation $\lambda \cdot \mathrm{card}(K)$6) and may employ adaptive per-step projections, line searches, or stochastic variants to handle inexact or zeroth-order (derivative-free) settings.

5. Applications and Impact

Analytical sparsity control objectives are critical in applications such as:

• Control Architecture Design: Structural feedback synthesis in power grids (Dörfler et al., 2013), stochastic linear systems (Guo et al., 2022), or cyber-physical systems over shared networks (Negi et al., 2019).
• Actuator and Sensor Placement: Sparse actuation or optimized sensor selection in high-dimensional models (Li et al., 2018, Kaiser et al., 2016).
• Machine Learning Model Compression: Directly controlling network sparsity in neural parameter pruning, ternary weight design, or activation sparsity (Deng et al., 2020, Gallego-Posada et al., 2022, Khan et al., 2019).
• Robust and Adaptive Control: Guaranteeing mean-square stability and robustness in the presence of noise with the fewest actuators possible (Guo et al., 2022).
• Combinatorial Decision Processes: Portfolio management, treatment planning, or any setting where sparsity and additional business, risk, or regulatory constraints must jointly be enforced (Cheng et al., 2022, Weisenthal et al., 2023).

These objectives offer provable guidelines for the trade-offs between parsimony and performance, enabling interpretable, resource-efficient designs.

6. Extensions and Open Directions

Recent advances are extending analytical sparsity control to:

• Nonconvex and Non-Lipschitz Domains: True $\lambda \cdot \mathrm{card}(K)$7 and nonconvex $\lambda \cdot \mathrm{card}(K)$8 functionals, with subdifferential calculus on Lebesgue spaces for PDE-constrained and infinite-dimensional settings (Mehlitz et al., 2021).
• Stochastic and Gradient-Free Regimes: Zeroth-order IHT with two-step projections, ensuring the removal of system errors previously inherent to stochastic/gradient-free methods (Vazelhes et al., 10 Jun 2025).
• Adaptive and Hierarchical Sparsity: Jointly controlling overall and group-wise sparsity or enforcing structured patterns (block, low-rank, or support-preserving constraints).
• Sparsity/Performance/Efficiency Frontiers: Analytically tracing the boundary of achievable solutions as a function of imposed sparsity (e.g., via the parameter $\lambda \cdot \mathrm{card}(K)$9 or enforced sparsity targets), supporting end-to-end system co-design.

7. Mathematical Underpinnings and Practical Considerations

Key mathematical and computational elements supporting analytical sparsity control include:

• Trade-off Quantification: Parameters (e.g., $\min f(x) + \lambda \|x\|_0$0, $\min f(x) + \lambda \|x\|_0$1, $\min f(x) + \lambda \|x\|_0$2) governing sparsity/optimality.
• Support-Preserving Projections: Formal characterizations of feasible sets amenable to modular projection algorithms.
• Complexity and Scalability: Guarantees for per-iteration and total computational complexity (e.g., optimal $\min f(x) + \lambda \|x\|_0$3 in projection-free methods (Cheng et al., 2022)).
• Interpretability and Structure Identification: The ability to identify and exploit minimal support, controller architecture, or relevant subspaces analytically, enabling data-efficient estimation and interpretable decision rules.

In summary, the analytical sparsity control objective synthesizes rigorous mathematical formulations, algorithmic strategies, and explicit trade-off quantification to achieve structured, minimal, and efficient solutions in control, estimation, and learning, under explicit and tunable sparsity constraints or penalties. Recent methods deliver global optimality bounds, transparent trade-off curves, and practical algorithms for high- and infinite-dimensional systems, with extensions across both deterministic and stochastic optimization landscapes.