BLOC: A Global Optimization Framework for Sparse Covariance Estimation with Non-Convex Penalties

Abstract: We introduce BLOC (Black-box Optimization over Correlation matrices), a general framework for sparse covariance estimation with non-convex penalties. BLOC operates on the manifold of correlation matrices and reparameterizes it via an angular Cholesky mapping, transforming the positive-definite, unit-diagonal constraint into an unconstrained search over a Euclidean hyperrectangle. This enables gradient-free global optimization of diverse objectives, including non-differentiable or black-box losses, using a pattern search routine with adaptive coordinate polling, run-wise restarts to escape local minima, and leveraging up to $d(d-1)$ parallel threads when optimizing a $d$-dimensional correlation matrix. The method is penalty-agnostic and ensures that every iterate is a valid correlation matrix, from which covariance estimates are obtained. We establish convergence guarantees, including stationarity, probabilistic escape from poor local minima, and sublinear rates under smooth convex losses. From a statistical perspective, we prove consistency, convergence rates, and sparsistency for penalized correlation estimators under general conditions, extending sparse covariance theory beyond the Gaussian setting. Empirically, BLOC with nonconvex penalties such as SCAD and MCP outperforms leading estimators in both low- and high-dimensional regimes, achieving lower estimation error and improved sparsity recovery. A parallel implementation enhances scalability, and a proteomic network application demonstrates robust, positive-definite sparse covariance estimation.

• The paper introduces a penalty-agnostic, global optimization framework for sparse covariance estimation using an angular reparameterization that guarantees valid positive definite matrices.
• It employs a derivative-free Recursive Modified Pattern Search in an unconstrained angular space to effectively navigate nonconvex landscapes and escape local minima.
• Empirical results demonstrate that BLOC outperforms traditional ℓ₁-penalized methods in accuracy, scalability, and support recovery in high-dimensional settings.

BLOC: A Global Optimization Framework for Sparse Covariance Estimation with Non-Convex Penalties

Problem Context and Motivation

Covariance estimation under sparsity and high-dimensionality is fundamental for statistical inference across fields such as genomics, finance, and signal processing. Conventional estimators become unstable as dimensionality increases, prompting the widespread adoption of regularization, particularly $\ell_1$-penalized approaches. However, $\ell_1$-penalties introduce shrinkage bias, compromising the recovery of moderate or strong correlations. Nonconvex penalties, such as SCAD and MCP, are known to mitigate this bias and provide improved support recovery (sparsistency), but their optimization is challenging due to nonconvexity and inherent constraints of the correlation matrix manifold. Existing solvers are largely local and often objective- or penalty-specific, limiting their applicability to broader loss functions or black-box objectives.

Methodological Formulation

BLOC (Black-box Optimization over Correlation matrices) addresses these challenges by translating covariance estimation into a global black-box optimization problem over the manifold $\mathcal{C}_d$ of correlation matrices. The method uses a two-pronged approach: first, a bijective angular Cholesky reparameterization converts the constrained search over correlation matrices (which are symmetric, positive definite, and unit-diagonal) into an unconstrained search over a Euclidean hyperrectangle. Second, BLOC applies a derivative-free global optimization routine—Recursive Modified Pattern Search (RMPS)—to efficiently search this angular space, balancing local refinement with global exploration.

Figure 1: Angular bijection reparameterizes the space of correlation matrices via Cholesky decomposition and hyperspherical coordinates, mapped into an unconstrained parameter space for black-box optimization.

This reparameterization ensures that any perturbation in the unconstrained space is mapped to a valid correlation matrix, avoiding the need for projection or penalty enforcement at each step. BLOC is further equipped to optimize arbitrary, potentially nondifferentiable objectives and is agnostic to penalty structure, facilitating application across user-defined loss functions—including those not stemming from classical likelihoods.

Algorithmic Framework

The BLOC algorithm alternates between coordinate-wise pattern searches in the angular parameter space and adaptive step-size updates, proceeding across multiple runs to escape local minima. At each iteration, up to $2N$ candidate perturbations are evaluated in parallel, where $N = d(d-1)/2$ is the number of free parameters in the angular representation for a $d$-dimensional matrix. The algorithm selects the best direction, reduces the step size when no improvement is observed, and restarts from the best solution to promote global search.

Figure 2: The BLOC algorithm's flowchart highlights parallel evaluation, adaptive step-size control, and systematic restarts for robust global optimization.

Key features include:

• Penalty-agnostic objective evaluation: The procedure only requires evaluability, not differentiability or explicit analytic form.
• Built-in parallelization: Candidate perturbations are independent and naturally distributed across computing threads.
• Guarantee of positive definiteness: Every iterate is a valid correlation matrix due to the angular parametrization.

Theoretical Guarantees

The paper establishes rigorous statistical and algorithmic guarantees for BLOC:

• Consistency and Convergence Rates: For any local minimizer of the penalized objective, BLOC attains optimal Frobenius-norm convergence rates and operator-norm rates under general losses, extending classical sparse theory beyond the Gaussian setting.
• Sparsistency: Under appropriate scaling and regularity conditions, BLOC recovers the true sparsity pattern in the off-diagonal entries with high probability, matching the strongest existing theoretical results for nonconvex penalties.
• Global Reachability and Sublinear Rate: The RMPS-based optimization enjoys stationarity guarantees and, under a randomized restart mechanism, can reach any prescribed neighborhood of the global minimizer with probability one. For smooth convex objectives, BLOC achieves an $O(1/r)$ rate, comparable to first-order methods.
• Escape from Local Minima: Restarts and adaptive polling ensure probabilistic escape from poor local minima, an essential property for highly nonconvex landscapes.

Empirical Evaluation

Extensive numerical experiments on classical benchmarks (Ackley, Griewank, Rosenbrock, and Rastrigin embedded in correlation matrix search spaces) demonstrate superior optimization accuracy and scalability relative to competing solvers, including MATLAB’s fmincon variants and the Manopt toolbox.

In simulation studies on synthetic covariance structures under Gaussian likelihood and Frobenius norm loss, BLOC (with SCAD and MCP penalties) outperforms leading estimators in both low- and high-dimensional regimes ($n>d$ and $d\ge n$), achieving reduced estimation error, improved edge selection metrics (TPR, FPR, MCC), and guaranteed positive definiteness. Notably, $\ell_1$-penalized baselines often failed or became unstable at high dimensions, while BLOC maintained robustness.

Biological Application

BLOC's penalty-agnostic and objective-agnostic flexibility is showcased in a pan-gynecologic proteomics network analysis using pathway-informed penalties. By penalizing only cross-pathway protein pairs, BLOC recovers biologically expected within-pathway correlations while highlighting tumor-specific integration patterns across pathways in cancers such as BRCA, UCEC, OV, CESC, and UCS.

Figure 3: Estimated sparse correlation heatmap for BRCA, illustrating preservation of within-pathway modules and selective cross-pathway edge recovery using structured penalization.

Practical and Theoretical Implications

Practically, BLOC’s parallelization and derivative-free mechanics make it suitable for large-scale, high-dimensional covariance estimation tasks, including settings with black-box or user-defined objectives. Theoretically, the framework unifies sparse covariance estimation under nonconvex penalties across general loss functions and expands the landscape of statistical guarantees beyond classical likelihood-based formulations.

Future developments may focus on acceleration strategies, integration with hyperparameter selection, and extension to broader classes of positive-definite matrix manifolds beyond correlations.

Conclusion

BLOC presents a penalty-agnostic, globally aware, and parallelizable framework for sparse covariance estimation with nonconvex penalties on the correlation matrix manifold. Its methodological innovations—angular reparameterization and recursive global search—yield both strong statistical performance and broad applicability, supported by rigorous convergence guarantees and empirical superiority over a range of competing methods. The approach enables adaptive domain-informed penalization and robust global optimization, suggesting substantial utility in high-dimensional statistical inference and structured network analysis.