Stabilizing Private LASSO under Heterogeneous Covariates via Anisotropic Objective Perturbation

Abstract: We study high-dimensional LASSO under differential privacy via objective perturbation with heterogeneous covariate scales. In practical scenarios, covariates often exhibit diverse scales; however, standard preprocessing is problematic under privacy constraints, as it consumes additional privacy budget. This heterogeneity induces effective anisotropy in the objective perturbation via the inverse Gram matrix of covariates, which can degrade the stability and accuracy of algorithms. To address this, we propose a Gram-based anisotropic objective perturbation, a ``pre-distortion" strategy that counteracts the distortion from the covariate structure to restore isotropy in the estimation process. Using an Approximate Message Passing (AMP) framework and state evolution analysis, we demonstrate that our proposed perturbation significantly stabilizes convergence and improves both statistical efficiency and privacy performance compared to standard uniform noise injection. Our results provide theoretical insights into designing stable and efficient private estimators without relying on data-dependent preprocessing.

• The paper introduces Gram-based anisotropic objective perturbation to address instability in private LASSO with heterogeneous covariates.
• It employs a pre-distortion noise scaling mechanism based on covariate variances to restore effective isotropy in the estimation process.
• Empirical and theoretical analyses show that the method improves convergence, stability, and the privacy-utility trade-off compared to isotropic noise injection.

Stabilizing Private LASSO under Heterogeneous Covariates via Anisotropic Objective Perturbation

Motivation and Problem Setting

The investigation centers on the high-dimensional LASSO under differential privacy (DP), leveraging objective perturbation to preserve privacy. A crucial issue arises in real-world data: covariates typically exhibit heterogeneous scales. Standard preprocessing (e.g., whitening, standardization) is infeasible under DP, as it consumes privacy budget and exposes global statistics. This heterogeneity induces effective anisotropy in the optimization landscape, especially when objective perturbation is applied. The injected noise becomes amplified along sensitive directions determined by the inverse Gram matrix, leading to instability and degraded estimator performance. The paper (2605.01492) addresses this challenge by proposing anisotropic (Gram-based) objective perturbation, restoring isotropy in the estimation process to improve stability, accuracy, and privacy.

Perturbation Design and Theoretical Analysis

Traditional objective perturbation injects isotropic noise, but the estimator’s sensitivity is governed by the spectral characteristics of $F^\top F$. Thus, even isotropic noise is transformed into effective anisotropy. The study formally demonstrates that objective perturbation amplifies noise in directions of high sensitivity, undermining algorithmic stability and degrading prediction accuracy.

In response, a pre-distortion strategy is introduced: Gram-based anisotropic perturbation. The noise components are scaled according to the covariate variances, compensating for the distortion imposed by the inverse Gram matrix. This results in more homogeneous effective noise, mitigating instability. The design entails allocating noise variance $\sigma_{\eta,i}^2$ proportionally to $v_i^2$, ensuring that after the transformation by $(F^\top F)^{-1}$ the effective perturbation is (approximately) isotropic.

(Figure 1)

Figure 1: Schematic of isotropic versus Gram-based perturbation, showing distortion by the inverse Gram matrix and restoration of isotropy through pre-distorted noise.

Approximate Message Passing (AMP) Under Heterogeneous Covariates

AMP is utilized to analyze LASSO estimators under privacy constraints in the high-dimensional regime ($N,M \gg 1$, $\alpha = M/N$). The model assumes $F_{\mu i} = \tilde{F}_{\mu i} v_i$, with $v_i$ drawn from either uniform or log-normal distributions to simulate scale heterogeneity. The objective perturbation enters the AMP denoising step, and the Gram-based strategy prompts component-wise thresholding: weak covariates (small $v$) receive enhanced thresholds, effectively suppressing high-sensitivity directions from privacy leakage.

(Figure 2)

Figure 2: Distribution of pre-thresholding estimates and shrinkage behavior for isotropic versus Gram-based perturbations.

The AMP algorithm incorporates these adaptations and demonstrates substantial improvements in stability, allowing lower values of the sparsity parameter $\lambda$.

Empirical and State Evolution Results

Comprehensive evaluation—combining AMP simulations and state evolution (SE) theory—reveals that Gram-based perturbation stabilizes convergence across a wide range of noise strengths, in contrast to isotropic noise, which triggers instability at modest levels of injected randomness. This behavior persists across both uniform and log-normal covariate distributions.

Figure 3: Generalization error versus privacy noise strength; Gram-based perturbation yields more gradual degradation and greater stability.

The $\sigma_{\eta,i}^2$0-dependence of generalization error and sparsity also confirms increased flexibility in parameter tuning under Gram-based noise allocation. Especially for uniform covariate settings, Gram-based perturbation suppresses instability induced by frequent low-scale (high-sensitivity) covariates.

Figure 4: Dependence of prediction error and fraction of non-zero components on $\sigma_{\eta,i}^2$1; Gram-based perturbation mitigates instability and enables denser solutions.

Privacy-Utility Trade-Off and KL Analysis

Privacy is quantified using On-Average KL divergence, which measures output sensitivity under membership inference attacks. Gram-based perturbation consistently achieves superior privacy-accuracy trade-offs compared to isotropic alternatives: both generalization error and KL divergence attain lower values, and minimal points approach the origin.

Figure 5: Privacy-accuracy trade-off for uniform and log-normal covariate scenarios; Gram-based perturbation optimizes the trade-off curve.

Theoretical SE results provide further insights, including the AMP stability condition: convergence persists only if the estimated fraction of non-zero components $\sigma_{\eta,i}^2$2 remains below $\sigma_{\eta,i}^2$3, substantiating empirical findings.

Implications and Future Directions

The findings underscore that uniform noise allocation—typical in objective perturbation—readily destabilizes LASSO estimators under heterogeneous covariates. In contrast, Gram-based anisotropic design delivers robust convergence, improved utility, and stronger privacy guarantees. While the requirement for Gram-matrix statistics poses additional privacy consumption, this perturbation framework establishes guiding principles for stability-aware privacy mechanism design.

Future directions include devising data-independent anisotropic designs (e.g., leveraging structural priors or auxiliary public data), and extending analysis to general non-diagonal covariances using Vector AMP (VAMP)—an approach robust to arbitrary, rotationally invariant matrices. Connections to Perturb-and-MAP, compressive sensing with matrix uncertainty, and geometry-aware optimization (such as natural gradients) offer fertile ground for further research and application.

Conclusion

The paper rigorously analyzes the impact of perturbation design in private LASSO under heterogeneous covariates, demonstrating that Gram-based anisotropic objective perturbation substantially improves algorithmic stability, estimator accuracy, and privacy protection. The AMP and SE analyses collectively validate the efficacy of pre-distortion noise allocation, and the results provide foundational guidance for designing privacy mechanisms in high-dimensional, heterogeneously scaled datasets. The integration of anisotropic noise design with scalable inference methods is poised to influence broader developments in private machine learning and robust statistical estimation.