Nonnegative Garrote on Kernel (NGK)

• NGK is a sparsity-oriented variable selection method that combines kernel methods and nonnegative garrote regularization to recover relevant predictors in complex, high-dimensional data.
• The approach constructs an additive kernel with nonnegative scale parameters and uses an ℓ1-type penalized least-squares loss to achieve consistent and sparse recovery.
• An efficient coordinate-descent/backfitting algorithm along with resampling techniques enhances NGK's stability and performance in both regression and classification challenges.

The Nonnegative Garrote on Kernel (NGK) is a sparsity-oriented variable selection approach for nonadditive nonparametric models, explicitly developed to address challenges in recovering variable relevance in the presence of high-dimensional predictors and complex, unknown interaction structures. NGK integrates kernel methods with a nonnegative garrote regularization scheme, operating on an additive decomposition of kernel similarity matrices over the predictors. The framework accommodates both additive and nonadditive model structures, leverages efficient optimization via coordinate descent/backfitting, and supports robust variable selection through resampling techniques. The method is rooted in theory guaranteeing square root consistency and sparsistency under suitable conditions (Fang et al., 2012).

1. Model Formulation and Kernel Construction

Consider $n$ centered responses $y=(y_1,\dots,y_n)^\top$ and $p$ centered predictors $X=[x_{ij}]_{n\times p}$. The function $f$ to be estimated is modeled in a reproducing kernel Hilbert space (RKHS) as

$f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$

where $K$ is a Gram (kernel) matrix ($K \in \mathbb{R}^{n \times n}$). The NGK approach constructs $K$ additively over the $p$ inputs by introducing nonnegative scale parameters $y=(y_1,\dots,y_n)^\top$0:

$y=(y_1,\dots,y_n)^\top$1

with each $y=(y_1,\dots,y_n)^\top$2 or, more generally, each similarity matrix $y=(y_1,\dots,y_n)^\top$3 representing the kernelized contribution of predictor $y=(y_1,\dots,y_n)^\top$4. For example, under a Gaussian kernel, $y=(y_1,\dots,y_n)^\top$5. The $y=(y_1,\dots,y_n)^\top$6 are scale parameters reflecting the relevance of each predictor and serve as targets for inducing sparsity.

2. Objective Function and Penalization Strategy

NGK employs a penalized criterion combining least-squares kernel-machine loss with an $y=(y_1,\dots,y_n)^\top$7-type penalty:

$y=(y_1,\dots,y_n)^\top$8

subject to $y=(y_1,\dots,y_n)^\top$9. The problem can be formulated in two equivalent ways:

• Joint minimization over $p$0: Directly optimizing both coefficients and regularization scales.
• Profiling out $p$1: Setting $p$2, yielding a reduced criterion

$p$3

which is minimized over $p$4 (Fang et al., 2012).

The additive structure of $p$5 and the $p$6-penalty on $p$7 provide variable selection by automatically shrinking unimportant variables to zero. This approach supports both additive and nonadditive effects and encompasses the linear nonnegative garrote and related kernel learning methods as special cases.

3. Optimization Algorithm and Computational Considerations

NGK adopts an efficient coordinate-descent/backfitting algorithm for optimizing $p$8. At each iteration, $p$9 is updated holding the remaining coordinates and the initial $X=[x_{ij}]_{n\times p}$0 fixed. The update for each $X=[x_{ij}]_{n\times p}$1 uses a local linearization:

$X=[x_{ij}]_{n\times p}$2

where $X=[x_{ij}]_{n\times p}$3. Defining $X=[x_{ij}]_{n\times p}$4 and the residual $X=[x_{ij}]_{n\times p}$5, the coordinate-wise update is

$X=[x_{ij}]_{n\times p}$6

Iterations proceed over a decreasing path of $X=[x_{ij}]_{n\times p}$7. For linear kernels, per-cycle computational complexity is $X=[x_{ij}]_{n\times p}$8; for Gaussian kernels, naive computation is $X=[x_{ij}]_{n\times p}$9, although low-rank or sparse approximations can substantially reduce this burden.

4. Theoretical Properties: Consistency and Sparsistency

NGK achieves strong theoretical guarantees under regularity and smoothness conditions:

• Root-$f$0 Consistency: If $f$1, there exists a local minimum $f$2 such that

$f$3

where $f$4 is the true sparse vector (Fang et al., 2012).

• Sparsistency (Exact Recovery): Given a $f$5-consistent initial $f$6 and an incoherence condition on the derivative matrices (a kernel analog of the lasso irrepresentable condition), and if $f$7, then with high probability,
  • The active set is at least recovered ($f$8),
  • Exact support recovery ($f$9) holds if the minimal nonzero $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$0 exceeds an explicit error bound $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$1.

Karush-Kuhn-Tucker conditions for the solution’s sign-consistency are derived explicitly in terms of the kernel derivatives $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$2.

5. Resampling and Variable Selection Enhancement

To buttress stability and power in variable selection, two resampling procedures are introduced:

• m-out-of-n bootstrap (large $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$3): At each bootstrap sample of size $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$4, the NGK path is run and variables are selected via metrics such as BIC. Variable inclusion frequencies across resamples guide final selection, typically using a threshold (e.g., 60–80%).
• Residual Permutation (small $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$5): After fitting NGK to obtain residuals, permutation of residuals is used to generate new target vectors $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$6, and NGK is re-run. Frequencies are again thresholded for robust variable choice.

These methods are demonstrated to enhance power and stability in high-dimensional, small-sample, or highly correlated regimes (Fang et al., 2012).

6. Practical Considerations, Extensions, and Applications

• Smoothing Parameter Estimation: The initial smoothing parameter $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$7 is estimated via Gaussian-process REML or MLE with a common-value $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$8.
• Variable Screening: For $f(x)=\sum_{i=1}^n \alpha_i K(x_i,x),$9, an initial screening step (e.g., sure-independence or nonparametric screening) is recommended to reduce candidate variables.
• Generalized Models: The framework extends to classification or other non-Gaussian regression contexts by replacing squared loss with, e.g., log-likelihood for logistic kernel machines.
• Group Selection: By grouping kernels and penalizing group-wise $K$0-norms, NGK generalizes to "group-NGK" for pathway or set selection.
• Computational Shortcuts: Low-rank approximations and randomized feature mappings can accelerate kernel computations in large-scale settings.

Empirical studies featured in (Fang et al., 2012) indicate that NGK attains exact variable selection in simulated nonadditive scenarios where additive-only rivals like COSSO fail, and it converges reliably even when $K$1. On cryptographic side-channel and diabetes pathway data, NGK with resampling achieves accurate and interpretable support recovery, outperforming direct alternatives.

7. Connections and Significance in the Literature

NGK unifies and extends various strands in sparse learning:

• As a nonlinear kernelized generalization of the linear nonnegative garrote,
• Connected to kernel learning and ARD techniques but with explicit, direct variable sparsity recovery,
• Extends lasso-style sparsistency theory into nonadditive kernel spaces with precise analogs of irrepresentable/incoherence conditions.

Its key strengths are the automatic accommodation of unknown high-order interactions, solid theoretical underpinnings, and adaptability to general regression/classification problems in high-dimensional, nonparametric settings (Fang et al., 2012).