Tuning parameter selection in high dimensional penalized likelihood (1605.03321v1)

Abstract: Determining how to appropriately select the tuning parameter is essential in penalized likelihood methods for high-dimensional data analysis. We examine this problem in the setting of penalized likelihood methods for generalized linear models, where the dimensionality of covariates p is allowed to increase exponentially with the sample size n. We propose to select the tuning parameter by optimizing the generalized information criterion (GIC) with an appropriate model complexity penalty. To ensure that we consistently identify the true model, a range for the model complexity penalty is identified in GIC. We find that this model complexity penalty should diverge at the rate of some power of $\log p$ depending on the tail probability behavior of the response variables. This reveals that using the AIC or BIC to select the tuning parameter may not be adequate for consistently identifying the true model. Based on our theoretical study, we propose a uniform choice of the model complexity penalty and show that the proposed approach consistently identifies the true model among candidate models with asymptotic probability one. We justify the performance of the proposed procedure by numerical simulations and a gene expression data analysis.

• The paper proposes using the Generalized Information Criterion (GIC) to select tuning parameters for high-dimensional generalized linear models, addressing challenges in selecting optimal parameters when dimensionality is vast.
• The authors show that the model complexity penalty must diverge at a rate dependent on log p and tail distribution, invalidating traditional AIC/BIC in ultra-high dimensions.
• Theoretical results demonstrate that GIC with the appropriate penalty achieves asymptotic consistency, validated through simulations and gene expression data analysis.

Tuning Parameter Selection in High-Dimensional Penalized Likelihood

The paper presented in "Tuning Parameter Selection in High-Dimensional Penalized Likelihood," authored by Yingying Fan and Cheng Yong Tang, addresses a prominent challenge in statistical modeling of high-dimensional data: the selection of an optimal tuning parameter for penalized likelihood methods. Penalized likelihood methods, vital in high-dimensional data analysis, involve the balance between model fit and complexity, governed by tuning parameters. This research explores generalized linear models (GLMs) where the number of potential covariates $p$ can grow exponentially with the sample size $n$, posing substantial challenges in selecting appropriate tuning parameters that ensure model consistency.

Key Contributions and Methodology

1. Generalized Information Criterion (GIC): This paper proposes using the generalized information criterion (GIC) to select tuning parameters for GLMs. The GIC optimizes the trading off between model fitting adequacy and model complexity by incorporating a penalty term $a_n$ influencing model complexity. The authors demonstrate that the correct choice of this penalty is crucial for model selection consistency, particularly when dimensionality $p$ is exceedingly large.
2. Model Complexity Penalty: Through rigorous analysis, the authors reveal that the model complexity penalty $a_n$ should diverge at a rate dependent on the power of $\log p$, which is further dictated by the tail distribution characteristics of the response variables. This insight invalidates the traditional models like AIC and BIC in ultra-high-dimensional settings, emphasizing the need for a modified BIC that adapts according to sample size and dimensionality.
3. Asymptotic Consistency: Theoretical results support that the GIC with an appropriately chosen $a_n$ consistently identifies the true model among many candidate models as the dimensionality increases. These conditions imply that the penalty factor must balance between diverging too quickly and facilitating model selection bias.
4. Simulation and Empirical Validation: To validate the theoretical claims, simulations and gene expression data analyses were conducted, demonstrating the proposed method's efficacy in practice. Notably, in examining a gene expression dataset for leukemia classification, the proposed method identified critical biomarkers, signifying its practical valuation in biomedical research.

Implications and Future Directions

The need for tailored model complexity penalties in high-dimensional settings reflects the growing intricacy of modern data. This paper broadens the understanding of statistical selection in expansive data environments, essentially paving the way for improved model accuracy and reliability.

The adoption of GIC presents substantial improvements over traditional criteria by systematically adapting for increased dimensionality and varying distribution properties. This opens future explorations into ensemble methods that could hybridize the GIC with other selection strategies to enhance robustness and scalability.

On a broader theoretical level, the insights provided pose intriguing questions on the interaction between signal strength, dimensionality, and penalization, suggesting new research avenues to refine model selection criteria as data sizes continue to escalate.

In conclusion, this paper lays significant groundwork in high-dimensional statistical inference, providing a pathway towards more consistent and accurate model selection methodologies suitable for the complexities of contemporary datasets, progressing the field of high-dimensional data analysis.