kNN Algorithm for Conditional Mean and Variance Estimation with Automated Uncertainty Quantification and Variable Selection

Abstract: In this paper, we introduce a kNN-based regression method that synergizes the scalability and adaptability of traditional non-parametric kNN models with a novel variable selection technique. This method focuses on accurately estimating the conditional mean and variance of random response variables, thereby effectively characterizing conditional distributions across diverse scenarios.Our approach incorporates a robust uncertainty quantification mechanism, leveraging our prior estimation work on conditional mean and variance. The employment of kNN ensures scalable computational efficiency in predicting intervals and statistical accuracy in line with optimal non-parametric rates. Additionally, we introduce a new kNN semi-parametric algorithm for estimating ROC curves, accounting for covariates. For selecting the smoothing parameter k, we propose an algorithm with theoretical guarantees.Incorporation of variable selection enhances the performance of the method significantly over conventional kNN techniques in various modeling tasks. We validate the approach through simulations in low, moderate, and high-dimensional covariate spaces. The algorithm's effectiveness is particularly notable in biomedical applications as demonstrated in two case studies. Concluding with a theoretical analysis, we highlight the consistency and convergence rate of our method over traditional kNN models, particularly when the underlying regression model takes values in a low-dimensional space.

• The paper introduces a novel kNN framework that jointly estimates conditional mean and variance, enhancing prediction intervals and uncertainty quantification.
• It integrates data splitting and variable selection techniques to mitigate selection bias and improve model interpretability in low-dimensional settings.
• Empirical simulations and biomedical case studies validate the method's robust performance, highlighting its practical impact on precision health research.

Introduction

The proliferation of data-driven science has necessitated the development of flexible and robust statistical methods. The traditional statistical framework has been predominantly focused on the estimation of conditional mean functions, often under parametric assumptions. However, understanding the entire conditional distribution, which includes both mean and variance components, is essential for a comprehensive analysis of data, particularly in fields where prediction intervals and uncertainty quantification are critical.

kNN-Based Approach and Methodology

The paper introduces an advanced k-nearest neighbors (kNN) regression method, integrating the method's computational efficiency with a novel variable selection technique and automated uncertainty quantification. Unlike previous kNN applications, this approach provides an enhanced framework for estimating not only the conditional mean but also the conditional variance function, capturing a more complete picture of the underlying distribution across different scenarios.

A pivotal element of the proposed method is the implementation of multiple data splitting strategies, ensuring a trustworthy process throughout the model development stages. This mitigates the post-selection bias commonly associated with model fitting. An innovative variable selection process is also introduced, leading to improved model interpretability and convergence rates, particularly in low-dimensional manifold structures.

Theoretical Contributions and Practicality

The paper underlines strong theoretical merits, including consistency and convergence rate analysis. The kNN algorithm's semi-parametric nature allows for robust prediction intervals and ROC curve estimation, particularly relevant in medical research for validating biomarkers and screening tests. Demonstrated through practical simulations and applied to biomedical case studies, the algorithm exhibits notable effectiveness, underscoring the substantial implications for large-scale medical research.

Future Research Directions

As we progress, it is envisioned to adapt the model further, aiming to refine variable selection locally and improve conditional distribution reconstructions. Comparative analyses with existing methods like GAMLSS will elucidate the unique advantages of the proposed approach in handling massive datasets. The development of personalized disease screening methods and precision health policies stands as a promising application of this statistical advancement.

In conclusion, this paper succeeds in advancing kNN regression techniques, extending their applicability, and providing deeper insights into data analysis across various scientific and applied fields.