Optimal Subsampling for Large Sample Logistic Regression (1702.01166v2)

Abstract: For massive data, the family of subsampling algorithms is popular to downsize the data volume and reduce computational burden. Existing studies focus on approximating the ordinary least squares estimate in linear regression, where statistical leverage scores are often used to define subsampling probabilities. In this paper, we propose fast subsampling algorithms to efficiently approximate the maximum likelihood estimate in logistic regression. We first establish consistency and asymptotic normality of the estimator from a general subsampling algorithm, and then derive optimal subsampling probabilities that minimize the asymptotic mean squared error of the resultant estimator. An alternative minimization criterion is also proposed to further reduce the computational cost. The optimal subsampling probabilities depend on the full data estimate, so we develop a two-step algorithm to approximate the optimal subsampling procedure. This algorithm is computationally efficient and has a significant reduction in computing time compared to the full data approach. Consistency and asymptotic normality of the estimator from a two-step algorithm are also established. Synthetic and real data sets are used to evaluate the practical performance of the proposed method.

• The paper presents an efficient subsampling method that approximates the maximum likelihood estimate in logistic regression, reducing computational costs without sacrificing accuracy.
• It proposes a two-step algorithm where a pilot subsample estimates the MLE and guides informed sampling based on A-optimality, minimizing asymptotic mean squared error.
• Theoretical guarantees of consistency and asymptotic normality are supported by extensive empirical validation on synthetic and real-world datasets.

Overview of Optimal Subsampling for Large Sample Logistic Regression

The paper "Optimal Subsampling for Large Sample Logistic Regression" authored by Wang et al., addresses the exigent demand for efficient computational methods in the wake of burgeoning data sizes. The central challenge tackled by this paper is the computational infeasibility of applying traditional logistic regression methods directly to excessively large datasets. To surmount this obstacle, the authors propose an efficient subsampling technique aimed at approximating the maximum likelihood estimate (MLE) in logistic regression, significantly reducing the computational burden without compromising on accuracy.

The main contributions of the paper are twofold: the theoretical characterization of optimal subsampling probabilities and the introduction of a two-step subsampling algorithm. The former is aimed at minimizing the asymptotic mean squared error (MSE) of the resultant estimator, leading to two optimal subsampling schemes based on $A$-optimality and computational efficiency. The latter serves as a practical means to implement the theoretical findings, yielding a robust estimator with reduced computational demands.

Key Findings and Methodology

1. Optimal Subsampling Probabilities: The paper introduces a set of subsampling probabilities that are designed to minimize the asymptotic MSE of the estimator, called the Optimal Subsampling Method A-optimality Criterion (OSMAC). The optimal probabilities depend on the logistic function evaluated at the full-data MLE, ensuring an efficient focus on influential data points.
2. Two-Step Algorithm: Due to the dependence of optimal subsampling probabilities on the full data MLE, the authors propose a two-step subsampling procedure. Initially, a small pilot subsample is drawn to obtain an estimate of the MLE, which is then used to guide the sampling probabilities for the main subsample. The second subsample, drawn with these informed probabilities, is used for the final estimation, ensuring computational efficiency and improved accuracy.
3. Consistency and Asymptotic Normality: The paper provides theoretical guarantees for the proposed subsampling strategy, establishing the consistency and asymptotic normality of the resultant estimator. This is a significant theoretical underpinning that consolidates the reliability of the subsampling approach in approximating full data MLE.
4. Empirical Performance and Validation: The authors validate their approach through extensive experiments on both synthetic and real-world data. These experiments demonstrate significant reductions in computational time compared to processing the full dataset, while the estimator efficiently approximates the results of the full data approach in terms of MSE and classification accuracy.

Implications and Future Directions

The implications of this research extend into various practical domains where logistic regression is a cornerstone technique, particularly with the advent of large-scale datasets in fields such as genetics, socio-economics, and physics. The reduction in computational burden facilitates the use of logistic regression in environments with limited computational resources.

Future investigations could explore extending the optimal subsampling framework to alternative statistical models, particularly those involving non-linear relationships or high-dimensional covariates. Moreover, examining the applicability of the proposed method to other distributional assumptions or in the presence of heavy-tailed covariate distributions could augment its versatility. Additionally, the trade-offs between bias and variance in the context of extremely unbalanced data, such as rare events, remain a fertile area for further exploration.

In conclusion, this paper provides a compelling approach to mitigating the computational challenges associated with logistic regression on large datasets, blending robust theoretical insights with practical algorithmic solutions. The proposed methods not only facilitate efficient computation but also maintain the rigor and accuracy expected from classical statistical techniques. Such developments are pivotal as the field of data science continues to adapt to the ever-increasing scale of data.