Element-wise Core-Sets

• Element-wise core-sets are a deterministic subset selection method for least squares estimation that targets high-magnitude entries in each column, ensuring unbiased and efficient results.
• The approach minimizes variance by focusing on the most informative data, providing concrete theoretical guarantees through spectral approximation and variance control.
• Enhanced with a median-of-means extension, the method robustly handles outliers while offering significant runtime savings compared to traditional row-based sampling techniques.

Element-wise core-sets (core-elements) constitute a deterministic subset selection methodology for large-scale least squares estimation. Unlike conventional coreset methods that subsample rows of the design matrix, the core-elements approach targets individual entries with highest magnitude in each column. This element-wise paradigm is especially effective for sparse or numerically sparse predictor matrices by allocating the sampling budget to the most informative data, yielding computationally efficient and statistically robust estimators with concrete theoretical guarantees (Li et al., 2022).

1. Definition and Construction of Core-Elements

Consider the linear model

$$y_i = x_i^\top\beta + \varepsilon_i, \quad i=1,\dots,n,$$

with $X\in\mathbb{R}^{n\times p}$ as the design matrix, $\beta\in\mathbb{R}^p$ the regression parameter, and $\varepsilon_i$ i.i.d. noise. Element-wise core-sets select $r$ entries per column, constructing an indicator matrix $S\in\{0,1\}^{n\times p}$ with $r$ ones in each column: $s_{ij} = \begin{cases} 1, & i\in\{\text{indices of %%%%6%%%% largest } |x_{ij}|\}, \ 0, & \text{otherwise}. \end{cases}$ The sparsified matrix is $X^* = S\odot X$ (Hadamard product), with exactly $rp$ retained entries. This method exploits sparsity by ensuring the sampling budget targets high-information content rather than allocating uniformly or by row.

2. Estimation Method and Unbiasedness

Given $X\in\mathbb{R}^{n\times p}$0, the core-elements estimator is

$X\in\mathbb{R}^{n\times p}$1

Unbiasedness holds when $X\in\mathbb{R}^{n\times p}$2 is invertible: $X\in\mathbb{R}^{n\times p}$3 This property marks a distinction from stochastic row sampling, as core-elements deterministically preserves—columnwise—the leading signal, subject to selection by magnitude.

3. Variance Bounds and Statistical Guarantees

The conditional variance is

$X\in\mathbb{R}^{n\times p}$4

Control of variance is achieved by selecting $X\in\mathbb{R}^{n\times p}$5 largest-magnitude entries, which approximately minimizes key matrix norms. For $X\in\mathbb{R}^{n\times p}$6 and spectral radius $X\in\mathbb{R}^{n\times p}$7, the upper bound is

$X\in\mathbb{R}^{n\times p}$8

Thus, variance is suppressed by minimizing $X\in\mathbb{R}^{n\times p}$9 and ensuring $\beta\in\mathbb{R}^p$0 is small, both favored by magnitude-based selection.

A finite-sample, coreset-style approximation is guaranteed. If $\beta\in\mathbb{R}^p$1 approximates $\beta\in\mathbb{R}^p$2 in operator norm: $\beta\in\mathbb{R}^p$3 for $\beta\in\mathbb{R}^p$4, then

$\beta\in\mathbb{R}^p$5

This shows near-optimality of the estimator under spectral approximation (Li et al., 2022).

4. Algorithmic Implementation and Computational Complexity

The deterministic construction involves:

1. Initializing $\beta\in\mathbb{R}^p$6.
2. For each column $\beta\in\mathbb{R}^p$7, identify indices of $\beta\in\mathbb{R}^p$8 largest $\beta\in\mathbb{R}^p$9 and set corresponding $\varepsilon_i$0.
3. Form $\varepsilon_i$1.
4. Solve for $\varepsilon_i$2 as above.

Computational costs:

• Column selection: $\varepsilon_i$3 using selection algorithms (e.g., partial sorting).
• Computation of $\varepsilon_i$4: $\varepsilon_i$5.
• Overall (for $\varepsilon_i$6): $\varepsilon_i$7, substantially faster than the $\varepsilon_i$8 cost of OLS or leverage-score methods.

5. Robust Extension via Median-of-Means

To address outlier contamination, median-of-means (MOM) is integrated:

1. Partition $\varepsilon_i$9 into $r$0 random, equal-sized blocks.
2. On block $r$1, apply core-elements with $r$2 budget to obtain $r$3.
3. Aggregate by coordinate-wise median: $r$4.

Under bounded per-block condition and few block corruptions, $r$5. The breakdown point is $r$6. This robustification ensures both statistical consistency and high breakdown under data contamination (Li et al., 2022).

6. Comparative Evaluation and Empirical Performance

Empirical studies benchmarked CORE and MOM-CORE against uniform subsampling, doubly-sketched, leverage methods (BLEV/SLEV), IBOSS, OSS, D-optimal (DOPT), and full OLS on synthetic (Normal, Log-Normal, $r$7) and real datasets, under varying sparsity ($r$8). Metrics included normalized estimation error (MSE) and prediction error (PMSE). Key findings:

• CORE obtains the lowest MSE/PMSE across all distributions and sparsity levels, with an advantage accentuated for sparse $r$9.
• MOM-CORE shows maximal robustness to outlier schemes.
• Runtime for CORE is marginally above UNIF/DOUBLY but dramatically below other deterministic methods, scaling as $S\in\{0,1\}^{n\times p}$0.

A real-world illustration, using single-cell RNA-seq ($S\in\{0,1\}^{n\times p}$1, $S\in\{0,1\}^{n\times p}$2, $S\in\{0,1\}^{n\times p}$3 zeros), demonstrates near-full-OLS accuracy with substantial runtime savings.

Method	Statistical Efficacy (MSE/PMSE)	CPU Time Scaling
CORE	Lowest across settings	$S\in\{0,1\}^{n\times p}$4
MOM-CORE	Highest robustness	Slightly above CORE
BLEV/SLEV	Higher error	$S\in\{0,1\}^{n\times p}$5
UNIF/DOUBLY	Fast, higher error	Fastest

7. Characteristics, Applications, and Future Directions

Element-wise core-sets are motivated by the inefficiency of row-based subsampling under high sparsity: many rows are mostly zero, leading to wasted sample budget. By contrast, element-wise selection extracts information where it is densest.

Advantages:

• Unbiasedness and near-optimal variance bounds.
• Deterministic selection, eliminating sampling variability.
• Runtime linear in $S\in\{0,1\}^{n\times p}$6 plus $S\in\{0,1\}^{n\times p}$7.
• Seamless integration with MOM for robust, efficient estimation.

Applications naturally include large-scale regressions, especially when the data is sparse or contaminated. Anticipated research directions include nonparametric or penalized spline extensions, privacy-preserving adaptations for federated settings, handling other loss functions (e.g., GLMs, quantile regression), and adaptation to streaming scenarios (Li et al., 2022).