Adaptive Greedy Support Point Selection

• Adaptive greedy support point selection is an iterative algorithm that adaptively selects representative points using error-based or kernel similarity criteria to ensure sparse yet accurate approximations.
• This approach underpins various applications including sparse Gaussian Processes, meshless PDE solvers, and surrogate modeling, significantly enhancing computational efficiency and stability.
• The method provides theoretical guarantees on error decay and coverage, enabling adaptive model sizes that align with intrinsic data complexity and ensure numerical robustness.

Adaptive greedy support point selection encompasses a class of algorithms that iteratively identify and append points to a subset of a domain or a dataset, guided by error or similarity criteria that adapt to the structure and distribution of observed data. This family of techniques is foundational to modern sparse learning, kernel-based approximation, surrogate modeling, and Bayesian inference, with applications spanning Gaussian Processes, PDE solvers, measure-valued optimization, nonlinear approximation, and deterministic sampling schemes.

1. Fundamental Principles and Problem Settings

Adaptive greedy support point selection treats the selection of representative points—often called inducing points (GPs), collocation points (PDE solvers), atoms (greedy learning), or support points (empirical measures)—as a sequential process. At each iteration, the algorithm evaluates a data- or model-driven selection criterion across candidate points not yet included in the current support set. The chosen criterion typically reflects notions of error, residual, kernel similarity, or a surrogate for approximation error. Each new point augments the support set, after which model parameters, representations, or interpolants are updated.

The methodological landscape spans:

• Sparse Gaussian Processes: Selection and incremental addition of inducing points for efficient variational inference in streaming or large-data regimes (Galy-Fajou et al., 2021).
• Greedy Dictionary/Atom Learning: Stepwise augmentation of basis functions or dictionary elements to best represent residual signals, including adaptive thresholding for stopping (Xu et al., 2014).
• Spline and Kernel Interpolation: Greedy identification of interpolation nodes or collocation points that minimize actual or surrogate error (e.g., Lebesgue constant control, power function) (Campagna et al., 2021, Wenzel et al., 2019).
• Meshless PDE Approximation: Kernel-based collocation methods selecting collocation functionals supporting improved convergence rates (Wenzel et al., 2022).
• Surrogate Rational Modeling: Greedy, error-indicator-driven selection of sample locations for efficient surrogate construction in frequency-domain simulation (Pradovera, 2023).
• Measure-Valued Optimization: Generalized conditional gradient (GCG) and its variants for sparse measure recovery, guaranteed by dual-certificate maximization and lazified/inexact subroutine solutions (Hnatiuk et al., 5 Aug 2025).
• Deterministic Sampling (Stein Points, Herding): Greedily constructed empirical measures via kernel Stein discrepancy or discrepancy-driven objectives for distribution approximation (Chen et al., 2018).

2. Core Algorithms and Selection Criteria

Algorithms are characterized by adaptive, local decisions on which support point to append, with selection criteria exhibiting varying dependency on observed data and model state. Pseudocode structure and computational costs typically reflect the following key elements:

A. Greedy Criterion

• Residual-based: Selection according to pointwise or functional residuals (e.g., $|f(x)-\hat f(x)|$ in splines (Campagna et al., 2021), $|α(r_n)|$ for PDEs (Wenzel et al., 2022)).
• Power/Lebesgue Functions: Maximal decay in power function (kernel-based), or reduction in Lebesgue constant, promoting stability and quasi-uniform coverage (Wenzel et al., 2019, Campagna et al., 2021).
• Dual Certificate Maximization: Maximizing $|p_u(x)|$ as a certificate for optimal support in measure-valued optimization (Hnatiuk et al., 5 Aug 2025).
• Kernel Similarity: Minimal maximal kernel similarity to existing points, as in the “half-greedy” approach to inducing point selection in GP streaming (Galy-Fajou et al., 2021).
• Error Indicators/Surrogates: Non-intrusive indicators such as $Q(z)^{-1}$ in rational surrogate modeling, where $Q(z)$ is a barycentric denominator (Pradovera, 2023).
• Discrepancy-driven: Minimizing a discrepancy measure (e.g., KSD in Stein points) by incremental pointwise minimization of a surrogate objective (Chen et al., 2018).

B. Per-Iteration Steps

General procedure for a greedy step:

1. Compute candidate scores using the selection criterion over all not-yet-chosen points.
2. Pick the maximizer (or satisfying some threshold, as in γ- or δ-thresholding).
3. Augment the support set with the optimal point.
4. Update model/interpolant/estimate.
5. Iterate until a stopping condition is reached.

Computational overhead per step ranges from $O(M)$ to $O(n)$ per point, given $M$ current support points or $n$ evaluation points.

3. Theoretical Guarantees and Error Bounds

Adaptive greedy support point selection is equipped with robust theoretical guarantees, including explicit bounds on the number of points selected, the decay rate of approximation errors, and the stability or conditioning of resulting interpolation models.

A. Coverage and Filling Numbers in Kernels/GPs

For streaming sparse GPs with stationary kernels, the expected number of inducing points $M$ saturates at the covering number $|α(r_n)|$0 for balls of radius set by the kernel threshold parameter ($|α(r_n)|$1), yielding an explicit, data-dependent control on sparsity (Galy-Fajou et al., 2021).

B. Empirical Risk and Learning Rate Bounds

In atom/dictionary selection (OGL), the δ-threshold rule achieves expected risk rate $|α(r_n)|$2, matching classical OGL, but with an entirely data-adaptive stopping rule and potentially sparser final representations (Xu et al., 2014).

C. Kolmogorov n-width and PDE Kernel Approximation

For meshless PDEs, the Kolmogorov width $|α(r_n)|$3 for the set of collocation functionals dictates the minimax error decay. Target-data-dependent (residual-driven) rules can yield rates $|α(r_n)|$4, strictly faster than target-independent rules ($|α(r_n)|$5) and dimension-independent in the extra $|α(r_n)|$6 factor (Wenzel et al., 2022).

D. Error Surrogates and Certified Convergence

Non-intrusive error indicators, such as the barycentric denominator in rational surrogate modeling, provide theoretical certification of maximal pointwise surrogate error, with decay rates aligned to the Kolmogorov $|α(r_n)|$7-width (Pradovera, 2023).

E. Stability and Quasi-Uniformity

The introduction of stabilization parameters (e.g., γ in γ-restricted kernel greedy) ensures control over matrix conditioning, separation, and fill distances, with algebraic error decay rates preserved across the entire stabilized family and improved practical robustness (Wenzel et al., 2019).

F. Optimal Local Recovery

Recent kernel-based local greedy selectors achieve $|α(r_n)|$8 error rates locally with the minimal number of support points needed to realize the Sobolev rate, and automatically suppress near-duplicate or collinear points (Schaback, 2024).

4. Practical Implementation and Algorithmic Details

Implementation is shaped by the structure of the problem, the type of kernel or functional used, the desired tradeoff between sparsity and accuracy, and the nature of the data (streaming, batch, structured, etc.).

• Initialization: Many methods start with a single point (first observation, posterior mode, endpoint), or a randomized small set to ensure numerical stability.
• Threshold Selection: Parameters like $|α(r_n)|$9 (kernel similarity threshold), δ (correlation/angle threshold), or γ (power function ratio cutoff) are tuned to balance accuracy and sparsity, often via cross-validation or empirical analysis (Galy-Fajou et al., 2021, Xu et al., 2014, Wenzel et al., 2019).
• Update Mechanisms: Upon point insertion, model parameters are immediately or periodically re-optimized. In variational GP, this includes updating moments $|p_u(x)|$0 over inducing variables; in measure optimization, coefficients and positions of Dirac masses are updated, optionally via Newton or local merging (Hnatiuk et al., 5 Aug 2025).
• Stopping Criteria: Data-driven stopping occurs when no candidate exceeds a threshold, residual norms drop below set tolerances, or error indicators certify sufficient accuracy. Empirically, this yields adaptive model sizes without prespecification (Xu et al., 2014, Pradovera, 2023).
• Complexity: Local point selection, Newton-basis updates, or banded matrix solves enable efficient per-point cost (e.g., $|p_u(x)|$1 for kernel similarity, $|p_u(x)|$2 for banded EPS splines, $|p_u(x)|$3 for local Sobolev recovery) (Galy-Fajou et al., 2021, Campagna et al., 2021, Schaback, 2024).
• Instability Control: “Near-duplicate” points or conditions approaching ill-posedness are detected and pruned, with thresholds on power functions or residuals guarding against numerical instability (Wenzel et al., 2019, Schaback, 2024).

5. Empirical Evaluation and Comparative Performance

Comparative studies across several domains universally demonstrate the adaptivity and efficiency of greedy support point selection.

Benchmark Results:

Setting	Baselines	Greedy method(s)	Key Observations
Streaming GP Regression	Grid, $|p_u(x)|$4-means	One-pass (half-greedy), OIPS+optim	OIPS adapts to gaps, fewer points needed in high-D; grid suffers curse
Spline Interpolation	Uniform downsampling	$|p_u(x)|$5-greedy, $|p_u(x)|$6-greedy	Fewer nodes, better clustering in high-gradient regions/near boundaries
PDE Meshfree Solution	Uniform/Random mesh	PDE-P/f-greedy	Data-driven selection yields faster convergence and better interior coverage
Surrogate Modeling	Static sampling	Indicator-driven greedy	Fewer queries for certified accuracy; critical for complex frequency responses
General Kernel Interp.	P-greedy only	γ-restricted/ $|p_u(x)|$7- or $|p_u(x)|$8-greedy	γ-tuning allows stability and error tradeoff, optimal algebraic rates
Local Sobolev Recovery	MLS/Static local rules	Local kernel-based greedy	Achieves minimal-point, optimal-rate local error with O(1) complexity

In situations beset by ill-conditioning (e.g., high kernel smoothness, nearly collinear support), stabilized greedy rules exhibit continued robustness and error decay, while unconstrained variants can stall or diverge (Wenzel et al., 2019, Schaback, 2024).

6. Extensions, Variants, and Open Directions

Recent work explores a spectrum of advanced features and hybrid criteria:

• Lazified Updates in Measure-Valued Optimization: Allowing inexact/approximate solutions (e.g., for dual maximization or coefficient optimization) still preserves convergence guarantees, with global greedy steps alternating with local Newton or coefficient updates and cluster-merging routines providing eventual quadratic convergence (Hnatiuk et al., 5 Aug 2025).
• Interleaving Multiple Criteria: Hybrid “β-greedy” rules in PDE and RBF methods interpolate between power- and residual-driven selection, enabling dynamic adaptation to data structure (Wenzel et al., 2022).
• Certification and Error Surrogates: Greedy selection driven by efficiently computable error indicators offers practical certified accuracy in surrogate modeling, rational interpolation, and parametric model reduction (Pradovera, 2023).
• Local vs. Global Selection: Local greedy approaches in Sobolev spaces allow minimal-point, local optimal recovery at per-query constant computational complexity, in contrast to globally constructed greedy sets (Schaback, 2024).
• Discrepancy Methods and Deterministic Sampling: Greedy minimization of discrepancy measures (e.g., kernel Stein discrepancy) gives deterministic samples that converge under weak topologies with dimension-independent rates (Chen et al., 2018).
• Multi-Objective and Regularized Greedy Selection: Regularization or penalization (e.g., on total variation or curvature) and adaptive switching between error and stability criteria improve performance in noisy or irregular data settings (Campagna et al., 2021, Wenzel et al., 2019).

7. Significance and Comparative Analysis

Adaptive greedy support point selection provides an effective tradeoff between the statistical efficiency of model representations and computational/algorithmic tractability. Key distinguishing features include:

• Data-Dependent Sparsity: Model or approximation size is adaptively matched to the intrinsic data complexity or distributional structure.
• Near-Optimal Approximation: Error rates match, and often saturate, best possible rates for approximation in the underlying function space, kernel, or PDE solution manifold.
• Stability/Conditioning Control: Algorithmic modifications (stabilization thresholds, local linear algebra) yield robust performance even in high-dimensional, ill-conditioned, or highly clustered regimes.
• Universality: The greedy selection framework unifies kernel interpolation, sparse learning, meshless PDE methods, and optimal deterministic quadrature/sampling under a single abstract paradigm.

Research trends continue toward certifiable, efficient, and theoretically principled greedy selection mechanisms that scale to large datasets, irregular domains, and complex models, with ongoing analysis clarifying convergence, stability, and optimality guarantees across a widening landscape of applications (Galy-Fajou et al., 2021, Xu et al., 2014, Campagna et al., 2021, Wenzel et al., 2022, Pradovera, 2023, Wenzel et al., 2019, Schaback, 2024, Hnatiuk et al., 5 Aug 2025, Chen et al., 2018).