State-of-art minibatches via novel DPP kernels: discretization, wavelets, and rough objectives

Abstract: Determinantal point processes (DPPs) have emerged as a kernelized alternative to vanilla independent sampling for generating efficient minibatches, coresets and other parsimonious representations of large-scale datasets. While theoretical foundations and promising empirical performance have been demonstrated, there are two challenges for current proposals for DPP-based coresets or minibatches. The first is the need for families of DPPs with certain key variance reduction properties, usually constructed in a continuous setting, of which there are few known examples. The second is the need for an ad-hoc construction of a discrete DPP defined on a given dataset, that inherits such variance reduction. In this work, we contribute to the programme of establishing DPPs as a subsampling toolbox for ML by advancing on these two fronts. First, we propose new DPPs on the Euclidean space based on wavelets, with provably better accuracy guarantees than the best known rates. Second, we introduce a general method to convert such continuous DPPs, which are more amenable to proving analytical statements, into discrete kernels, which are pertinent for subsampling tasks such as minibatch and coreset constructions. This conversion mechanism simultaneously preserves the desired variance decay and reveals a low-rank decomposition of the discrete kernel, which makes sampling the corresponding DPP computationally inexpensive. En route, we enlarge the class of ML tasks amenable to improvements via DPP-based minibatches and coresets to include objective functions with arbitrarily low regularity, and rate guarantees that explicitly adapt to this regularity.

• The paper establishes a wavelet-based DPP framework that adaptively reduces variance for non-smooth objectives.
• It introduces a computationally efficient discretization pipeline that transfers variance guarantees from continuous to discrete settings.
• Empirical results verify superior integration, coreset construction, and SVM convergence compared to traditional sampling methods.

State-of-the-Art DPP-based Minibatch Design: Discretization, Wavelets, and Variance Control for Rough Objectives

Overview and Motivation

This paper rigorously advances the theory and practice of constructing high-quality minibatches and coresets for large-scale machine learning by leveraging determinantal point processes (DPPs) with new kernel constructions and a robust discretization scheme. The two main contributions are: (1) establishing a novel class of DPPs built from wavelet bases that yield explicitly regularity-adaptive variance reduction—surpassing the rate guarantees of previous DPP constructions—especially for nonsmooth objectives, and (2) devising a systematic and computationally efficient pipeline for transferring favorable variance properties from continuous-space DPPs to discrete DPPs acting directly on datasets.

These methodological developments address critical limitations in prior DPP-based subsampling—for instance, the lack of flexible kernels with strong variance decay, and the absence of general procedures to discretize such DPPs without sacrificing analytical properties or incurring prohibitive computational cost.

DPPs and Variance Reduction in Minibatch and Coreset Sampling

DPPs define negatively dependent random subsets, parameterized by kernels, that favor diversity. While their classical use in minibatch selection or core set construction was mostly restricted to kernels based on multivariate orthogonal polynomials (OPEs), such constructions both limited achievable variance reduction rates and were analytically tractable only for smooth loss or test functions.

The central quantity for theoretical and practical performance is the standard error of a linear statistic $$\Lambda_{\mathcal{S}}(f)=\sum_{Y\in\mathcal{S}} f(Y)$$ for $f$ in a function class $\mathcal{F}$. For previous OPE-based DPPs and $C^1$ functions, the variance decays as $n^{-(\frac{1}{2}+\frac{1}{2d})}$, improving over $n^{-1/2}$ for i.i.d. sampling but not exploiting additional regularity or structure.

The new wavelet-based DPP construction generalizes beyond polynomials, directly controlling the variance decay for functions in Hölder or fractional Sobolev classes, and crucially adapts the rate to the function's regularity parameter $s$:

$$\operatorname{Var}[\Lambda_{\mathcal{S}}(f)] \leq C |f|_s^2\, n^{1-2s/d}$$

where $|f|_s$ is the Hölder or Sobolev seminorm of $f$. For $f$0, this yields $f$1, which matches the best rate achieved by more specialized constructions (e.g., Bergman kernel DPPs on complex manifolds) without relying on such geometric assumptions. For low-regularity $f$2, this bound in fact sharply outperforms all existing DPP-based sampling schemes (2605.13127).

Wavelet-DPPs: Construction and Variance Guarantees

The paper constructs a family of DPP kernels using compactly supported, orthonormal wavelet scaling functions. For a given resolution $f$3, the DPP kernel becomes:

$f$4

where $f$5 denotes multivariate dilates and translates of the wavelet scaling basis, and $f$6 indexes a grid adapted to the domain $f$7. This construction generalizes stratified sampling: for Haar wavelets, the resulting DPP coincides with independent uniform draws from each dyadic cell—a fact rigorously proved in the paper.

For both $f$8 and $f$9 test functions, a direct analysis of the projection structure and local support of the wavelet basis allows for tight control of the projection residual, leading to the key variance guarantee outlined above.

Empirical results confirm significant variance reduction for DPP quadrature estimators, including for non-smooth integrands:

Figure 1: d = 1: Variance decay for $\mathcal{F}$0 (one-dimensional function with fractional regularity) demonstrates the strong variance reduction of wavelet-based DPP quadrature over i.i.d. sampling.

Figure 2: d = 2: Variance decay for $\mathcal{F}$1 in two dimensions confirms that performance advantage holds across dimensions.

Discretization Pipeline: From Continuous to Discrete DPPs

Translating variance guarantees from the continuous to purely discrete DPPs is nontrivial. Existing techniques, such as spectral truncation or case-specific OPE projections, were limited either by inflexibility, computational cost, or lack of general theoretical control.

The paper introduces a universal pipeline to discretize a projection DPP defined via a kernel $\mathcal{F}$2 (expressed as a sum over $\mathcal{F}$3 orthonormal functions $\mathcal{F}$4) into a DPP on an $\mathcal{F}$5-sized dataset $\mathcal{F}$6 conditional on the data. Explicit analytic bounds show that the variance of linear statistics for the discrete DPP is sandwiched above and below by that of the continuous DPP up to explicitly computable, negligible error terms when $\mathcal{F}$7, the regime relevant for coresets and minibatches. This resolves a key challenge: variance guarantees are preserved under discretization, and the low-rank kernel structure is leveraged for efficient sampling.

Algorithmically, the construction requires only evaluating wavelet functions on $\mathcal{F}$8 and forming Gram matrices, leading to $\mathcal{F}$9 cost for sampling—a favorable scaling as $C^1$0 in practical applications.

Practical Application to Coreset Construction and Learning

The theoretical contributions are demonstrated in two settings:

1. Quadrature/Integration: Minibatches generated by the wavelet DPP yield unbiased estimators of integrals with mean squared error decaying as $C^1$1 for $C^1$2 test functions.
2. Coreset Construction for ML Losses: Combining the novel DPP and discretization pipeline delivers explicit high-probability PAC-style coreset guarantees, i.e., uniform relative error bounds across wide classes of predictors and losses. Importantly, these extend to non-smooth, merely Hölder-continuous objectives—a regime out of scope for OPE DPPs and prior work.

On real and synthetic datasets, the empirical $C^1$3 quantile of the relative coreset error is consistently lower for wavelet-based DPP samplers (haar and db2) compared to uniform and OPE-based samplers, for both $C^1$4-means clustering tasks and complex, multi-modal data:

Figure 3: $C^1$5 (90% quantile of relative error) vs. minibatch size $C^1$6, showing the superior accuracy of wavelet-based DPPs for coresets, especially in real-world settings.

The benefits extend to stochastic learning with non-smooth losses: DPP minibatching dramatically improves convergence and generalization for SVM/hinge loss stochastic subgradient descent. Test error and parameter convergence are consistently better for the wavelet-based samplers:

Figure 4: Test error trajectories for MNIST binary classification with non-smooth hinge loss, revealing consistently lower misclassification using wavelet-based DPP minibatches.

Theoretical and Practical Implications

Several theoretical upshots and practical implications emerge:

• Adaptivity to Regularity: The wavelet-DPP family is regularity-adaptive: variance decays rapidly for both smooth and rough functions, enabling efficient learning and estimation with nonsmooth, real-world objectives.
• Discretization Without Compromise: The continuum-to-discrete pipeline transfers variance reduction properties without requiring case-specific, complex-analytic or geometric analysis, or costly high-rank approximations.
• Computational Scalability: The low-rank structure guarantees linear (in $C^1$7) cost for sampling and facilitates practical deployment on large-scale tasks.

Future Research Directions

The paper posits several concrete research directions:

• Sharpness and Optimality: Investigating whether further kernel constructions can push variance reduction rates beyond those obtained here, especially in terms of dimension dependence.
• Cardinality-Decoupling: Developing DPP designs where sample size control is decoupled from dimension, currently exponential for full support exploration.
• Optimal Regularity Dependence: Characterizing how variance can be tuned or optimized for functions with minimal regularity and connecting with potential ML loss landscapes.
• Wider Applications: Extending the approach to high-dimensional learning with intrinsic low-dimensional structure and to other negative dependence mechanisms in probabilistic subsampling.

Conclusion

This work represents a comprehensive advancement in DPP-based subsampling for machine learning, overcoming the barriers of kernel expressivity, computational tractability, and theoretical robustness. The introduction of wavelet-based continuous DPPs, coupled with a general low-rank-preserving discretization strategy, delivers explicit, regularity-adaptive variance control for a broad class of objectives—including non-smooth losses. These contributions are validated both theoretically and empirically, suggesting substantial impact for scalable, variational, and robust learning tasks, as well as refreshingly broad applicability to integration, coreset design, and stochastic optimization (2605.13127).