Gaussian Process Distance Correlation (GPDC)

• GPDC is a Bayesian optimization technique that leverages distance correlation to measure statistical dependence and balance exploration and exploitation automatically.
• It employs Gaussian process surrogates with posterior sampling and supports multiple observation modalities, facilitating both function estimation and global maximization tasks.
• Empirical results demonstrate that GPDC can reduce the number of evaluations needed and achieve lower cumulative regret compared to standard methods.

Searching arXiv for the specified paper and closely related work to ground the article. {"^^^^^^^^3queries3 Distance Correlation for Efficient Bayesian Optimization\"","limit":5},{"^^^^^^^^3query3 process distance correlation Bayesian optimization arXiv","limit":^^^^3query3queries3^^^^}]} {"^^^^^^^^3queries3 Gaussian Process Distance Correlation, commonly abbreviated GPDC, denotes a Gaussian process–based Bayesian optimization and sequential design scheme in which distance correlation or distance covariance is used as the acquisition principle. In the primary arXiv source on the method, the overall scheme is introduced as BDC in the abstract and explicitly described in the text as GP-DC, i.e., a Gaussian Process–Distance Correlation approach (&&&^^^^3query3^^^^&&&). The method addresses expensive black-box objectives and related experimental-design problems by drawing posterior samples from a Gaussian process surrogate and selecting the next ^^^^^^^^3query3^^^^^^^^ according to the statistical dependence between candidate observations and either the unknown maximum of the function or the global function configuration itself. Its defining claim is that this dependence-based criterion balances exploration and exploitation automatically and requires no explicit acquisition hyperparameter (&&&^^^^3query3^^^^&&&).

^^^^3query3^^^^. Conceptual scope and problem setting

GPDC is situated within standard Gaussian process–based Bayesian optimization (GP-BO), but it is formulated for two distinct tasks. The first is function estimation / experimental design, where the goal is to reconstruct an unknown function globally with as few measurements as possible. The second is black-box maximization, where the goal is to identify PRESERVED_PLACEHOLDER_^^^^3queries3^^^^ using as few evaluations as possible (&&&^^^^3query3^^^^&&&).

A distinguishing feature of the framework is that the function-estimation setting admits multiple observation modalities. The source describes point observations PRESERVED_PLACEHOLDER_^^^^3query3^^^^, integral observations with adjustable width in one and two dimensions, and smoothed derivative observations or coarse-grained gradients. The algorithm therefore chooses not only where to observe, but, in the multi-modality setting, also which observation width or modality to use at each step (&&&^^^^3query3^^^^&&&).

The proposed acquisition principle is motivated by the limitations of standard BO rules. The source contrasts GPDC with EI, PI, GP-UCB, GP-MI, ES, PES, and MES/OPES, emphasizing that many of these methods require explicit exploration parameters, entropy approximations, nested sampling, or numerical integration. GPDC instead uses distance-based dependence between posterior-derived random quantities as the basis for ^^^^^^^^3query3^^^^^^^^ selection. This gives the method an information-theoretic in flavor interpretation while remaining nonparametric and computationally simpler than mutual-information-based alternatives, according to the source description (&&&^^^^3query3^^^^&&&).

^^^^3ti:\3^^^^. Statistical basis: distance covariance and distance correlation

The acquisition mechanism is built on distance covariance and distance correlation as introduced by Székely and collaborators, with the paper also considering a generalized form with exponent PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^ in the function-estimation experiments (&&&^^^^3query3^^^^&&&). At the population level, for random vectors $X \in \mathbb{R}^p$ and $Y \in \mathbb{R}^q$, distance covariance is defined through a weighted $L^2$ discrepancy between the joint characteristic function and the product of marginals: $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$ Distance variance is obtained by setting the two arguments equal, and distance correlation is the normalized quantity

$$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$

with the convention that it is zero when the denominator vanishes (&&&^^^^3query3^^^^&&&).

The properties used directly by GPDC are the ones emphasized in the source: $\mathcal{R}(X,Y)\in[0,1]$; $\mathcal{R}(X,Y)=0$ if and only if PRESERVED_PLACEHOLDER_^^^^3query3queries3^^^^ and PRESERVED_PLACEHOLDER_^^^^3query3query3^^^^ are statistically independent; the measure detects any kind of dependence, not only linear association; it applies in arbitrary dimensions; and it is nonparametric. The source also states that, in practice, the quantity can be estimated reliably with relatively few posterior samples, with experiments using approximately PRESERVED_PLACEHOLDER_^^^^3query3ti:\3^^^^ to PRESERVED_PLACEHOLDER_^^^^3query33^^^^ samples (&&&^^^^3query3^^^^&&&).

For empirical computation, the method uses i.i.d. samples PRESERVED_PLACEHOLDER_^^^^3query3limit9^^^^ pairwise distance matrices

PRESERVED_PLACEHOLDER_^^^^3query3^^^^5

double-centering,

PRESERVED_PLACEHOLDER_^^^^3query3^^^^6

and then

PRESERVED_PLACEHOLDER_^^^^3query3^^^^7

The corresponding empirical distance variances and distance correlation are formed analogously. The source specifies that computations were carried out numerically with the Python library dcor, and that distance correlation, rather than distance covariance, is used in the core selection rules for function estimation because covariance is scale-dependent across modalities (&&&^^^^3query3^^^^&&&).

3. Gaussian process surrogate and posterior sampling

GPDC inherits the surrogate model of standard GP regression. The unknown objective is modeled as

PRESERVED_PLACEHOLDER_^^^^3query3^^^^8

on a domain PRESERVED_PLACEHOLDER_^^^^3query3,9^^^^ with the source noting that a Matérn kernel, especially Matérn 5/^^^^3ti:\3^^^^, is used consistently in the experiments, following Snoek et al. (^^^^3ti:\3queries3query3ti:\3^^^^) (&&&^^^^3query3^^^^&&&). Given data

PRESERVED_PLACEHOLDER_^^^^3ti:\3queries3^^^^

with Gaussian noise, the GP posterior at PRESERVED_PLACEHOLDER_^^^^3ti:\3query3^^^^ has predictive mean and variance

PRESERVED_PLACEHOLDER_^^^^3ti:\3ti:\3^^^^

PRESERVED_PLACEHOLDER_^^^^3ti:\33^^^^

where PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^4 is the Gram matrix, PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^5 is the covariance vector to the test point, and PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^6 is the observation-noise variance (&&&^^^^3query3^^^^&&&).

Posterior sampling is central to GPDC. At each iteration, the method evaluates the GP posterior over a finite set of representative points PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^7 and draws PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^8 posterior function samples

PRESERVED_PLACEHOLDER_^^^^3ti:\3^^^^9

These samples induce empirical distributions for quantities such as the maximum value

$X \in \mathbb{R}^p$^^^^3queries3^^^^

and the maximizing location

$X \in \mathbb{R}^p$^^^^3query3^^^^

which then enter the distance-correlation acquisition (&&&^^^^3query3^^^^&&&).

The same GP formalism also supports non-point observation models. Because the GP is closed under linear operations, the source uses integrated kernels for line integrals, area integrals, and convolution-type smoothed derivatives. For integral observations, kernels of the form

$X \in \mathbb{R}^p$^^^^3ti:\3^^^^

are employed, although the source notes that these integrations can be computationally expensive (&&&^^^^3query3^^^^&&&).

4. Acquisition constructions and algorithmic variants

The source distinguishes between a function-estimation version and a maximization version of GPDC. In both cases, the common principle is to choose the next ^^^^^^^^3query3^^^^^^^^ that is most strongly distance-correlated with a posterior-derived random object of primary interest.

For function estimation, the algorithm first draws posterior function samples $X \in \mathbb{R}^p$3, each discretized on the representative grid. For every candidate width or modality $X \in \mathbb{R}^p$4, it computes the predictive variance $X \in \mathbb{R}^p$5, chooses the most uncertain location

$X \in \mathbb{R}^p$6

and then evaluates, for each posterior sample, the hypothetical observation at $X \in \mathbb{R}^p$7, denoted $X \in \mathbb{R}^p$8. The acquisition score is the distance correlation between the collection of global function samples and the hypothetical observation values,

$X \in \mathbb{R}^p$9

after which the selected modality is

$Y \in \mathbb{R}^q$^^^^3queries3^^^^

The source interprets this as selecting the candidate observation that is most informative, in a dependence sense, about the entire function. It further notes that this induces an automatic coarse-to-fine behavior: wider integrals are often selected early, and narrower widths or point observations become favored later (&&&^^^^3query3^^^^&&&).

For black-box maximization, the basic variants are GP-dCor and GP-dCov. For each candidate point $Y \in \mathbb{R}^q$^^^^3query3^^^^, the method forms the vector of posterior sample values at that point,

$Y \in \mathbb{R}^q$^^^^3ti:\3^^^^

and computes either

$Y \in \mathbb{R}^q$3

for GP-dCor or

$Y \in \mathbb{R}^q$4

for GP-dCov. The next evaluation is then

$Y \in \mathbb{R}^q$5

This is explicitly described as selecting the point whose posterior value is most strongly dependent on the eventual maximum value of the function (&&&^^^^3query3^^^^&&&).

A second family of variants, GP-dCor-X and GP-dCov-X, replaces the sampled maximum values $Y \in \mathbb{R}^q$6 with sampled argmax locations $Y \in \mathbb{R}^q$7. Their corresponding acquisition scores are

$Y \in \mathbb{R}^q$8

or

$Y \in \mathbb{R}^q$9

These versions target dependence with the location of the maximizer rather than its value (&&&^^^^3query3^^^^&&&).

Variant	Dependence target	Dependence measure
GP-dCor	$L^2$^^^^3queries3^^^^ vs. $L^2$^^^^3query3^^^^	Distance correlation
GP-dCov	$L^2$^^^^3ti:\3^^^^ vs. $L^2$3	Distance covariance
GP-dCor-X	$L^2$4 vs. $L^2$5	Distance correlation
GP-dCov-X	$L^2$6 vs. $L^2$7	Distance covariance

The source repeatedly emphasizes that this acquisition rule contains no tunable exploration parameter analogous to EI’s $L^2$8, UCB’s $L^2$9, or GP-MI’s $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$^^^^3queries3^^^^. Exploration and exploitation are instead said to emerge from the dependence structure itself: highly uncertain regions can score well because they co-vary strongly with the posterior maximum, while already-explained or redundant regions score poorly because their values are weakly tied to changes in the maximum (&&&^^^^3query3^^^^&&&).

5. Computational profile and implementation considerations

GPDC relies on repeated posterior sampling over a representative grid and repeated evaluation of pairwise-distance statistics. For function estimation, the source identifies three main costs: GP posterior sampling over $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$^^^^3query3^^^^ points, which requires a Cholesky decomposition of an $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$^^^^3ti:\3^^^^ covariance matrix and therefore scales as $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$3; integrated-kernel computations for non-point observations, which may involve multidimensional numerical integration; and distance-correlation computation, which costs roughly $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$4 per evaluation because it forms $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$5 distance matrices (&&&^^^^3query3^^^^&&&).

For maximization, the same sampling cost appears, together with a stated cost of approximately $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$6 for scoring all $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$7 candidate points once $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$8 posterior samples have been drawn. The experiments use moderate values of $$\mathcal{V}^2(X,Y) = \int_{\mathbb{R}^{p+q}} \frac{\left|\phi_{X,Y}(t,s)-\phi_X(t)\phi_Y(s)\right|^2} {c_p c_q \,\|t\|_p^{1+p}\|s\|_q^{1+q}} \, dt\, ds.$$9, typically about $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$^^^^3queries3^^^^ to $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$^^^^3query3^^^^, and moderate candidate grids, which the source describes as feasible in practice (&&&^^^^3query3^^^^&&&).

The paper also provides specific practical recommendations. It recommends Matérn 5/^^^^3ti:\3^^^^ as a robust kernel choice and repeatedly updates kernel length scale and noise level using leave-one-out cross-validation as described in Rasmussen and Williams, Section 5.4.^^^^3ti:\3^^^^. For implementation, posterior samples may be generated by drawing $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$^^^^3ti:\3^^^^ and setting

$$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$3

where $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$4 is the Cholesky factor of the posterior covariance. For large $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$5 or higher-dimensional settings, the source recommends random Fourier feature approximations and spectral sampling methods, citing SSGP and Hernandez-Lobato (^^^^3ti:\3queries3query3^^^^4), in order to mitigate the cubic scaling of full-grid GP sampling (&&&^^^^3query3^^^^&&&).

A notable applied constraint appears in the two-dimensional terrain example. Because of the high cost of computing four-dimensional integrated kernels, the source states that each iteration and each candidate width used a $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$6 mesh from which ^^^^3query3queries3queries3^^^^ points were randomly subsampled as the representative set. This illustrates that the computational bottleneck of GPDC in multi-modality settings is often the kernel integration rather than the dependence statistic itself (&&&^^^^3query3^^^^&&&).

6. Empirical behavior, comparative position, and open questions

The empirical study in the source covers both sequential function estimation and conventional black-box maximization. For synthetic ^^^^3query3^^^^D function estimation with integral observations, the experiments use 64 random ^^^^3query3^^^^D functions on $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$7, 35 observations sequentially, ^^^^3query3ti:\3queries3^^^^ equidistant grid points, $$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$8 posterior samples, and widths

$$\mathcal{R}(X,Y)= \frac{\mathcal{V}(X,Y)} {\sqrt{\mathcal{V}(X,X)\mathcal{V}(Y,Y)}},$$9

The reported metric is the coefficient of determination $\mathcal{R}(X,Y)\in[0,1]$^^^^3queries3^^^^ between the GP posterior mean and the true function. The paper reports that GP-DC significantly outperforms both random choice and a pure point-observation max-variance policy until approximately step $\mathcal{R}(X,Y)\in[0,1]$^^^^3query3^^^^, and that it selects broad widths early before shifting to narrower widths and eventually to point observations (&&&^^^^3query3^^^^&&&).

For coarse-grained gradient observations, the study uses 48 random ^^^^3query3^^^^D functions, starts with observations at $\mathcal{R}(X,Y)\in[0,1]$^^^^3ti:\3^^^^ and $\mathcal{R}(X,Y)\in[0,1]$3, and then performs 33 sequential observations with widths

$\mathcal{R}(X,Y)\in[0,1]$4

using $\mathcal{R}(X,Y)\in[0,1]$5 and $\mathcal{R}(X,Y)\in[0,1]$6. The paper reports higher and more stable $\mathcal{R}(X,Y)\in[0,1]$7 than random selection and a narrow-gradient baseline, again with a gradual shift toward smaller widths over time (&&&^^^^3query3^^^^&&&).

For the Grand Canyon terrain experiment, the data are a digital elevation model of size $\mathcal{R}(X,Y)\in[0,1]$8 pixels, scaled to $\mathcal{R}(X,Y)\in[0,1]$9, with disk integrals

$\mathcal{R}(X,Y)=0$^^^^3queries3^^^^

The paper reports that a point-only policy performs worst, that GP-DC initially trails random selection up to about step $\mathcal{R}(X,Y)=0$^^^^3query3^^^^, and that GP-DC subsequently overtakes it. In the reported comparison, random selection needs 45 steps to reach $\mathcal{R}(X,Y)=0$^^^^3ti:\3^^^^, whereas GP-DC needs approximately 3^^^^3queries3^^^^ steps, described as about 3^^^^3queries3^^^^\% fewer steps (&&&^^^^3query3^^^^&&&).

For maximization on random ^^^^3query3^^^^D functions, the study uses ^^^^3ti:\3^^^^56 random ^^^^3query3^^^^D functions, 5^^^^3queries3^^^^ observations per function, and compares the proposed GP-dCor, GP-dCov, GP-dCor-X, GP-dCov-X, and GP-MIS against Random, VarMax, PI, EI, GP-UCB, GP-MI, MES, and Brent. Regret is defined as

$\mathcal{R}(X,Y)=0$3

The paper reports that all GP-DC variants outperform the baselines across the full horizon in average regret, with GP-UCB as the strongest baseline and with the GP-DC family exhibiting shorter error bars and lower accumulated regret (&&&^^^^3query3^^^^&&&).

For standard ^^^^3ti:\3^^^^D benchmark functions—Himmelblau, Eggholder, Branin, and Goldstein–Price, optimized via their negatives—the source reports normalized median cumulative regret over steps $\mathcal{R}(X,Y)=0$4 to $\mathcal{R}(X,Y)=0$5, with the random policy fixed at $\mathcal{R}(X,Y)=0$6. The average normalized scores reported for the proposed methods are ^^^^3queries3^^^^.49 for GP-dCor, ^^^^3queries3^^^^.59 for GP-dCov, ^^^^3queries3^^^^.39 for GP-dCor-X, ^^^^3queries3^^^^.5^^^^3ti:\3 for GP-dCov-X, and ^^^^3queries3^^^^.43 for GP-MIS, with GP-dCor-X identified as best overall across the four functions (&&&^^^^3query3^^^^&&&).

The paper’s principal caution is theoretical rather than empirical. It explicitly states that it does not provide formal regret bounds, convergence-rate analysis, or consistency proofs for the distance-correlation acquisitions. The justification is instead conceptual and empirical: if a candidate evaluation is nearly independent of the maximum under the current posterior, its distance correlation with that maximum will be near zero, whereas any functional dependence, linear or nonlinear, will raise the score. This suggests a close relationship to information-seeking acquisition design, but a formal theory comparable to GP-UCB’s regret analysis remains open (&&&^^^^3query3^^^^&&&).