Dataset Performance-Based Similarity Estimation

• Dataset Performance-Based Similarity Estimation (DPSE) is a framework that quantifies dataset similarity by linking latent feature distance with downstream model performance degradation.
• DPSE employs diverse methods, including UMAP-based clustering, autoencoder reconstruction errors, and bidirectional cross-generalization, to estimate transferability between datasets.
• Empirical studies in wireless sensing, computer vision, and meta-learning confirm that DPSE effectively ranks datasets to guide model selection, augmentation, and transfer learning decisions.

Searching arXiv for the cited DPSE-related papers to ground the article in the current literature. arXiv search query: id:([2412.05556](/papers/2412.05556)) OR id:([2605.22467](/papers/2605.22467)) OR id:([2308.03580](/papers/2308.03580)) OR id:([2001.04893](/papers/2001.04893)) OR id:([1902.06585](/papers/1902.06585)) OR id:([2603.19888](/papers/2603.19888)) OR id:([2510.10866](/papers/2510.10866)) OR id:([2312.04078](/papers/2312.04078)) Dataset Performance-Based Similarity Estimation (DPSE) denotes a family of approaches that quantify how similar datasets are by tying similarity to downstream model behavior rather than to distributional comparison alone. Across the literature, the term covers distance measures whose values correlate with cross-dataset performance degradation, composite scores that combine distance and observed performance, retrieval objectives that identify datasets with similar performance profiles, and bidirectional cross-generalization criteria for transfer learning. The concept has been instantiated in wireless communications and sensing, synthetic-to-real computer vision, transfer learning, and meta-learning; at the same time, the literature shows that the term is not standardized and that no single universal DPSE formula exists (Morais et al., 2024, Bartkowiak et al., 21 May 2026, Klironomos et al., 20 Mar 2026, Stolte et al., 2023).

1. Core idea and terminological variation

In one prominent formulation, DPSE is a principled, model-agnostic framework for quantifying how “close” two wireless communications or sensing datasets are in a way that predicts the performance loss of a model when it is trained on one dataset and evaluated on another. The central intuition is that if two datasets are similar in terms of their underlying data distributions after mapping into a suitable latent space, then a model trained on one will perform almost as well on the other; conversely, large dataset distances correlate with significant performance drops (Morais et al., 2024).

A different formulation defines DPSE as an explicit score that fuses a model’s performance on an unseen dataset with the feature-space distance between the unseen dataset and the training dataset. In that usage, high DPSE means both that the unseen dataset is near the training dataset in feature space and that the trained model performs well on it (Achara et al., 2023).

In meta-learning, DPSE is posed as a retrieval problem: datasets are considered similar if they exhibit similar behavior across pipelines. In that setting, the objective is to identify datasets with similar performance patterns rather than to measure only raw sample-space proximity (Klironomos et al., 20 Mar 2026).

The terminological heterogeneity is explicit in the review literature. Stolte et al. state that no method under the name “Dataset Performance-Based Similarity Estimation” is introduced in their taxonomy, and identify the closest match as “Methods based on binary classification,” in which similarity is inferred from how well a classifier can discriminate two datasets (Stolte et al., 2023). This indicates that DPSE is best understood as a research direction organized around performance-linked similarity, not as a single canonical estimator.

2. Formal objectives and mathematical formulations

A standard DPSE objective is to relate inter-dataset distance to cross-dataset performance degradation. For datasets $D_1,\dots,D_n$, the wireless formulation produces a distance matrix

$D_{ij}=d(D_i,D_j)$

and a performance drop matrix

$\Delta P_{ij}=P_{ii}-P_{ij},$

where $P_{ii}$ is the baseline performance of a model trained and tested on $D_i$. The intended outcome is that $d(D_i,D_j)$ and $\Delta P_{ij}$ exhibit a strong, ideally monotonic, relationship. The association is measured by the Pearson coefficient

$$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$

and high $|\rho|$, for example $>0.85$, indicates that the chosen metric predicts model degradation across datasets (Morais et al., 2024).

A second formalization uses explicit feature-space distances. Let $D_{ij}=d(D_i,D_j)$0 be the primary dataset, $D_{ij}=d(D_i,D_j)$1 a secondary dataset, and $D_{ij}=d(D_i,D_j)$2 a feature extractor. After optional PCA projection, the image-image distance is Euclidean, the image-dataset distance is the sum of distances from one image in $D_{ij}=d(D_i,D_j)$3 to all images in $D_{ij}=d(D_i,D_j)$4, and the dataset-dataset distance is the mean of these image-dataset distances. Performance and distance are then normalized into $D_{ij}=d(D_i,D_j)$5, and the DPSE score is defined by the convex combination

$D_{ij}=d(D_i,D_j)$6

Here, similarity is a scalar that simultaneously encodes generalization quality and domain shift (Achara et al., 2023).

A third formulation replaces distance estimation with bidirectional cross-generalization. The Cross-Learning Score (CLS) is defined as

$D_{ij}=d(D_i,D_j)$7

Lower CLS implies higher predictive similarity. In this construction, similarity is not proximity in feature space but symmetry of cross-domain predictive error (Sun et al., 13 Oct 2025).

In meta-learning, DPSE can also be defined as cosine similarity between dataset embeddings: $D_{ij}=d(D_i,D_j)$8 Ground-truth similarity is computed from performance vectors on common pipelines,

$D_{ij}=d(D_i,D_j)$9

and retrieval quality is evaluated by whether embedding-space neighbors recover datasets with similar empirical performance profiles (Klironomos et al., 20 Mar 2026).

These formulations show that the “performance-based” aspect may enter as correlation with performance drop, direct inclusion of observed performance, similarity of performance vectors, or bidirectional transfer loss. This suggests that DPSE is a unifying principle rather than a single mathematical object.

3. Methodological families

The methodological spectrum is broad, but several recurring families can be distinguished (Temel et al., 2019, Hwang et al., 2020, Morais et al., 2024, Bartkowiak et al., 21 May 2026, Klironomos et al., 20 Mar 2026, Sun et al., 13 Oct 2025).

Instantiation	Core representation	Similarity/performance linkage
Feature-signature methods	Aggregated handcrafted or deep features	Distance to a reference signature estimates performance
SimEx	Fleet of pretrained autoencoders	Reconstruction error ranks reference datasets
Wireless DPSE	UMAP latent space with K-means or KNN clusters	Wasserstein or Euclidean distance correlates with performance drop
SADGE	Appearance and geometry scores	Constrained bilinear fusion predicts synthetic-to-real transfer
KGmetaSP	Dataset and pipeline KG embeddings	Cosine retrieval approximates similarity of performance profiles
CLS	Cross-evaluated predictors	Average bidirectional loss estimates transferability

Feature-signature approaches aggregate per-image descriptors into a dataset signature, often by taking a mean feature vector $\Delta P_{ij}=P_{ii}-P_{ij},$0, and compare datasets by distances such as $\Delta P_{ij}=P_{ii}-P_{ij},$1, $\Delta P_{ij}=P_{ii}-P_{ij},$2, cosine, or Canberra. In a controlled object-recognition setting, handcrafted features based on color, texture, and shape were compared with deep features from VGG16, and a monotonic mapping from feature distance to recognition performance was evaluated by Spearman’s rank-order correlation (Temel et al., 2019).

Reconstruction-based DPSE is exemplified by SimEx. Reference data are partitioned into subsets $\Delta P_{ij}=P_{ii}-P_{ij},$3, one autoencoder $\Delta P_{ij}=P_{ii}-P_{ij},$4 is pretrained per subset, and an unknown dataset $\Delta P_{ij}=P_{ii}-P_{ij},$5 is passed through each $\Delta P_{ij}=P_{ii}-P_{ij},$6. The dataset-level reconstruction error

$\Delta P_{ij}=P_{ii}-P_{ij},$7

is converted into similarity by $\Delta P_{ij}=P_{ii}-P_{ij},$8. This treats transfer-relevant similarity as the extent to which a specialized autoencoder can reconstruct the unknown data without further training (Hwang et al., 2020).

Latent-topological DPSE uses a task-specific, model-agnostic pipeline. UMAP constructs a weighted $\Delta P_{ij}=P_{ii}-P_{ij},$9-nearest-neighbor graph in the original space with connection strengths

$P_{ii}$0

defines low-dimensional weights

$P_{ii}$1

and minimizes a fuzzy topological cross-entropy. The resulting latent datasets are clustered by K-means or KNN, producing discrete cluster distributions. Inter-dataset distance is then measured either by the $P_{ii}$2-Wasserstein distance between cluster distributions or by the average pairwise Euclidean distance between cluster centroids,

$P_{ii}$3

The same work gives practical guidelines: $P_{ii}$4 should start with $P_{ii}$5, $P_{ii}$6 values in $P_{ii}$7 are typical, and $P_{ii}$8 suffices for many tasks. It also states that UMAP scales $P_{ii}$9 and that distance computations in $D_i$0 cost $D_i$1 (Morais et al., 2024).

Synthetic-to-real DPSE is instantiated by SADGE. For each real image $D_i$2, a matching synthetic image $D_i$3 is obtained either by aligned pairs or by retrieval over a pool of size $D_i$4. Appearance similarity is cosine similarity between pretrained feature embeddings,

$D_i$5

and geometric similarity is the number of inliers found by MASt3R under RANSAC, log-stabilized after dataset-level averaging. After z-score normalization, the final score is

$D_i$6

The best configuration uses DINOv3 appearance similarity and MASt3R geometric consistency with a constrained bilinear interaction (Bartkowiak et al., 21 May 2026).

KG-based DPSE represents datasets and pipelines inside a unified knowledge graph. KGmetaSP merges dataset metadata and pipeline structure, then learns embeddings with walk-based RDF2Vec extended with MKGA for numeric literals. Random walks use 10 walks per entity with length 20; Word2Vec is trained with embedding dimensionality $D_i$7, window size $D_i$8, negative samples $D_i$9, epochs $d(D_i,D_j)$0, and $d(D_i,D_j)$1. Dataset embeddings are aggregated either from data-entity nodes, from evaluated pipelines, or by the simple meta-embedding

$d(D_i,D_j)$2

and similarity is then cosine similarity in embedding space (Klironomos et al., 20 Mar 2026).

Label-aware DPSE is represented by CLS. Rather than estimating feature distributions, it cross-evaluates predictors trained on source and target. The same work establishes that in probit-regression and LDA settings, CLS is a decreasing function of the cosine-angle between decision normals, thereby interpreting dataset similarity as decision-boundary similarity. It also introduces a transfer-zone framework and an encoder-head variant for modern deep transfer pipelines (Sun et al., 13 Oct 2025).

4. Empirical evaluation across domains

The empirical literature evaluates DPSE with heterogeneous criteria, but the common requirement is alignment between the similarity estimate and downstream performance.

In wireless communications and sensing, DPSE was validated on an unsupervised channel state information compression task using a ray-traced DeepMIMO ASU campus dataset with $d(D_i,D_j)$3 users and $d(D_i,D_j)$4 truncated channel matrices. Convolutional autoencoders compressed CSI to a 32-dim bottleneck and performance was measured by normalized MSE in dB. Raw-input-space baselines showed correlations of approximately $d(D_i,D_j)$5 for Wasserstein, $d(D_i,D_j)$6 for Energy, and $d(D_i,D_j)$7 for pairwise Euclidean distance, while Grassmann had negative correlation. An AE-derived latent space yielded the upper-bound correlation $d(D_i,D_j)$8, and UMAP2 plus cluster centroids with either Euclidean or Wasserstein distance achieved $d(D_i,D_j)$9. The same study reports that UMAP drastically reduces runtime compared to raw Wasserstein, with an example reduction from $\Delta P_{ij}$0 to tens of seconds, and that its UMAP-based metrics offered $\Delta P_{ij}$1 improvement in correlation with model performance relative to baselines (Morais et al., 2024).

In synthetic-to-real computer vision, SADGE was evaluated on five public benchmark families and 15 dataset-level variants comprising $\Delta P_{ij}$2 image pairs. The downstream tasks were object detection, semantic segmentation, and pose estimation. SADGE with DINOv3 and MASt3R achieved Pearson $\Delta P_{ij}$3 and Spearman $\Delta P_{ij}$4 over all $\Delta P_{ij}$5 variants, with approximate $\Delta P_{ij}$6. The strongest single-factor baselines were geometry-only MASt3R with $\Delta P_{ij}$7 and appearance-only LPIPS with $\Delta P_{ij}$8. Leave-one-dataset-out splits yielded $\Delta P_{ij}$9, and SADGE remained the top predictor in every split (Bartkowiak et al., 21 May 2026).

SimEx evaluates inter-dataset and inter-class similarity by agreement with transfer-learning rankings. On MNIST, Rotated-MNIST, Background-MNIST, Fashion-MNIST, and EMNIST-Letters, the reported Spearman correlations between SimEx rankings and transfer-learning rankings reached $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$0 to $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$1 for several base datasets. Runtime latency was approximately $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$2 per $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$3 comparison on GPU, whereas the best transfer-learning baseline took $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$4, yielding $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$5 speed-up. In inter-class augmentation experiments, SimEx-based pairing yielded higher final test accuracies than sample-space or embedding-distance pairings, and within-dataset confusion analysis showed higher mean Spearman $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$6 than sample-based baselines (Hwang et al., 2020).

Feature-similarity-based performance estimation for object recognition under changes in background, acquisition device, and object orientation reported that deep learning-based image representations can estimate recognition performance variation with a Spearman’s rank-order correlation of $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$7 (Temel et al., 2019).

In meta-learning, KGmetaSP constructs a benchmark of $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$8 OpenML experiments and a MetaExe-KG containing approximately $$\rho_{D,\Delta P}=\frac{\mathrm{cov}(D,\Delta P)}{\sigma_D\sigma_{\Delta P}},$$9 entities and $|\rho|$0 triples. For DPSE retrieval at threshold $|\rho|$1, the best KGmetaSP variant $|\rho|$2 achieved Hit@1 $|\rho|$3, Hit@2 $|\rho|$4, Hit@5 $|\rho|$5, and AvgHit $|\rho|$6. The best NDCG results were obtained by $|\rho|$7, with NDCG@1 $|\rho|$8, NDCG@2 $|\rho|$9, NDCG@5 $>0.85$0, and AvgNDCG $>0.85$1. Removing the MLSea-KG enrichment reduced Hit@1 from $>0.85$2 to $>0.85$3 and NDCG@1 from $>0.85$4 to $>0.85$5 (Klironomos et al., 20 Mar 2026).

CLS was evaluated on synthetic classification and regression settings and on real transfer tasks. In every linear-boundary synthetic case, oracle and estimated CLS tracked true cosine similarity almost perfectly with $>0.85$6. The transfer-zone analysis showed that negative zones produced no helpful transfer methods, ambiguous zones produced mixed outcomes, and positive zones produced uniformly helpful transfer. In real tasks, encoder-head CLS predicted positive transfer for Kaggle Dogs vs. Wolves and negative transfer for Cats vs. Dogs and Horses vs. Camels when the target was Roboflow Dogs vs. Wolves; the subsequent fine-tuning outcomes matched those predictions. On USPS as target, encoder-head CLS placed both MNIST and EMNIST in the positive-transfer zone, and transfer fine-tuning produced relative error reductions of $>0.85$7 and $>0.85$8 (Sun et al., 13 Oct 2025).

A separate line of work emphasizes that model performance and dataset distance can jointly guide generalization. In crack-detection experiments, adding only $>0.85$9, $D_{ij}=d(D_i,D_j)$00, or $D_{ij}=d(D_i,D_j)$01 carefully selected images from each unseen dataset improved unseen-data F-score, and DPSE-guided ranking of candidate architectures selected better generalizers than ranking by training-set accuracy or distance alone (Achara et al., 2023).

5. Applications and decision support

Several applications recur across domains. In the wireless setting, high correlation between dataset distance and performance drop enables prediction of performance on new datasets without retraining, selection of which real or simulated datasets to augment for best generalization, and detection of dataset shifts that guide transfer-learning decisions (Morais et al., 2024).

In computer vision, zero-shot DPSE addresses the bottleneck of choosing among synthetic variants without expensive downstream training. SADGE is explicitly designed to estimate which synthetic dataset will yield the best performance on a real target dataset for object detection, semantic segmentation, or pose estimation, and it does so by combining appearance and geometric signals rather than relying on either alone (Bartkowiak et al., 21 May 2026).

In transfer learning, DPSE can be used not merely to rank datasets but to classify them into transfer regimes. CLS defines thresholds

$D_{ij}=d(D_i,D_j)$02

and labels a source as Positive Transfer if $D_{ij}=d(D_i,D_j)$03, Ambiguous if $D_{ij}=d(D_i,D_j)$04, and Negative Transfer if $D_{ij}=d(D_i,D_j)$05. This makes the similarity estimate operational in source selection (Sun et al., 13 Oct 2025).

DPSE has also been used for model selection and low-shot adaptation. The crack-detection study reports that combining distance with model performance helps in selecting an appropriate model or architecture from a pool of candidate architectures, and that adding only a small number of unseen images into training reduces training and annotation costs while improving generalization in dynamic environments (Achara et al., 2023).

In meta-learning, DPSE supports distance-based retrieval of related datasets and pipeline-agnostic meta-models for pipeline performance estimation. This suggests a role for DPSE as an indexing mechanism over prior experimental records, rather than only as a direct measure of domain shift (Klironomos et al., 20 Mar 2026).

SimEx extends the same decision-support logic to early comparison against many known datasets or classes without newly training anything at comparison time. Its use cases include informed dataset selection for transfer or augmentation and analysis of inter-class confusion structure (Hwang et al., 2020).

6. Limitations, assumptions, and contested points

A central limitation is conceptual rather than computational: DPSE is not a single method family with a shared formal core. The review literature treats the nearest analogue as classifier-based dataset comparison rather than as a distinct standardized class, and also notes that performance-based measures depend strongly on the predictive model and its tuning (Stolte et al., 2023). A common misconception is therefore to treat all DPSE scores as interchangeable. The surveyed papers do not support that interpretation.

Method-specific assumptions are substantial. The wireless UMAP-based framework assumes unsupervised or self-supervised tasks with no label-driven structure, is sensitive to embedding hyperparameters, and may not directly apply to classification or regression without adaptation; cross-validation is recommended (Morais et al., 2024). SimEx assumes that the autoencoders have sufficient capacity to learn each reference subset and that the number of autoencoders matches the granularity of the desired comparison (Hwang et al., 2020).

Pretrained components introduce domain dependence. SADGE relies on DINOv3 and MASt3R; if target domains differ substantially from their pretraining distributions, such as medical imagery or thermal data, reliability may degrade. Geometry matching can fail on textureless, highly repetitive, deformable, or heavily occluded objects; in such cases the geometric term may be zero and the metric reduces to appearance only. The same work also states that image-based metrics cannot capture label-noise, annotation-policy differences, temporal cues, depth channels, or non-RGB modalities, and that safety-critical applications require real-domain validation because SADGE is a comparative ranking tool, not a certification (Bartkowiak et al., 21 May 2026).

More generally, DPSE trades universality for task relevance. Distribution-free distances aim to compare datasets independent of downstream models, whereas DPSE methods explicitly align similarity with performance, transferability, or generalization. This suggests that DPSE is most appropriate when the objective is operational—predicting degradation, ranking candidate sources, deciding whether transfer is worthwhile, or selecting data for augmentation—rather than purely distributional.