Source-Independent Transferability Estimation (SITE)

• SITE is a transferability estimation framework that ranks pretrained models based on their expected performance on target tasks without using source training data.
• It employs both static methods, which use direct statistical cues, and dynamic methods, which learn to map raw statistics into informative scores, to predict transfer performance.
• SITE research identifies challenges such as benchmark design flaws, trivial model hierarchies, and the need for domain-specific adaptations in areas like medical imaging and surgical phase recognition.

Source-Independent Transferability Estimation (SITE), also called source-free model transferability estimation (SF-MTE), is the problem of predicting how well a pretrained model will transfer to a target task without actually fine-tuning every candidate model on that target and without requiring access to the source training dataset. In the standard formulation, SITE is a model-selection problem over a model hub: it assigns each candidate pretrained model a scalar score intended to correlate with downstream transfer performance after fine-tuning or linear probing, so that the target-side ranking induced by the score approximates the ranking induced by true transferred accuracy (Ding et al., 2024, Singh et al., 7 Oct 2025).

1. Formal problem statement and access assumptions

A common formalization of SITE assumes a labeled target dataset

$\mathcal{D}=\{(\mathbf{x}_n,y_n)\}_{n=1}^N$

and a set of $M$ pretrained models

$\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$

where $\phi_m(x)\in\mathbb{R}^d$ is the feature extractor and $\psi_m$ is the classification head. A SITE metric computes a scalar score $T_m$ for each model, with the intended ranking coherence

$$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$

and downstream performance expressed by

$$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$

In survey form, the same problem is written over a target dataset $\mathcal{D}^T$ and a model hub $\mathcal{M}=\{\phi_i\}_{i=1}^{n_\mathcal{M}}$, with predicted scores $M$0 expected to correlate with ground-truth transferred accuracies $M$1 through

$M$2

This framing makes SITE fundamentally a ranking and selection problem rather than a full transfer-training procedure (Ding et al., 2024).

The defining access constraint is the absence of source training data. Source-free methods use the candidate pretrained models and the target dataset; source-dependent methods additionally require source data. The survey explicitly characterizes SF-MTE input as model hub plus target set, whereas SD-MTE uses model hub plus source set plus target set. Many SITE methods assume labeled target data because they rely on class means, inter-class separation, conditional entropy, or label likelihood, although the survey also notes a trend toward unlabeled target settings, including RankMe, CLID, and some self-supervised evaluation settings (Ding et al., 2024).

A domain-specific extension appears in surgical phase recognition, where the target dataset is a set of static frames sampled from videos and partitioned into $M$3 disjoint subsets, for example different videos or segments. For model $M$4 and subset $M$5, a subset-level score

$M$6

is computed and then aggregated into a global score by a statistic such as mean, minimum, or maximum. This subset-wise formulation reflects heterogeneity across cases rather than assuming a single homogeneous target distribution (Singh et al., 22 Aug 2025).

2. Methodological taxonomy and representative metrics

The survey divides SITE into two major families: static methods and dynamic methods. Static methods compute a score directly from statistical information obtained by running the pretrained model on target data, such as features or logits. Dynamic methods incorporate a learning or mapping mechanism to transform raw statistics into a more informative space before scoring (Ding et al., 2024).

Static methods are further grouped into four categories. Feature structure-based methods assume that a model organizing target features into more class-separable or compact geometry will transfer better. Representative examples include GBC, which models per-class Gaussians and measures separation using the Bhattacharyya coefficient; H-score, which uses a matrix product involving the pseudoinverse of the feature matrix and the covariance of class means; Separation Index (SI), which scores nearest-neighbor label agreement; and CLID, which evaluates KNN clustering learnability. The same family also includes intra-class structure metrics such as TMI and CC-FV, and NC-oriented methods that analyze intra-class variability collapse and inter-class discriminability (Ding et al., 2024).

Bayesian statistic-based methods estimate transferability through likelihood, posterior evidence, or probabilistic compatibility between source model predictions and target labels. LEEP computes the average log-likelihood of the log-expected empirical predictor. $M$7LEEP replaces the output layer with a Gaussian mixture model so that it can be used with unsupervised or self-supervised pretrained models. The survey also places MS-LEEP, E-LEEP, IoU-LEEP, SoftIoU-LEEP, SFDA, and PACTran in this broader probabilistic family (Ding et al., 2024).

Information theory-based methods quantify transferability with entropy- and mutual-information-style criteria. NCE measures the negative conditional entropy between target labels and pretrained features. TransRate measures the mutual information between target features and labels and is interpreted through code-rate arguments. Matrix analysis-based methods examine structure or redundancy in the feature matrix itself; RankMe uses the rank of the feature matrix as a proxy for representational diversity, and H-score variants also sit near this category (Ding et al., 2024).

Dynamic methods include several recurrent design patterns. Energy model methods include ETran, which uses an energy score to quantify whether target data are in-distribution for a candidate model, and PED, which models fine-tuning as a physical system moving toward minimal potential energy. Linear framework methods simulate downstream learning with lightweight linear surrogates; the survey identifies LogME as one of the most important source-free methods because it fits a linear model on target features and estimates the maximum average log evidence of labels given the features. TLogME extends this idea to detection, and SFDA uses regularized Fisher Discriminant Analysis together with self-challenging to simulate fine-tuning. Model vectorization methods such as SynLearn and Model Spider map models and tasks into a shared latent space and rank models by similarity there (Ding et al., 2024).

Across the benchmarking literature, commonly reviewed SITE metrics include LogME, TransRate, SFDA, ETran, PED, LEEP, N-LEEP, H-Score, GBC, LEAD, and NCTI. The benchmark critique paper uses this list to illustrate how diverse methodological assumptions are being compared within a single evaluation regime (Singh et al., 7 Oct 2025).

3. Evaluation protocols, ground truth, and score semantics

SITE methods are usually evaluated by correlating predicted scores with actual downstream performance. The survey identifies Pearson correlation $M$8, Kendall rank correlation $M$9, and weighted Kendall $\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$0 as the common metrics, with the intended behavior being a positive coefficient close to $\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$1 (Ding et al., 2024).

A widely used SITE benchmark in vision adopts a fixed model pool of mostly ImageNet-pretrained CNNs—ResNet34, ResNet50, ResNet101, ResNet151, DenseNet121, DenseNet169, DenseNet201, MobileNet, Inceptionv3, MNASNet, and GoogleNet—and a target suite of mostly image classification datasets, including CIFAR10, CIFAR100, Pets, Aircraft, Food-101, and DTD. Ground truth is obtained by fine-tuning each model on each target dataset, with hyperparameters selected by grid search over learning rates $\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$2 and weight decay $\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$3. The resulting test accuracy is treated as the true transferability $\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$4 for model $\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$5 (Singh et al., 7 Oct 2025).

The dominant benchmark score is weighted Kendall’s tau, defined as

$\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$6

and

$\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$7

with hyperbolic weights

$\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$8

Because these weights emphasize top-ranked models, errors near the top of the predicted ranking dominate the evaluation (Singh et al., 7 Oct 2025).

The benchmark critique argues that a practical transferability metric should not only rank models correctly but also assign score magnitudes that are meaningful. To formalize this, it defines pairwise differences

$\{\Phi_m=(\phi_m,\psi_m)\}_{m=1}^M,$9

and

$\phi_m(x)\in\mathbb{R}^d$0

with the ideal property

$\phi_m(x)\in\mathbb{R}^d$1

The proposed fidelity evaluation computes the Pearson correlation between pairwise accuracy differences and pairwise metric differences, thereby testing whether score gaps can be interpreted as performance gaps rather than mere ordinal cues (Singh et al., 7 Oct 2025).

4. Benchmark pathologies and the critique of static leaderboards

A central recent claim is that SITE is an appealing idea but that the benchmarks used to evaluate SITE metrics are fundamentally misleading. The critique identifies three fundamental problems: unrealistic model spaces, static leaderboards or fixed performance hierarchies, and fixed rank correlation metrics that ignore fidelity to accuracy differences (Singh et al., 7 Oct 2025).

The first problem is the benchmark model pool itself. It is dominated by variants of the same families, mostly ResNets and DenseNets, and mixes architectures across very different sizes. The authors argue that this makes the problem too easy because larger models generally outperform smaller ones predictably. They further criticize including MobileNet and MNASNet because these are edge-oriented models and almost always occupy the bottom ranks, adding little meaningful challenge. Their recommended fixes are to exclude edge-oriented models when irrelevant, use at most one representative per architecture family, and match models on computational budget such as parameter count or FLOPs. In an ablation that removes ResNet-152, ResNet-101, DenseNet-169, and DenseNet-201, most SITE metrics suffer a sharp drop; except on DTD and Pets, all metrics fall below $\phi_m(x)\in\mathbb{R}^d$2, and the paper states that none of the metrics are robust to changes in the model space (Singh et al., 7 Oct 2025).

The second problem is the existence of a static ranking heuristic that effectively solves the benchmark without using target data. The paper reports that ResNet-152 is ranked first in 8 of the 10 datasets in the benchmark, and that second place is always one of the top 3 models: ResNet-152, DenseNet-201, or ResNet-101. It then evaluates the fixed, dataset-agnostic ordering

$\phi_m(x)\in\mathbb{R}^d$3

which uses only model size and family ordering. This heuristic achieves average $\phi_m(x)\in\mathbb{R}^d$4, whereas the best evaluated SITE metric, LogME, reaches $\phi_m(x)\in\mathbb{R}^d$5. The same heuristic obtains $\phi_m(x)\in\mathbb{R}^d$6 on Aircraft, $\phi_m(x)\in\mathbb{R}^d$7 on CIFAR10, $\phi_m(x)\in\mathbb{R}^d$8 on CIFAR100, $\phi_m(x)\in\mathbb{R}^d$9 on DTD, $\psi_m$0 on Food, and $\psi_m$1 on Pets. The paper’s interpretation is that the benchmark rewards memorizing a fixed hierarchy rather than true task-specific transferability estimation (Singh et al., 7 Oct 2025).

The third problem is score semantics. Nearly all metrics have weak correlation with accuracy differences under the pairwise fidelity test. The paper gives a concrete example: for LogME on Pets, a score difference of $\psi_m$2 can correspond to an accuracy gap anywhere from $\psi_m$3 to $\psi_m$4. This indicates that even when a metric gets the ranking roughly right, its values are not reliably interpretable as calibrated performance gaps (Singh et al., 7 Oct 2025).

The same critique therefore recommends a different benchmark design. Reproducibility should include releasing metric code, datasets and source-task links, pretrained models, computed scores, final accuracies, and evaluation scripts. The model space should be diverse and non-trivial, including different architectural paradigms such as ConvNeXt, ViT, Swin, and MLP-Mixer, while keeping computational budgets comparable and avoiding trivial hierarchies. The dataset suite should be diverse and challenging, with sufficient performance headroom and domain variety, including fine-grained classification, medical imaging, satellite imagery, and texture analysis. Finally, the model pool and dataset pool should be co-designed to yield performance spread and rank dispersion, so that different models win on different tasks rather than reproducing a static leaderboard (Singh et al., 7 Oct 2025).

5. Domain-specific SITE: medical imaging and surgical phase recognition

Medical image classification exposes a setting in which natural-image transferability metrics are often suboptimal. The medical imaging study argues that transferability in this domain is not only a matter of how “good” source features are on the target but also of how adaptable the source representation is to subtle local patterns and texture. It identifies two failure modes of natural-image metrics: medical images rely on subtle local patterns and texture rather than ImageNet-style globally recognizable objects, and many existing metrics produce self-source bias by suggesting that the target dataset is the best source for itself (Juodelyte et al., 2024).

To address this, the paper proposes a source-independent transferability estimator with two components. The feature-suitability term uses penultimate-layer target features

$\psi_m$5

followed by Neighborhood Component Analysis, which learns a linear projection $\psi_m$6 to increase class separability,

$\psi_m$7

A 5-NN classifier then estimates $\psi_m$8, producing

$\psi_m$9

The adaptability term performs a single backward pass with a triplet margin loss on penultimate-layer embeddings,

$T_m$0

and defines a gradient-ratio score over the first two convolutional layers,

$T_m$1

After normalization across candidate sources, the final score is

$T_m$2

On a medical source-selection benchmark with 14 source datasets, 11 MedMNIST targets, ResNet18 throughout, and more than 20,000 trained models, this method achieves the highest weighted Kendall correlation on 8 targets and average rank $T_m$3, ahead of $T_m$4LEEP at $T_m$5, LEEP at $T_m$6, LogME at $T_m$7, PARC at $T_m$8, NCTI at $T_m$9, and SFDA at $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$0. The paper also reports that medical-specific sources outperform ImageNet on 4 of 11 targets, and gives OCT as an example where RadImageNet reaches $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$1 AUC versus $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$2 for ImageNet (Juodelyte et al., 2024).

A different domain-specific specialization appears in surgical phase recognition. Here SITE is formalized over static frames sampled from videos, and the dataset is partitioned into disjoint subsets so that per-subset transferability scores can be aggregated by mean, minimum, or maximum. The benchmark uses two datasets, RAMIE and AutoLaparo, and a diverse set of pretrained backbones that includes MetaFormer, MetaFormer_in21k, SurgeNet_Cholec, SurgeNet_Public, SurgeNet_XL, TimeSformer, ConvNeXtV2, SurgeNet_ConvNeXtV2, PVTv2, SurgeNet_PVTv2, GastroNet_RN50, GastroNet_ViTS, EndoFM, GSViT, and EndoViT. Ground truth comes from a two-stage TeCNO-style fine-tuning pipeline with backbone training on frame-wise phases followed by MS-TCN temporal refinement, and rankings are evaluated with Pearson correlation $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$3 and weighted Kendall’s tau $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$4. In this setting, LogME is the strongest-performing metric overall, especially under minimum aggregation. On AutoLaparo, mean/min/max LogME obtain $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$5, $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$6, and $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$7; on RAMIE, they obtain $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$8, $$T_m \geq T_n \quad \text{iff} \quad \hat{\Phi}_m \text{ is more likely to predict correctly than } \hat{\Phi}_n,$$9, and $$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$0. H-Score has only weak predictive ability, and all TransRate variants have negative Kendall’s $$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$1, around $$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$2, indicating systematic inverse ranking. The ablations show that when candidate models differ only by about $$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$3–$$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$4, the ranking degrades or even inverts, which the paper interprets as a loss of discriminative power when candidate performance is tightly clustered (Singh et al., 22 Aug 2025).

Taken together, these domain studies suggest that SITE behavior is strongly task- and modality-dependent. A plausible implication is that feature quality alone is often insufficient in domains where local texture, case heterogeneity, or temporal structure materially affect adaptation, and that domain-specific benchmark construction can change not only relative metric performance but also the sign of the observed correlation, as in the reported cross-domain transfer from ImageNet to medical targets (Juodelyte et al., 2024, Singh et al., 22 Aug 2025).

6. Related approaches, limitations, and research directions

The survey presents SITE as an efficient prioritization tool rather than a guarantee of best downstream transfer. It recommends interpreting SITE scores as model-selection proxies, preferring rankings over raw scores unless the method is calibrated, and treating small score differences cautiously because benchmarking is not standardized and transferability is affected by the downstream transfer algorithm itself (Ding et al., 2024).

Several limitations recur across the literature. Results can change with dataset choice, architecture, training details, and evaluation protocol. Many methods depend on labeled target data, which restricts applicability in fully unsupervised transfer settings. Some metrics exhibit high-dimensional fragility; H-score is explicitly noted to degrade in high-dimensional settings, motivating a shrinkage-based variant. Many methods are task-specific, with specialized metrics for classification, segmentation, detection, regression, multimodal learning, and medical imaging. The survey also emphasizes weak standardization: there is no universal benchmark that jointly standardizes model hubs, target tasks, and evaluation procedures while separating source-free from source-dependent methods fairly (Ding et al., 2024).

Open problems identified in the survey include robustness across datasets, architectures, random seeds, preprocessing, and transfer settings; unified benchmarks; better support for unlabeled target tasks; heterogeneous model hubs involving multiple architectures, pretrained datasets, algorithms, and layers; transferability beyond the standard pretrain-then-finetune paradigm; and transferability estimation for large foundation models such as GPT-like and CLIP-like systems (Ding et al., 2024).

A related but conceptually distinct line of work is Principal Gradient Expectation (PGE), which is positioned against feature-distribution-based estimators and contrasted conceptually with SITE-like methods. PGE defines

$$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$5

using a restart scheme over random initializations, computes only backbone gradients, and ranks sources by the normalized gap between source and target principal gradient expectations,

$$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$6

The paper presents PGE as stable, reliable, and efficient, reporting an average Kendall’s $$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$7 of $$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$8 across the reported classification targets and a timing result on CIFAR10$$\frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_m) \geq \frac{1}{N}\sum_{n=1}^N p(y_n|\mathbf{x}_n; \hat{\Phi}_n).$$9CIFAR100 with ResNet18 of $\mathcal{D}^T$0s for PGE versus roughly $\mathcal{D}^T$1k s for the comparison methods. It characterizes PGE as source-independent in a stronger operational sense because it does not depend on pre-training parameters or source-model feature distributions (Qi et al., 2022).

The broader trajectory of SITE research therefore combines two simultaneous demands. One is methodological: scores must be informative under realistic source-free access constraints. The other is evaluative: benchmarks must avoid trivial hierarchies, static winner effects, and score semantics that collapse to rank-only proxies. This suggests that progress in SITE will depend as much on benchmark design and domain-specific task construction as on the invention of new transferability metrics themselves (Singh et al., 7 Oct 2025, Ding et al., 2024).