TransRate: Transfer Learning Metric

• TransRate is a transferability estimation metric that quantifies the suitability of pre-trained models by approximating mutual information through coding rates, balancing feature completeness and class-conditional compactness.
• It operates efficiently with a single forward pass over labeled target data, enabling both layer and model selection without expensive fine-tuning.
• Empirical evaluations demonstrate significant speedups and competitive ranking performance, though its accuracy can vary in domain-specific tasks such as surgical phase recognition.

Searching arXiv for TransRate and related transferability estimation papers. TransRate is a transferability estimation metric for transfer learning that scores a pre-trained representation by approximating the mutual information between target-task features and target labels, with the explicit aim of selecting a pre-trained model or intermediate layer before expensive fine-tuning (Huang et al., 2021). It is formulated to be source-data-free, optimization-free on the target task, and computable from a single forward pass over labeled target examples. In the original formulation, TransRate is positioned as a feature-based criterion that evaluates whether a representation is both sufficiently informative for the target task and class-conditionally compact; later work has also examined its behavior in domain-specific source-independent transferability estimation settings, including surgical phase recognition (Singh et al., 22 Aug 2025).

1. Problem setting and historical placement

TransRate was introduced in the context of transfer learning under model-selection and layer-selection constraints. A pre-trained network is written as

$F = f_{L+1} \circ \cdots \circ (f_2 \circ f_1),$

where the first $L$ layers constitute the feature extractor and $f_{L+1}$ is the classifier. If the first $K$ layers are transferred, the transferred feature extractor is

$g = f_K \circ \cdots \circ (f_2 \circ f_1),$

and the extracted feature for target input $x_i$ is

$z_i = g(x_i).$

The original paper defines transferability through the expected test log-likelihood after optimal transfer learning,

$$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$

where $z^* = g^*(x)$, and $g^*, w^*$ are the optimal transferred feature extractor and head after training on the target task (Huang et al., 2021).

The motivating claim is that existing estimators were either expensive because they required target-task training or optimization, or limited because they did not support layer-wise comparison. Taskonomy and Task2Vec were described as closely following actual transfer learning and therefore requiring target-task training or expensive optimization. RSA and DEPARA avoided full fine-tuning but required an encoder pre-trained on the target task or similarly expensive target-side machinery. NCE and LEEP were efficient but unsuitable for layer selection because they depended primarily on labels or source-label predictions rather than intermediate representations. H-Score and LogME assumed a linear classifier on top of the representation and typically considered only the penultimate layer. TransRate was proposed to retain efficiency while supporting both model selection and layer selection across supervised, unsupervised, and self-supervised pre-trained models (Huang et al., 2021).

2. Information-theoretic formulation

The central definition of TransRate is the mutual information between the target labels $L$0 and the extracted features $L$1:

$L$2

Here $L$3 denotes differential entropy, $L$4 denotes Shannon entropy, and $L$5 is a quantized version of $L$6 (Huang et al., 2021).

Because direct mutual-information estimation in high-dimensional continuous spaces is difficult, TransRate replaces entropy by coding rate. For $L$7 samples

$L$8

the coding rate is defined as

$L$9

where $f_{L+1}$0 is the feature dimension and $f_{L+1}$1 is the distortion rate. For class $f_{L+1}$2, with feature matrix $f_{L+1}$3, the class-conditional coding rate is

$f_{L+1}$4

The practical TransRate score is therefore

$f_{L+1}$5

Expanded directly,

$f_{L+1}$6

This construction makes TransRate a label-aware but training-free criterion: it requires target labels only to partition features by class, not to train a target classifier (Huang et al., 2021).

The original paper interprets the two terms representationally. The global coding-rate term measures completeness, meaning whether the representation contains sufficient information for the target task. The negative class-conditional term measures compactness, rewarding within-class concentration. High TransRate therefore corresponds to representations that are both complete and compact (Huang et al., 2021).

3. Computational procedure and implementation

The operational pipeline is deliberately minimal. For a chosen pre-trained model or layer, one performs a single forward pass over the labeled target dataset to extract features $f_{L+1}$7, forms the matrix $f_{L+1}$8, centers the features by subtracting the empirical mean, optionally normalizes scale across models by

$f_{L+1}$9

partitions features into classwise subsets $K$0, computes the total and class-conditional coding rates, and returns their difference as the TransRate score (Huang et al., 2021).

The paper gives a commutative determinant form,

$K$1

which follows from Sylvester’s determinant theorem. This permits computing a $K$2 determinant when $K$3 and an $K$4 determinant when $K$5. An equivalent singular-value form is

$K$6

where $K$7 are the nonzero singular values of $K$8 (Huang et al., 2021).

The default distortion parameter in the implementation is $K$9. The appendix reports total complexity

$g = f_K \circ \cdots \circ (f_2 \circ f_1),$0

when $g = f_K \circ \cdots \circ (f_2 \circ f_1),$1 and $g = f_K \circ \cdots \circ (f_2 \circ f_1),$2, with classwise computations parallelizable. In a timing example for ImageNet $g = f_K \circ \cdots \circ (f_2 \circ f_1),$3 CIFAR-100 transfer, fine-tuning ResNet-18 required $g = f_K \circ \cdots \circ (f_2 \circ f_1),$4 s, whereas TransRate required $g = f_K \circ \cdots \circ (f_2 \circ f_1),$5 s, corresponding to roughly $g = f_K \circ \cdots \circ (f_2 \circ f_1),$6 speedup in that setup. The implementation uses slogdet for numerical stability. For regression tasks, the paper extends the method by ranking target values and binning them into $g = f_K \circ \cdots \circ (f_2 \circ f_1),$7 equal-frequency ranges, after which the same class-conditional coding-rate construction is applied (Huang et al., 2021).

4. Theoretical properties and interpretive claims

The theoretical justification proceeds by replacing differential entropy and mutual information with coding-rate approximations derived from rate-distortion arguments. The paper states

$g = f_K \circ \cdots \circ (f_2 \circ f_1),$8

and uses coding rate as a tractable surrogate for quantized entropy (Huang et al., 2021).

A central claim is the connection between TransRate and downstream target-task log-likelihood. Under the assumption of a uniform target label distribution,

$g = f_K \circ \cdots \circ (f_2 \circ f_1),$9

the appendix states

$x_i$0

where $x_i$1 is the maximal log-likelihood with the pre-trained feature extractor $x_i$2 and optimal head $x_i$3. The intended interpretation is that TransRate is theoretically tied to the optimal target-task log-likelihood and not merely to an ad hoc proxy (Huang et al., 2021).

The score is nonnegative:

$x_i$4

with equality when all class covariance structures coincide,

$x_i$5

The paper also gives an upper bound with equality when different classes are mutually orthogonal,

$x_i$6

For binary classification, under fixed within-class second-order structure, the score is maximized when the two classes are orthogonal and minimized when the class feature sets coincide. These results formalize the completeness-and-compactness interpretation in terms of class separability (Huang et al., 2021).

5. Empirical performance in the original evaluations

The original evaluation spans 32 pre-trained models and 16 downstream tasks, with Pearson correlation $x_i$7, Kendall’s $x_i$8, and weighted Kendall’s $x_i$9 used to compare estimated transferability against actual downstream performance. The paper emphasizes weighted Kendall’s $z_i = g(x_i).$0 because transferability estimation is often used to identify the top candidate rather than only to approximate absolute performance (Huang et al., 2021).

For source selection among 11 source datasets with ResNet-18 models, TransRate achieved $z_i = g(x_i).$1, $z_i = g(x_i).$2, and $z_i = g(x_i).$3 on CIFAR-100, and $z_i = g(x_i).$4, $z_i = g(x_i).$5, and $z_i = g(x_i).$6 on FMNIST. Across 12 source-selection settings in the appendix, it obtained 20 of 36 best results overall, including 10 of 12 best weighted Kendall scores and 7 of 12 best Kendall scores. H-Score was reported as the strongest competitor in some source-selection tasks, particularly in Pearson correlation (Huang et al., 2021).

Layer selection is the setting in which TransRate is portrayed as especially strong. For SVHN $z_i = g(x_i).$7 CIFAR-100 with ResNet-20 layers, it achieved $z_i = g(x_i).$8, $z_i = g(x_i).$9, and $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$0. For Birdsnap $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$1 CIFAR-100 with ResNet-18 layers, it achieved $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$2, $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$3, and $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$4. Over 15 layer-selection experiments in the appendix, TransRate correctly predicted the full ranking in 9 cases, compared with 3 for the best competitor, LFC (Huang et al., 2021).

For architecture selection among ImageNet-pretrained models transferred to CIFAR-100, the reported performance was $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$5, $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$6, and $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$7. Over 8 model-selection tasks in the appendix, it achieved 16 of 24 best results in a 7-model setting and 18 of 24 best results in a 13-model setting. The paper also reports strong results for self-supervised ImageNet models transferred to CIFAR-100 and for molecular property prediction with self-supervised GNNs. At the same time, it does not claim universal dominance: in some source-selection tasks H-Score or LogME had better Pearson correlation, in one self-supervised model-selection task on Caltech-101 TransRate performed poorly relative to baselines, and in some architecture-selection tasks LEEP or NCE had slightly higher $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$8 even when TransRate retained better rank-based metrics (Huang et al., 2021).

6. Subsequent domain-specific reassessment and limits

A later benchmark on source-independent transferability estimation for surgical phase recognition reevaluated TransRate alongside LogME and H-Score on RAMIE and AutoLaparo, using extracted target-frame embeddings and phase labels without access to source training data (Singh et al., 22 Aug 2025). In that study, candidate models were scored on target subsets and aggregated by mean, minimum, or maximum. The paper explicitly characterized TransRate as an information-theoretic metric that “measur[es] the transferability as the mutual information between features of target examples extracted by a pre-trained model and their labels,” but did not restate the original TransRate coding-rate equations in its own methodology (Singh et al., 22 Aug 2025).

Within that surgical benchmark, TransRate performed poorly and often inverted the true ranking of models. On AutoLaparo, the reported configurations were: mean aggregation with $$\mathrm{Trf}_{T_s \rightarrow T_t}(g) := \mathbb{E}\big[\log p(y \mid z^*; g^*, w^*)\big],$$9 and $z^* = g^*(x)$0, minimum aggregation with $z^* = g^*(x)$1 and $z^* = g^*(x)$2, and maximum aggregation with $z^* = g^*(x)$3 and $z^* = g^*(x)$4. On RAMIE, the corresponding results were: mean aggregation with $z^* = g^*(x)$5 and $z^* = g^*(x)$6, minimum aggregation with $z^* = g^*(x)$7 and $z^* = g^*(x)$8, and maximum aggregation with $z^* = g^*(x)$9 and $g^*, w^*$0. The benchmark concluded that LogME, particularly with minimum per-subset aggregation, aligned most closely with fine-tuning accuracy, H-Score yielded only weak predictive power, and TransRate often inversed true model rankings (Singh et al., 22 Aug 2025).

These findings delimit the scope of the original claims rather than nullifying them. The original TransRate paper already noted several constraints: the method requires labeled target data, its principal theoretical result assumes a uniform class distribution, empirical estimation quality degrades as target sample size shrinks, cross-architecture comparability may require centering and scale normalization, and the regression extension relies on discretization into equal-frequency bins (Huang et al., 2021). The surgical benchmark adds a domain-specific caution: frame-wise transferability estimates can be misaligned with downstream pipelines whose final accuracy depends on temporal refinement via MS-TCN rather than frame embeddings alone (Singh et al., 22 Aug 2025). A plausible implication is that TransRate’s coding-rate surrogate is most reliable when the downstream task is well captured by static feature-label structure and less reliable when temporal dynamics, procedural continuity, or severe domain shift dominate the transfer problem.