Hardness Adjusted Transfer (HAT) Score

• HAT Score is a hardness-aware metric that quantifies transferability by adjusting the source log-likelihood with the target conditional entropy.
• It is applied in supervised classification and multilingual evaluations to isolate true transfer effects from improvements in source task performance.
• Empirical studies on diverse datasets demonstrate that lower conditional entropy correlates with higher expected transferability.

Searching arXiv for the cited papers and related HAT terminology. Hardness Adjusted Transfer (HAT) Score denotes a hardness-aware measure of transferability, but its precise meaning depends on context. In supervised classification task transfer, the paper "Transferability and Hardness of Supervised Classification Tasks" does not use the name explicitly, yet it naturally yields the quantity

$$\mathrm{HAT}(T^Z \rightarrow T^Y)=l_Z(w_Z,h_Z)-H(Y\mid Z),$$

an information-theoretic lower bound on transferred log-likelihood that combines source-task easiness with cross-task label dependence (Tran et al., 2019). In multilingual evaluation, the term is explicit: "Are Multilingual Models Actually Improving? Isolating True Cross-Lingual Transfer" defines HAT Score as

$$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$

and, under a linear estimator, as the slope of target pass-rate on source pass-rate, thereby isolating transfer strength from general source-language improvement (Bajpai et al., 20 Jun 2026). Across these usages, the unifying idea is that transfer should be evaluated relative to hardness rather than by raw downstream accuracy alone.

1. Terminological scope and historical placement

The expression "Hardness Adjusted Transfer (HAT) Score" is not uniform across the literature. In the 2019 task-transfer work, the paper does not use the name “Hardness Adjusted Transfer (HAT) score” explicitly; the designation is a natural mapping onto its lower bound

$l_Z(w_Z,h_Z)-H(Y\mid Z),$

where the first term captures source-task optimal log-likelihood and the second captures residual uncertainty of target labels given source labels (Tran et al., 2019). By contrast, the 2026 multilingual work introduces HAT Score as a named metric intended to measure true cross-lingual transfer while factoring out improvements that merely reflect better source-language accuracy (Bajpai et al., 20 Jun 2026).

A related but distinct development is HASTE, introduced in "Towards Estimating Transferability using Hard Subsets." HASTE is presented as a general mechanism that takes any transferability metric and evaluates it on a harder subset of target data, identified either by class-agnostic representation mismatch or by class-specific atypicality in source-feature space (Menta et al., 2023). This suggests a broader family of hardness-adjusted transfer estimators in which hardness enters either through label statistics, as in the 2019 formulation, or through sample selection, as in HASTE.

A persistent source of confusion is acronym overload. In speech recognition, "HAT" can mean Hybrid Autoregressive Transducer; that work explicitly does not define a Hardness Adjusted Transfer score, and “HAT” there always refers to the model architecture rather than a metric (Moriya et al., 2024).

2. Information-theoretic formulation in supervised classification

In the supervised classification setting of (Tran et al., 2019), transferability is defined operationally for a source task $T^Z=(X,Z)$ and a target task $T^Y=(X,Y)$ that share the same inputs $X$ but differ in labels. A source model $M=(w,h)$ consists of a representation function $w:\mathcal X\to\mathbb R^D$ and a classifier $h:\mathbb R^D\to\mathcal P(\mathcal Z)$. After training $(w_Z,h_Z)$ on $$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$0 with cross-entropy, transfer to $$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$1 fixes $$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$2 and trains a new classifier $$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$3. True transferability is the expected target accuracy

$$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$4

while the paper uses training log-likelihood on $$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$5 as a proxy under a non-overfitting assumption:

$$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$6

Task hardness is defined by the optimal achievable cross-entropy under a chosen model class:

$$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$7

Larger loss and lower log-likelihood therefore mean a harder task.

The central modeling move is solution-agnostic. Tasks are treated purely as label sequences on a shared input set, and the only statistics used are empirical label distributions and co-occurrences. Let $$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$8 denote the empirical joint distribution of source and target labels:

$$2\int_0^1 \mathbb{E}_{D,\ell}[T_\ell\mid S=s,M]\,ds,$$9

with empirical marginal $l_Z(w_Z,h_Z)-H(Y\mid Z),$0. Conditional entropy is then

$l_Z(w_Z,h_Z)-H(Y\mid Z),$1

The interpretation is direct: lower $l_Z(w_Z,h_Z)-H(Y\mid Z),$2 means the source labels are more predictive of the target labels, and therefore the source task should be more transferable.

3. The lower bound and the “hardness-adjusted” term

The theoretical core of (Tran et al., 2019) is Theorem 1, which shows that under the described transfer procedure there exists a classifier $l_Z(w_Z,h_Z)-H(Y\mid Z),$3 such that

$l_Z(w_Z,h_Z)-H(Y\mid Z),$4

Equivalently,

$l_Z(w_Z,h_Z)-H(Y\mid Z),$5

This is the quantity naturally interpreted as a Hardness Adjusted Transfer score:

$l_Z(w_Z,h_Z)-H(Y\mid Z),$6

The adjustment is explicit. The term $l_Z(w_Z,h_Z)-H(Y\mid Z),$7 is the negative of source-task hardness, while $l_Z(w_Z,h_Z)-H(Y\mid Z),$8 is the residual uncertainty about the target after observing the source label. Rewriting with hardness gives

$l_Z(w_Z,h_Z)-H(Y\mid Z),$9

A harder source task reduces the lower bound; stronger label dependence across tasks increases it.

The construction is asymmetric because $T^Z=(X,Z)$0 in general. Transfer from $T^Z=(X,Z)$1 to $T^Z=(X,Z)$2 need not mirror transfer from $T^Z=(X,Z)$3 to $T^Z=(X,Z)$4. For a fixed source task, $T^Z=(X,Z)$5 is constant, so ranking target tasks reduces to ranking them by conditional entropy; lower $T^Z=(X,Z)$6 implies higher expected transferability.

The same theorem yields a hardness proxy without training a model for each task. If $T^Z=(X,Z)$7 is a trivial constant-label task, then $T^Z=(X,Z)$8, and

$T^Z=(X,Z)$9

Since $T^Y=(X,Y)$0 is constant, $T^Y=(X,Y)$1 reduces to the entropy of the label distribution of $T^Y=(X,Y)$2. This produces a label-only estimate of task hardness.

4. Estimation and empirical behavior

The label-statistics HAT formulation is estimated directly from paired labels on a shared dataset. For each task pair, the empirical procedure is to compute joint counts $T^Y=(X,Y)$3, form $T^Y=(X,Y)$4, derive $T^Y=(X,Y)$5, and evaluate

$T^Y=(X,Y)$6

The appendix gives a two-loop algorithm: one pass to compute $T^Y=(X,Y)$7, and a second pass to accumulate the sample-wise log-probability term. No special smoothing or bias correction is used; the approach is reported to work well at the data scales considered (Tran et al., 2019).

Empirical validation was performed on three multi-attribute datasets: CelebA with approximately 202k face images and 40 binary attributes plus identity labels, Animals with Attributes 2 with approximately 37k images, 50 animal classes, and 85 binary attributes, and Caltech-UCSD Birds 200 with approximately 11.8k images and 312 attributes after confidence filtering. Across these 437 classification tasks, conditional entropy computed from labels alone showed strong positive correlation with empirical transfer test error, while $T^Y=(X,Y)$8 showed strong positive correlation with dedicated-network test error (Tran et al., 2019).

A case study transferred a face-recognition model to CelebA attributes. The source was a ResNet-101 trained on more than 10,000 identities using MS-Celeb-1M and VGGFace2 with cosine margin loss; target tasks were the 40 CelebA attributes. The empirical alignment between low conditional entropy and high transfer was particularly clear for identity-linked attributes.

Attribute	CE $T^Y=(X,Y)$9	Transferred vs dedicated accuracy
Male	0.017	0.992 vs 0.985
Bald	0.026	0.991 vs 0.990
Smiling	0.521	0.909 vs 0.933
Mouth Slightly Open	0.551	0.901 vs 0.943
Attractive	0.361	0.820 vs 0.823

These examples show the intended semantics of the score. Attributes with lowest CE, such as Male, Bald, Gray Hair, Mustache, and Double Chin, were the most transferable from identity. Attributes with highest CE, such as High Cheekbones, Smiling, Mouth Slightly Open, and Attractive, were the least transferable. The paper reports that the overall average difference between transferred attributes and the best published systems is only approximately $X$0 despite using a very simple lSVM head (Tran et al., 2019).

5. Explicit HAT Score in multilingual evaluation

In multilingual evaluation, HAT Score is defined differently and explicitly. The motivating claim is that source accuracy and average target accuracy often lie on a strong linear trend with $X$1, so raw target accuracy conflates genuine cross-lingual transfer with general improvements in the source language (Bajpai et al., 20 Jun 2026). The metric therefore conditions on source difficulty, described as “hardness” at the example level through the source pass-rate $X$2.

The transfer profile of a model is

$X$3

with $X$4 on the horizontal axis and expected target pass-rate on the vertical axis. Perfect transfer corresponds to the line $X$5. HAT Score is then defined as the area under this transfer profile under a uniform hardness distribution:

$X$6

The factor $X$7 is a normalization factor to keep HAT in $X$8. In practice, the conditional expectation is approximated by a linear regressor through the origin, $X$9, where $M=(w,h)$0 and $M=(w,h)$1 are instance-level source and averaged-target pass-rates estimated from $M=(w,h)$2 passes. Under this estimator,

$M=(w,h)$3

This formulation is designed to be invariant to shifts in the source-accuracy distribution $M=(w,h)$4. Average target accuracy, XLT gap, and ECLeKTic Transfer Score are criticized because they aggregate over the model-dependent source distribution and can therefore change even when the conditional transfer profile does not (Bajpai et al., 20 Jun 2026).

The reported empirical findings are threefold. First, transfer in small models is not broken: small models have HAT scores in the 50–80 range across benchmarks, suggesting that low raw target accuracy mainly reflects weak source capability. Second, progress with model size is slower than expected: for example, on MMLU-ProX-Lite, HAT rises from 58.4 for Gemma-3-1B to 77.2 for Gemma-3-27B. Third, there has been clear progress over time: median HAT on MGSMv2 rises from 0.90 to 0.97, and on MMLU-ProX-Lite from 0.75 to 0.93, whereas the pattern is weaker on ECLeKTic (Bajpai et al., 20 Jun 2026).

6. Related metrics, misconceptions, and limitations

Several neighboring metrics clarify what HAT is and is not. HASTE treats hardness as a property of target samples rather than of label mappings. It identifies hard subsets either by class-agnostic multi-layer cosine dissimilarity to the source domain or by class-specific Mahalanobis distance from the class mean, then computes a base transferability metric such as LEEP, NCE, or GBC on the selected subset. The paper states that HASTE modified metrics are consistently better or on par with the state of the art transferability metrics, especially under substantial source-target mismatch (Menta et al., 2023).

Within broader transferability-metric research, H-score, NCE, and LEEP occupy related but non-identical positions. H-score is covariance-based and target-aware; NCE and LEEP are label-based and use conditional entropy or soft source predictions. The analysis in "Newer is not always better" shows that NCE and LEEP are sensitive to the number of classes and class imbalance in target-task selection, and recommends normalization by target label entropy together with evaluation against relative accuracy in settings with varying label cardinality (Ibrahim et al., 2021). This suggests caution when interpreting hardness-adjusted transfer estimates across heterogeneous tasks.

The principal assumptions of the 2019 label-statistics formulation are stringent. All tasks must share the same input domain and identical instances; labels are discrete; empirical co-occurrence statistics are available; and the link from training log-likelihood to test accuracy depends on train and test being drawn from the same distribution. The multilingual HAT formulation has different limitations: it requires multiple passes per prompt, depends on a linear approximation to the transfer profile, and is not immune to contamination or benchmark saturation (Tran et al., 2019, Bajpai et al., 20 Jun 2026).

A final misconception concerns the acronym itself. In ASR, HAT refers to Hybrid Autoregressive Transducer, and that paper explicitly states that it does not define any metric called “Hardness Adjusted Transfer (HAT) Score” (Moriya et al., 2024). The shared abbreviation therefore names an information-theoretic lower bound in one literature, a hardness-conditioned multilingual transfer slope in another, and an ASR architecture in a third.