---
title: 'LEAD: Learning Decomposition for SF-UniDA'
url: https://www.emergentmind.com/topics/lead
type: topic
---

# LEAD: Learning Decomposition for SF-UniDA

LEAD, short for Learning Decomposition, is a method for source-free universal domain adaptation that addresses the central identification problem in Universal Domain Adaptation under simultaneous covariate shift and label shift: determining whether an unlabeled target sample belongs to shared source-known categories or to target-private unknown categories when source data are unavailable during adaptation [2403.03421]. Its defining idea is to decompose target features into source-known and source-unknown components using the source classifier’s weight geometry, then replace global thresholding or iterative clustering with adaptive, instance-level decision boundaries for pseudo-labeling [2403.03421].

## 1. Problem setting and adaptation objective

Universal Domain Adaptation relaxes the closed-set assumption by allowing both covariate shift and label shift. In the formulation used by LEAD, the source label set is $\mathcal{Y}_s$ and the target label set is $\mathcal{Y}_t$, with shared labels $\mathcal{Y}_k=\mathcal{Y}_s\cap\mathcal{Y}_t$ and target-private unknown labels $\mathcal{Y}_u=\mathcal{Y}_t\setminus\mathcal{Y}_s$ [2403.03421]. The distribution shift has two components: $P_s(x)\neq P_t(x)$ and $\mathcal{Y}_s\neq \mathcal{Y}_t$, with the target label discrepancy unknown beforehand in the UniDA setting [2403.03421].

Source-free UniDA adds an additional deployment constraint: no source data are accessible during adaptation. LEAD assumes only a source-trained model $f^s_\theta=h^s_\theta\circ g^s_\theta$ is provided, together with unlabeled target samples $\{x_i^t\}_{i=1}^{N_t}$ [2403.03421]. As in prior source-free adaptation, the classifier is frozen, $h^t_\theta=h^s_\theta$, while the target feature extractor $g^t_\theta$ is updated on the target domain [2403.03421].

Within this setting, LEAD pursues two coupled objectives. The first is unknown detection: identifying target samples in $\mathcal{Y}_u$ and avoiding assignment of known-class labels. The second is target pseudo-labeling: assigning robust pseudo-labels to target samples likely to belong to $\mathcal{Y}_k$, so that those labels can supervise adaptation of the target encoder [2403.03421]. The method is motivated by the observation that the key bottleneck in SF-UniDA is not merely classification under shift, but discrimination between covariate-shifted known samples and genuinely target-private unknown samples.

## 2. Orthogonal feature decomposition and unknownness estimation

LEAD constructs an orthogonal decomposition of each normalized target feature into components aligned with source-known and source-unknown subspaces [2403.03421]. Let $f\in\mathbb{R}^d$ denote a unit-normalized target feature and let $\mathcal{S}_k\subset\mathbb{R}^d$ be a learned source-known subspace. The corresponding decomposition is
\[
f_k=P_k f,\qquad f_u=(I-P_k)f,
\]
where $P_k=U_kU_k^\top$ for an orthonormal basis $U_k\in\mathbb{R}^{d\times r}$ of $\mathcal{S}_k$ [2403.03421]. The intended interpretation is that known target instances should remain more aligned with the source-known subspace, whereas target-private unknown instances should exhibit larger projections onto the orthogonal complement.

In LEAD, the source-known subspace is derived from the source classifier weights $W_{\mathrm{cls}}\in\mathbb{R}^{C\times d}$. Because those class weight vectors are not necessarily orthonormal, the method performs singular value decomposition,
\[
W_{\mathrm{cls}}=U\Sigma V^\top,
\]
and uses the first $C$ columns of $V$, $\{v_n\}_{n=1}^{C}$, as the known basis, while the remaining columns $\{v_n\}_{n=C+1}^{d}$ define the unknown basis [2403.03421]. For a normalized target feature
\[
z_i^t=\frac{g_\theta^t(x_i^t)}{\|g_\theta^t(x_i^t)\|_2},
\]
LEAD writes
\[
z_i^t=z_{i,\mathrm{knw}}^t+z_{i,\mathrm{unk}}^t,
\]
with $z_{i,\mathrm{knw}}^t\in\mathrm{span}\{v_n\}_{n=1}^C$, $z_{i,\mathrm{unk}}^t\in\mathrm{span}\{v_n\}_{n=C+1}^d$, and
\[
\|z_{i,\mathrm{knw}}^t\|_2^2+\|z_{i,\mathrm{unk}}^t\|_2^2=1
\]
[2403.03421].

The unknownness score is defined as
\[
u(z_i^t)=\|z_{i,\mathrm{unk}}^t\|_2.
\]
LEAD models the distribution of $\{u(z_i^t)\}$ with a two-component Gaussian Mixture Model, exploiting a bimodal pattern with a low-mean common component and a high-mean private component, denoted by $\mu_{\mathrm{com}}<\mu_{\mathrm{pri}}$ [2403.03421]. This gives the method a one-dimensional, decomposition-based indicator of unknownness rather than a confidence- or entropy-based heuristic. The reported ablations further show that using entropy as the decomposition indicator under the same framework is inferior to $\|z^t_{\mathrm{unk}}\|_2$ [2403.03421].

## 3. Instance-level decision boundaries and pseudo-label assignment

A central claim of LEAD is that global thresholds are sub-optimal under UniDA because covariate shift is heterogeneous across classes and samples [2403.03421]. To address this, the method constructs instance-level decision boundaries that combine distances to target prototypes and source anchors.

Target prototypes $\{c_c^t\}$ are estimated from top-$K$ high-confidence target instances for each class, while source anchors $\{c_c^s\}$ are taken directly from $W_{\mathrm{cls}}$ [2403.03421]. Using cosine distance
\[
d(a,b)=1-\cos(a,b),
\]
LEAD defines two commonness scores for sample $i$ and class $c$:
\[
\epsilon_{i,c}^t = 1-\exp(d(z_i^t,c_c^t)-1),
\]
\[
\epsilon_{i,c}^s = \exp(-d(z_i^t,c_c^s)),
\]
clips both to $[0,1]$, and fuses them through the geometric mean
\[
\epsilon_{i,c}=\sqrt{\epsilon_{i,c}^t\cdot \epsilon_{i,c}^s}
\]
[2403.03421]. Under equal distances, $\epsilon_{i,c}^s\geq \epsilon_{i,c}^t$, which encodes a bias toward source-anchor proximity as evidence of knownness and treats target-prototype proximity alone as less decisive because target-private contamination is possible [2403.03421].

LEAD then estimates a class-wise expectation $\mu_c$ of $\|z_{i,\mathrm{unk}}^t\|_2$ using the same top-$K$ sampling employed for prototype computation. With $\mu_{\mathrm{pri}}$ taken from the GMM private component, it defines the per-instance boundary
\[
\rho_{i,c}=\mu_c+\epsilon_{i,c}\cdot(\mu_{\mathrm{pri}}-\mu_c)
\]
[2403.03421]. This boundary varies across both classes and samples, rather than being fixed globally.

Pseudo-label assignment proceeds by first choosing
\[
\kappa=\arg\max_c \epsilon_{i,c}.
\]
If
\[
\|z_{i,\mathrm{unk}}^t\|_2 \ge \rho_{i,\kappa},
\]
the sample is classified as unknown; otherwise it is assigned the known pseudo-label
\[
\hat{y}_i^t=\mathrm{1\mbox{-}hot}(\kappa)
\]
[2403.03421]. This direct computation of $\rho_{i,c}$ removes the need for iterative clustering to derive pseudo-labeling thresholds.

A plausible implication is that LEAD turns unknown detection from a global calibration problem into a local compatibility test between decomposition magnitude and class-conditioned commonness. That interpretation follows from the structure of $\rho_{i,c}$, but the concrete mechanism remains the boundary definition above.

## 4. Training objective and source-free optimization procedure

LEAD optimizes the target encoder with three losses: pseudo-label learning, feature decomposition regularization, and feature consensus regularization [2403.03421]. The pseudo-label learning term uses certainty weighting based on the sample’s distance from the adaptive boundary. With predicted softmax probabilities $\delta_c(f_\theta^t(x_i^t))$, the certainty weight is defined through a Student’s $t$-based form,
\[
\tau_i^t \propto 1-\left(1+\frac{(\rho_{i,\kappa}-\|z_{i,\mathrm{unk}}^t\|_2)^2}{\alpha}\right)^{-(\alpha+1)/2},
\]
with $\alpha=1\mathrm{e}{-4}$ [2403.03421]. The corresponding cross-entropy term is
\[
\mathcal{L}_{ce}
=
-\frac{1}{N}\sum_i \tau_i^t \sum_{c=1}^{C}\hat{y}_{i,c}^t\log \delta_c(f_\theta^t(x_i^t)).
\]
Unknown-labeled instances are not mapped to an extra $(C+1)$-th class; instead they are assigned a uniform target distribution to avoid biasing known-class decision boundaries [2403.03421].

The feature decomposition regularizer explicitly encourages separation between common and private samples. LEAD defines
\[
p_i
=
\frac{\exp(\|z_{i,\mathrm{unk}}^t\|_2)}
{\exp(\|z_{i,\mathrm{unk}}^t\|_2)+\exp(\|z_{i,\mathrm{knw}}^t\|_2)},
\]
and then uses
\[
\mathcal{L}_{reg}
=
-\frac{1}{N}\sum_i \tau_i^t
\left[
\bar{y}_i^t\log p_i + (1-\bar{y}_i^t)\log(1-p_i)
\right],
\]
where $\bar{y}_i^t\in\{0,1\}$ indicates private versus common pseudo-label assignment [2403.03421]. This term pushes private samples toward larger unknown projections and common samples toward larger known projections.

The feature consensus regularizer promotes neighborhood consistency in feature space. If $L_i^t$ denotes the nearest-neighbor set for sample $i$ constructed by cosine similarity, and empirically $|L_i^t|=4$, then
\[
l_{i,c}^t
=
\frac{1}{|L_i^t|}
\sum_{x_j^t\in L_i^t}
\delta_c(f_\theta^t(x_j^t)),
\]
and
\[
\mathcal{L}_{con}
=
-\frac{1}{N}\sum_i\sum_c l_{i,c}^t \log \delta_c(f_\theta^t(x_i^t))
\]
[2403.03421]. The full objective is
\[
\mathcal{L}=\lambda \mathcal{L}_{ce}+\mathcal{L}_{reg}+\mathcal{L}_{con},
\]
with $\lambda>0$ controlling the trade-off [2403.03421].

The adaptation procedure begins with $g_\theta^t\leftarrow g_\theta^s$ and frozen $h_\theta^t=h_\theta^s$. It then computes the SVD of $W_{\mathrm{cls}}$, decomposes each target feature into known and unknown components, fits the two-component GMM to $\{\|z_{i,\mathrm{unk}}^t\|_2\}$, constructs target prototypes with top-$K$ sampling where $K=N_t/\hat{C}_t$ and $\hat{C}_t$ is estimated via Silhouette-based selection, computes $\epsilon_{i,c}$ and $\rho_{i,c}$, assigns pseudo-labels, and updates $g_\theta^t$ by SGD [2403.03421]. As features evolve, decomposition-based quantities and boundaries are periodically refreshed.

The method’s stabilization strategy consists of certainty weighting $\tau_i^t$, consensus regularization, and temperature-free softmax supervision; no teacher EMA or temperature scaling is required [2403.03421].

## 5. Empirical behavior, benchmarks, and computational profile

LEAD is evaluated on OPDA, OSDA, and PDA settings across Office-31, Office-Home, VisDA, and DomainNet, with H-score used for OSDA and OPDA:
\[
H=\frac{2\cdot Acc_{\mathrm{known}}\cdot Acc_{\mathrm{unknown}}}
{Acc_{\mathrm{known}}+Acc_{\mathrm{unknown}}}
\]
[2403.03421]. Implementation uses a single RTX-3090, SGD with momentum $0.9$, batch size $64$, learning rates $1\mathrm{e}{-3}$ for Office-31 and Office-Home and $1\mathrm{e}{-4}$ for VisDA and DomainNet, with $\lambda$ set to $0.3$ on Office-31, $1.0$ on VisDA, and $2.0$ on Office-Home and DomainNet. Inference uses normalized entropy threshold $\omega=0.55$ [2403.03421].

The reported headline OPDA results are as follows.

| Scenario | Result |
|---|---|
| VisDA OPDA | LEAD achieves $H=76.6\%$, surpassing GLC $(73.1\%)$ by $+3.5\%$ |
| Office-Home OPDA | LEAD $75.0\%$ vs UMAD $70.1\%$; LEAD+UMAD $78.0\%$ |
| Office-31 OPDA | LEAD $87.8\%$ average H-score; LEAD+UMAD $92.0\%$ |
| DomainNet OPDA | LEAD $50.8\%$ average H-score |

The OSDA results are Office-Home $67.2\%$, Office-31 $90.3\%$, and VisDA $74.2\%$, with gains over UMAD of $+1.2\%$, $+2.5\%$, and $+3.4\%$, respectively [2403.03421]. In PDA, the reported accuracies are Office-Home $73.8\%$, Office-31 $95.5\%$, and VisDA $75.3\%$, with gains over UMAD of $+8.7\%$, $+4.1\%$, and $+9.6\%$ [2403.03421].

The paper attributes these results to three properties: decomposition reveals structure unavailable to entropy or confidence alone, instance-level boundaries adapt to per-class and per-sample variability, and the method avoids high-dimensional clustering instability while achieving large runtime savings [2403.03421]. The efficiency claims are concrete. Projection and distance computations are $O(d)$ per instance, while the SVD is $O(\min\{Cd^2,dC^2\})$ once. LEAD avoids iterative $K$-means inner loops for boundary derivation, relying instead on a one-dimensional GMM over $u_i$ [2403.03421].

On VisDA, boundary derivation time is reduced from $0.46\,\mathrm{s}$ to $0.11\,\mathrm{s}$, approximately a $76\%$ reduction. On DomainNet, the reported comparison is $897.65\,\mathrm{s}$ to $0.29\,\mathrm{s}$ [2403.03421]. The abstract also highlights that, in the VisDA OPDA scenario, LEAD outperforms GLC by $3.5\%$ overall H-score and reduces $75\%$ time to derive pseudo-labeling decision boundaries [2403.03421]. The slight difference between “approximately $76\%$” and “$75\%$” reflects the two reported phrasings in the source material.

## 6. Ablations, robustness, complementarity, and limitations

The ablation results indicate that all three loss components are complementary. In OPDA on VisDA, performance rises from $61.5\%$ with $\mathcal{L}_{ce}$ alone to $75.7\%$ with $\mathcal{L}_{ce}+\mathcal{L}_{con}$, and to $76.6\%$ with all three losses [2403.03421]. Instance-level boundaries also clearly outperform global thresholds: on Office-Home OPDA, the comparison is $75.0\%$ versus $63.1\%$ [2403.03421]. The method is reported to be stable around the chosen defaults for $\lambda$, stable near $|L_i^t|=4$ for nearest-neighbor size, and moderately robust to the inference threshold $\omega$ [2403.03421]. It also maintains stable H-scores under varying numbers of unknown classes [2403.03421].

LEAD is described as complementary to most existing methods. The paper gives a blending form
\[
\mathcal{L}_{overall}=\gamma \mathcal{L}_{LEAD}+(1-\gamma)\mathcal{L}_{baseline},
\]
with $\gamma\approx 0.7$ [2403.03421]. Gains are substantial with UMAD and moderate with GLC, though occasional small trade-offs are noted, such as PDA on VisDA at $-0.7\%$, which the paper suggests may reflect gradient conflicts; joint tuning of $\gamma$ and $\lambda$ is suggested as mitigation [2403.03421].

The limitations are stated in terms of assumptions and failure modes. The source-known subspace must capture discriminative directions of shared classes; if $W_{\mathrm{cls}}$ is poorly trained or the domain shift is extreme, the projections may become unreliable [2403.03421]. If the bimodality of $\|z_{\mathrm{unk}}^t\|_2$ weakens, for example under severe noise or minimal overlap in $\mathcal{S}_k$ across domains, GMM separation may be ambiguous [2403.03421]. Target prototypes estimated from top-$K$ target samples can also be contaminated when the classifier is highly uncertain at early stages, although periodic refresh and certainty weighting are reported to mitigate this issue [2403.03421].

The practical guidance follows directly from these observations. The recommended hyperparameter range for $\lambda$ is dataset-dependent within $[0.3,2.0]$, $\alpha=1\mathrm{e}{-4}$ is described as robust for certainty weighting, $|L_i^t|\approx 4$ is used for consensus, and $\omega\approx 0.55$ gives good H-scores, with slight adjustment in the range $0.55$–$0.60$ possible per dataset [2403.03421]. The method is positioned for settings such as safety-critical recognition under privacy constraints, large-scale deployments where source data retention is infeasible, and scenarios requiring efficient adaptation without iterative clustering [2403.03421].

Taken together, LEAD defines SF-UniDA adaptation as a decomposition-and-boundary problem: it extracts an unknown-aware feature component from the source classifier geometry and uses that component in a per-instance decision rule coupled to pseudo-label supervision [2403.03421]. This suggests a broader methodological pattern in source-free adaptation: when access to source data is removed, the classifier itself becomes a structural prior from which usable target-side uncertainty geometry can be recovered.

Source: https://www.emergentmind.com/topics/lead