STPL: Source-Free Domain Adaptation

• The paper introduces STPL which leverages stored batch normalization statistics as proxies for source distributions to align target features.
• It integrates mutual information maximization to maintain discriminative and balanced class predictions during adaptation.
• STPL demonstrates robust performance on benchmarks like USPS→MNIST by efficiently adapting without accessing any source samples.

Source-Free Domain Adaptation (STPL)

Source-Free Domain Adaptation (SFDA) refers to the class of unsupervised domain adaptation techniques that adapt a model, previously trained on a labeled source domain, to an unlabeled target domain without access to any source data during the adaptation process. STPL—Source-free Domain Adaptation via Distributional Alignment by Matching Batch Normalization Statistics (Ishii et al., 2021)—is a representative method in this category. STPL circumvents the inability to observe source samples by leveraging the stored batch normalization (BN) statistics within the pre-trained model as a proxy for the source feature distribution. Adaptation is driven by aligning the target feature statistics to these stored values, coupled with mutual information maximization to preserve discriminative and diverse representations. The framework is computationally efficient and empirically validated on standard benchmarks, yielding competitive results with other state-of-the-art DA approaches.

1. Problem Setting and Motivation

STPL addresses the SFDA task where adaptation from source to target proceeds in the complete absence of source data. The only inputs for adaptation are:

• A source pre-trained model $f_{\theta_s}$, with access to its parameters, including the fixed running means and variances stored in batch normalization layers.
• An unlabeled set of target domain samples $\mathcal{D}_t = \{ x_j^t \}$.

The core challenge is to transfer knowledge to the target domain while respecting privacy, regulatory, or logistical constraints that prohibit retaining source datasets. Traditional DA methods perform direct source–target distribution alignment (e.g., via adversarial or discrepancy-based losses) but are inapplicable here due to the lack of reference source data. The key insight of STPL is that BN statistics internal to the source model encode the first and second moments of the source feature distribution for each channel. These stored values serve as a surrogate for source-domain feature statistics, enabling indirect alignment.

2. Distributional Alignment via Batch Normalization Statistics

Within the STPL framework, the source-pretrained model is partitioned into an adaptable feature encoder $g_\theta$ and a frozen classifier $h$. The adaptation procedure operates as follows:

• During forward passes on target data, $g_\theta$ produces features $z=g_\theta(x^t)$.
• For each target mini-batch, empirical per-channel mean ($\mu_c$) and variance ($\sigma_c^2$) are computed using:

$$\mu_c = \frac{1}{B n_c} \sum_{i=1}^B \sum_{j=1}^{n_c} z_i^{(c)}[j], \quad \sigma_c^2 = \frac{1}{B n_c} \sum_{i=1}^B \sum_{j=1}^{n_c} (z_i^{(c)}[j] - \mu_c)^2$$

Here, $B$ is the mini-batch size and $\mathcal{D}_t = \{ x_j^t \}$0 is the number of spatial elements per channel.

• Alignment is enforced by minimizing the average Kullback–Leibler divergence between the channel-wise Gaussian approximations of the stored source statistics ($\mathcal{D}_t = \{ x_j^t \}$1) and those empirically computed on target features ($\mathcal{D}_t = \{ x_j^t \}$2):

$\mathcal{D}_t = \{ x_j^t \}$3

By Pinsker’s inequality and established domain adaptation risk bounds, minimizing this KL divergence reduces target error by aligning the first two moments (mean and variance) of the feature distribution.

3. Mutual Information Maximization

Marginal distribution alignment in feature space does not guarantee that target features preserve class-discriminativeness. Therefore, STPL incorporates an information maximization (IM) objective, inspired by SHOT: $\mathcal{D}_t = \{ x_j^t \}$4 with softmax predictions $\mathcal{D}_t = \{ x_j^t \}$5. The negative entropy of the batch mean enforces diversity (i.e., spreading predictions over classes), whereas the average entropy term drives confidence (i.e., low-entropy, near one-hot vectors).

The overall objective for adaptation is: $\mathcal{D}_t = \{ x_j^t \}$6 where $\mathcal{D}_t = \{ x_j^t \}$7 balances consistency with the stored source statistics versus the production of sharp, class-balanced predictions. In reported experiments, $\mathcal{D}_t = \{ x_j^t \}$8 is typically used; performance is stable for a broad range $\mathcal{D}_t = \{ x_j^t \}$9.

4. Optimization Procedure

Adaptation is performed as follows:

• The classifier $g_\theta$0 and its BN running statistics are frozen.
• Only the encoder parameters $g_\theta$1 are optimized.
• Adam (or similar optimizer) is used, with learning rate, batch size, and BN momentum typically inherited from the source pre-training regime. A schedule of 30,000 iterations with a batch size of 64 is standard.

Frozen BN statistics in the classifier ensure that the target encoder’s output distribution stays gravitationally tethered to the source statistics. This provides stability and circumvents drift that otherwise could arise in the absence of source references.

5. Experimental Evaluation and Analysis

STPL is evaluated on standard benchmarks—USPS⟺MNIST, SVHN→MNIST (both with LeNet variants), and Office-31 (A, D, W with ResNet-50). Results are summarized as follows (average over five runs, with comparative SOTA baselines):

Dataset	A→D	A→W	USPS→MNIST	SVHN→MNIST
SHOT	94.0	90.1	—	—
Model Adapt.	92.7	93.7	—	—
STPL	89.0	91.7	99.1	99.1

Notably, STPL achieves 99.1% on both digit adaptation tasks (second best reported), and performance is consistently robust even when the target set is small (e.g., ≥98% on SVHN→MNIST with only 600 target samples). Office-31 results position STPL as highly competitive despite not utilizing source data.

Ablation studies reveal:

• Performance is insensitive to a wide range of $g_\theta$2 values.
• The test accuracy improves monotonically during adaptation, indicating stable convergence.
• Matching BN statistics leverages information effectively even in low-data regimes.

6. Strengths, Limitations, and Extensions

Strengths:

• STPL does not require any source samples after pre-training; only BN statistics and a frozen classifier suffice.
• The approach is computationally efficient, avoiding adversarial training and extra generative models.
• Theoretically grounded by the domain adaptation risk theory and empirical moment matching.
• Demonstrates remarkable robustness to both hyperparameters and reduced target data.

Limitations:

• The per-channel Gaussian assumption underlying BN statistics alignment can miss higher-order or multi-modal feature structure.
• Only first and second moments are matched; higher moments (skewness, kurtosis), full covariance, or class-conditional alignment are not modeled.
• STPL may not directly apply where other normalization layers (e.g., LayerNorm) replace BN, or where BN statistics are not preserved.

Extensions/Questions:

• Aligning BN statistics at multiple network layers may address deeper distributional mismatches.
• Adaptive channel weighting could further reflect channel semantic importance for the task.
• Weak pseudo-label refinement or confidence-threshold-based sample selection could complement the existing objectives.
• Generalization to different network normalization topologies or to different tasks (e.g., object detection, segmentation) remains an open direction.

7. Position within the SFDA Landscape

STPL exemplifies the family of SFDA approaches that utilize internal network statistics as a substitute for inaccessible source data. It contrasts with methods based on pseudo-label refinement, prototype clustering, distribution estimation via target data, or contrastive clustering in the feature space. STPL’s reliance on parameter-level “memory” (i.e., frozen batch-statistics) makes it uniquely simple and orthogonal in the method space. The approach is particularly relevant when privacy or storage constraints preclude even indirect sharing of source-domain feature statistics. Its simplicity, efficiency, and empirical validation anchor it as a foundational reference point in source-free adaptation research (Ishii et al., 2021).