Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation (2110.04202v3)

Abstract: Domain adaptation (DA) aims to alleviate the domain shift between source domain and target domain. Most DA methods require access to the source data, but often that is not possible (e.g. due to data privacy or intellectual property). In this paper, we address the challenging source-free domain adaptation (SFDA) problem, where the source pretrained model is adapted to the target domain in the absence of source data. Our method is based on the observation that target data, which might no longer align with the source domain classifier, still forms clear clusters. We capture this intrinsic structure by defining local affinity of the target data, and encourage label consistency among data with high local affinity. We observe that higher affinity should be assigned to reciprocal neighbors, and propose a self regularization loss to decrease the negative impact of noisy neighbors. Furthermore, to aggregate information with more context, we consider expanded neighborhoods with small affinity values. In the experimental results we verify that the inherent structure of the target features is an important source of information for domain adaptation. We demonstrate that this local structure can be efficiently captured by considering the local neighbors, the reciprocal neighbors, and the expanded neighborhood. Finally, we achieve state-of-the-art performance on several 2D image and 3D point cloud recognition datasets. Code is available in https://github.com/Albert0147/SFDA_neighbors.

Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation

The paper "Exploiting the Intrinsic Neighborhood Structure for Source-free Domain Adaptation" addresses the prominent challenge of domain adaptation (DA) when the source data is not available during the adaptation phase. Traditional DA methods require access to source data, which may not always be feasible due to privacy concerns or ownership issues. This work focuses on source-free domain adaptation (SFDA), where only a pretrained model from the source domain is accessible for adapting to the target domain.

One of the key insights of this paper is the observation that, despite a domain shift, the target domain data still tends to form discernible clusters in feature space. The authors propose leveraging the intrinsic neighborhood structure of target data to enhance domain adaptation. Their approach, termed Neighborhood Reciprocity Clustering (NRC), exploits local affinities among data points within the target domain to encourage label consistency.

Methodology

The methodology revolves around the concept of local affinity and neighborhood relationships among data points in the target domain:

1. Reciprocal Nearest Neighbors (RNN): The authors distinguish between reciprocal nearest neighbors, where two data points consider each other as near neighbors, and non-reciprocal ones, emphasizing the importance of reciprocal relationships for reliable adaptation. Reciprocal neighbors are given higher affinity values as they are more likely to have the same label, thereby serving as trustworthy signals for adaptation.
2. Affinity-based Label Consistency: By establishing affinity values based on neighborhood relationships, the method adjusts the influence of neighbor data points on the adaptation process. This weighted approach ensures that the model is more heavily influenced by semantically similar points.
3. Expanded Neighborhoods: NRC considers not only direct neighbors but also expanded neighborhoods, which include neighbors of neighbors. This strategy is designed to capture a broader context, increasing the robustness of the adaptation without introducing significant noise from dissimilar classes.
4. Self-Regularization: To mitigate the potential influence of noisy neighbors, a self-regularization term is introduced, which reinforces the model's current predictions when they are uncertain. This term acts as a counterbalance to potentially erroneous neighbor influences.
5. Diversity Loss: The method adopts a diversity loss to encourage predictions to be evenly distributed across classes, guarding against collapse onto a subset of classes.

Results and Implications

The effectiveness of the proposed method is demonstrated through extensive experiments on both 2D image datasets (Office-31, Office-Home, VisDA) and a 3D point cloud dataset (PointDA-10). NRC achieves state-of-the-art performance in SFDA settings, often surpassing methods that have access to source data during adaptation. For instance, in the challenging VisDA dataset, NRC substantially outperforms both the prior SFDA approach SHOT and other source-present domain adaptation methods.

The practical implications of this work are significant, particularly in scenarios where data privacy and ownership restrict source data accessibility. By harnessing the inherent structure in the target data, NRC offers a viable solution for adapting models without source data, which is crucial for deploying machine learning systems in privacy-sensitive and proprietary environments.

Speculations and Future Directions

The paper opens several avenues for future research in SFDA:

• Advanced Affinity Estimation: Enhancing the method of determining affinity values based on more sophisticated graph-theoretical approaches could further improve adaptation performance by more accurately capturing data structure.
• Application to Dynamic Domains: Exploring the robustness and applicability of the NRC approach in continuously evolving environments where both domains may shift over time could be an intriguing direction.
• Cross-modal and Cross-lingual Adaptation: Extending the concept of neighborhood affinity to more complex adaptation tasks involving different modalities or languages represents a promising challenge.

In summary, the paper presents a robust framework for source-free domain adaptation by strategically utilizing the neighborhood structure of target domain data, offering a practical approach that mitigates the challenges associated with the absence of source data in adaptation tasks.