HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation (1912.11976v1)

Abstract: Minimizing the discrepancy of feature distributions between different domains is one of the most promising directions in unsupervised domain adaptation. From the perspective of distribution matching, most existing discrepancy-based methods are designed to match the second-order or lower statistics, which however, have limited expression of statistical characteristic for non-Gaussian distributions. In this work, we explore the benefits of using higher-order statistics (mainly refer to third-order and fourth-order statistics) for domain matching. We propose a Higher-order Moment Matching (HoMM) method, and further extend the HoMM into reproducing kernel Hilbert spaces (RKHS). In particular, our proposed HoMM can perform arbitrary-order moment tensor matching, we show that the first-order HoMM is equivalent to Maximum Mean Discrepancy (MMD) and the second-order HoMM is equivalent to Correlation Alignment (CORAL). Moreover, the third-order and the fourth-order moment tensor matching are expected to perform comprehensive domain alignment as higher-order statistics can approximate more complex, non-Gaussian distributions. Besides, we also exploit the pseudo-labeled target samples to learn discriminative representations in the target domain, which further improves the transfer performance. Extensive experiments are conducted, showing that our proposed HoMM consistently outperforms the existing moment matching methods by a large margin. Codes are available at \url{https://github.com/chenchao666/HoMM-Master}

• The paper’s main contribution is extending moment matching to third- and fourth-order statistics for more robust domain alignment.
• It proposes practical group and random sampling strategies to reduce the computational cost of high-dimensional tensor calculations.
• Numerical evaluations on benchmark datasets demonstrate improved feature transferability and performance over traditional methods.

Higher-order Moment Matching for Unsupervised Domain Adaptation: A Technical Overview

The paper "HoMM: Higher-order Moment Matching for Unsupervised Domain Adaptation" addresses a critical challenge in unsupervised domain adaptation (UDA): reducing the discrepancy between feature distributions of distinct domains. Traditional moment-based domain adaptation approaches predominantly focus on aligning either first-order (mean) or second-order (covariance) statistics, such as Maximum Mean Discrepancy (MMD) and Correlation Alignment (CORAL). However, these methods can fall short when dealing with complex, non-Gaussian feature distributions commonly encountered in real-world applications. The authors propose a novel approach, Higher-order Moment Matching (HoMM), which extends domain alignment to higher statistical orders, including third-order and fourth-order moments.

The core idea of HoMM is to incorporate higher-order statistics to achieve a more nuanced and precise domain alignment. The paper demonstrates that HoMM, particularly the third-order and fourth-order moment matching, can effectively approximate a wider array of non-Gaussian distributions. An interesting aspect of this work is the equivalence it draws with existing strategies: first-order HoMM aligns with MMD, and second-order HoMM corresponds to CORAL. Moreover, the authors extend HoMM within the framework of reproducing kernel Hilbert spaces (RKHS), allowing the method to perform arbitrary-order moment tensor matching.

A significant component of this research is the introduction of practical strategies—specifically group moment matching and random sampling matching—to mitigate the computational expenses associated with high-dimensional tensor calculations. These strategies ensure the scalability and practical applicability of HoMM in real-world scenarios.

Numerical evaluations reveal that HoMM consistently surpasses existing moment matching methods by notable margins across several benchmark datasets, including the digits recognition datasets (e.g., SVHN to MNIST) and domain adaptation challenges in the Office-31 and Office-Home datasets. Notably, the approach demonstrates superior performance in aligning domains with complex distributions, thereby enhancing model transferability.

Additionally, the authors propose an innovative approach to address the sparsity of labels in target domains by leveraging pseudo-labeled target data. This further refines discriminative feature learning within the target domain, enhancing transfer performance.

The implications of this work are substantial for the field of unsupervised domain adaptation and highlight the potential of higher-order statistics in achieving finer-grained domain alignment. Theoretically, HoMM extends the understanding of moment matching by incorporating higher-order statistics, while practically, it provides a robust framework for improving performance in domain adaptation tasks.

Looking forward, the integration of HoMM into broader AI applications such as knowledge distillation and style transfer could be explored. This could pave the way for advancements in these domains by enabling more sophisticated and accurate feature representation alignment. As the field continues to evolve, the methodologies introduced in HoMM will likely inspire further research into higher-order statistical methods for various machine learning tasks.