On the Versatile Uses of Partial Distance Correlation in Deep Learning (2207.09684v3)

Abstract: Comparing the functional behavior of neural network models, whether it is a single network over time or two (or more networks) during or post-training, is an essential step in understanding what they are learning (and what they are not), and for identifying strategies for regularization or efficiency improvements. Despite recent progress, e.g., comparing vision transformers to CNNs, systematic comparison of function, especially across different networks, remains difficult and is often carried out layer by layer. Approaches such as canonical correlation analysis (CCA) are applicable in principle, but have been sparingly used so far. In this paper, we revisit a (less widely known) from statistics, called distance correlation (and its partial variant), designed to evaluate correlation between feature spaces of different dimensions. We describe the steps necessary to carry out its deployment for large scale models -- this opens the door to a surprising array of applications ranging from conditioning one deep model w.r.t. another, learning disentangled representations as well as optimizing diverse models that would directly be more robust to adversarial attacks. Our experiments suggest a versatile regularizer (or constraint) with many advantages, which avoids some of the common difficulties one faces in such analyses. Code is at https://github.com/zhenxingjian/Partial_Distance_Correlation.

• The paper introduces a theoretical framework that adapts distance and partial distance correlation for comparing and conditioning neural network features.
• The paper demonstrates that training networks with independent features improves resistance to adversarial attacks.
• The study proposes a practical regularization method for network comparisons, advancing interpretability and guiding architectural decisions.

A Study on the Uses of Partial Distance Correlation in Deep Learning

The paper "On the Versatile Uses of Partial Distance Correlation in Deep Learning" investigates the application of distance correlation, and its partial variant, within the context of deep learning. It addresses the challenge of comparing neural networks, especially of varying architectures, in a systematic and quantitative manner. This research endeavors to provide enhanced methodologies for comparing neural networks beyond the currently sparse landscape of tools like Canonical Correlation Analysis (CCA).

Summary and Key Contributions

The paper begins by introducing distance correlation as a versatile statistical measure capable of capturing both linear and nonlinear dependencies between random variables of different dimensions. Unlike CCA, which relies on feature space projections that could be inefficacious for nonlinear dependencies or computationally intensive when extended to deep networks, distance correlation provides a suitable alternative without the stringent requirement for similar dimensions across datasets.

Key contributions of the paper include:

1. Theoretical Review: A succinct review of the mathematical foundation underpinning distance correlation and partial distance correlation is provided, highlighting its benefits and the algorithmic adjustments needed for its deployment in neural network training.
2. Independent Features for Robustness: Demonstrating that training multiple networks to learn mutually independent features can enhance robustness against adversarial attacks, the paper introduces a formulation for minimizing the distance correlation between feature sets from different networks.
3. Network Comparison and Conditioning: The notion of partial distance correlation is extended to propose a framework for conditioning the learning of one network relative to another, thereby quantitatively measuring what unique information one network learns over the other. This can be especially useful in understanding how architectural variations (e.g., Vision Transformers vs. CNNs) encode different feature representations.
4. Practical Applications: The research illustrates the practical implementation of distance correlation as both a regularizer and a conditioning tool, which can be incorporated into various tasks such as learning disentangled representations, stimulating model diversity for shared tasks, and advancing understanding of network functional similarities and differences.

Implications for the Field

The implications of this paper are multifold. Practically, the work could lead to the development of neural networks with improved adversarial resistance by fostering independence among ensemble members. This is achieved through a training paradigm that minimizes transferability of adversarial examples by enforcing feature independence.

Theoretically, the proposed methods pave the way for a richer understanding of neural network behavior, providing tools for deeper inquiries into what constitutes learned representations across different model architectures. The use of distance and partial distance correlation potentially offers a unifying framework for such comparisons, which could form the basis for further studies in neural network interpretability or in informing architectural design choices.

Future Directions

The proposed application of distance correlation and its promising initial results open several avenues for future exploration:

• Extending this framework to other network comparison tasks beyond computer vision, potentially in LLMs or cross-modal networks, to assess its versatility.
• Investigating the robustness of these methods on larger datasets or under different training regimes could provide further confirmation of its practical utility.
• The real-time application of distance correlation during training could be explored, potentially offering dynamic regularization techniques adaptive to training progression.

In conclusion, this research puts forth significant strides in leveraging statistical measures for advancing the interpretability and robustness of deep learning models. Distance correlation, as posited by the authors, could become central to a new wave of methods examining inter-model comparisons and conditioning in neural networks.