Hilbert-Schmidt Independence Criterion

• Hilbert-Schmidt Independence Criterion (HSIC) is a non-parametric kernel-based measure that maps random variables to reproducing kernel Hilbert spaces to evaluate their statistical dependence.
• HSIC enables robust nonlinear evaluation of dependencies in applications such as image self-supervised learning and molecular representation by minimizing cross-correlation between learned embeddings.
• Methods like Barlow Twins and TwinBooster build on HSIC principles to enforce independence and reduce redundancy in deep representations, although challenges like batch size constraints remain.

The Hilbert-Schmidt Independence Criterion (HSIC) is a kernel-based statistical measure of dependence between random variables, widely adopted for feature learning, multi-view representation, and self-supervised objectives. HSIC quantifies the dependence between two random variables by mapping each variable to a reproducing kernel Hilbert space (RKHS) and evaluating the squared Hilbert-Schmidt norm of their cross-covariance operator. This approach enables non-parametric, nonlinear assessment of statistical dependence and forms the mathematical foundation behind several contemporary redundancy reduction and representation learning techniques, notably the Barlow Twins framework and its modern variants (Bandara et al., 2023, Schuh et al., 9 Jan 2024, Podsiadly et al., 24 Aug 2025).

1. Mathematical Formulation

Let $X$ and $Y$ be random variables with respective distributions. The HSIC is constructed as follows:

Given positive definite kernels $k$ and $\ell$ defined on the respective domains of $X$ and $Y$, HSIC is defined as the squared Hilbert-Schmidt norm of the cross-covariance operator between the RKHSs $\mathcal{H}_k$ and $\mathcal{H}_\ell$ induced by these kernels. Empirically, for a set of samples $\{(x_i, y_i)\}_{i=1}^N$, the empirical HSIC is given by:

$$\text{HSIC}(X, Y) = \frac{1}{(N-1)^2} \operatorname{Tr}(K H L H)$$

where $K, L \in \mathbb{R}^{N \times N}$ are kernel matrices with entries $K_{ij} = k(x_i, x_j), L_{ij} = \ell(y_i, y_j)$, $H = I - \frac{1}{N}\mathbf{1}\mathbf{1}^\top$ is the centering matrix, and $\operatorname{Tr}$ denotes the trace operation. When using the linear kernel $k(x, x') = x^\top x'$, HSIC reduces to a measure of second-order dependence.

Barlow Twins and related methods present a simplification that is equivalent to minimizing the empirical cross-covariance between representations—this aligns directly with the core mechanism of HSIC (Bandara et al., 2023). Specifically, for batch-normalized representations $Z^A, Z^B \in \mathbb{R}^{N \times d}$, the cross-correlation matrix

$$C_{ij} = \frac{1}{N} \sum_{b=1}^N Z^A_{b,i} Z^B_{b,j}$$

serves as a linear-kernel instantiation of HSIC. The Barlow Twins objective minimizes both $(1 - C_{ii})^2$ (invariance of diagonals) and $C_{ij}^2$ for $i \ne j$ (redundancy reduction), which is mathematically equivalent to minimizing the off-diagonal Hilbert-Schmidt norm component (Bandara et al., 2023).

2. Theoretical Properties

HSIC provides several desirable statistical properties:

• Characteristic Kernels: With characteristic kernels (e.g., Gaussian), HSIC vanishes if and only if $X$ and $Y$ are statistically independent, ensuring that the test is consistent against all dependencies.
• Non-parametric Nature: HSIC requires no explicit parametric model, making it applicable to arbitrary input domains and distributions.
• Connection to Self-supervised Losses: The Barlow Twins loss and its extensions exploit the equivalence between correlation minimization and HSIC, operationalizing independence constraints in deep representations (Bandara et al., 2023).

A plausible implication is that HSIC-regularized objectives provide more stable self-supervised learning signals in overparameterized neural networks, as evidenced by the improved generalization and robustness in high-dimensional settings (Bandara et al., 2023, Schuh et al., 9 Jan 2024).

3. Practical Implementations in Self-Supervised Learning

The HSIC principle underpins several self-supervised and multi-view learning algorithms:

• Barlow Twins: Minimizes the off-diagonal cross-correlation of feature embeddings to enforce statistical independence between representation components, with a loss function

$$\mathcal{L}_{\mathrm{BT}} = \sum_{i}(1 - C_{ii})^{2} + \lambda_{\mathrm{BT}} \sum_{i \neq j}C_{ij}^{2}$$

where $C$ is the (linear-kernel) cross-correlation matrix (Bandara et al., 2023).

• Mixed Barlow Twins: Enhances Barlow Twins with a mixed-sample regularization term, injecting synthetic interpolated data (via MixUp) and penalizing deviations from linearity in the embedding space. The total loss is

$$\mathcal{L} = \mathcal{L}_{\mathrm{BT}} + \lambda_{\mathrm{reg}}\mathcal{L}_{\mathrm{reg}}$$

where $\mathcal{L}_{\mathrm{reg}}$ enforces consistency of correlations for mixed samples, further stabilizing independence (Bandara et al., 2023).

• TwinBooster: Applies the HSIC-inspired objective to heterogeneous modalities (molecular fingerprints and text embeddings) in a shared projected space, enforcing cross-modal independence and invariance in molecular property prediction pipelines (Schuh et al., 9 Jan 2024).

For practical purposes, the cross-correlation version is favored due to computational efficiency on mini-batches and seamless integration with automatic differentiation frameworks.

4. Applications and Empirical Insights

HSIC-driven objectives have demonstrated efficacy across domains:

• Vision: In image self-supervised learning, Barlow Twins and Mixed Barlow Twins yield competitive or superior $k$-NN and linear evaluation accuracies compared to contrastive methods, especially in regimes with limited data, large representation dimensions, or prolonged training (e.g., CIFAR-10, ImageNet) (Bandara et al., 2023).
• Molecular Representation: TwinBooster leverages a mixed Barlow Twins objective for zero-shot molecular property prediction, outperforming contrastive and prototypical networks on the FS-Mol benchmark (Schuh et al., 9 Jan 2024).
• Multimodal Learning: The shared projector and HSIC-motivated loss in TwinBooster enable effective fusion of textual assay context and molecular graph structure, supporting strong generalization to unseen assays and molecules (Schuh et al., 9 Jan 2024).
• Redundancy Reduction in Transformers: Integration with self-distillation (DINO) yields robust, label-efficient vision transformers with minimal loss of semantic information, highlighting the compatibility of HSIC objectives with student-teacher and hybrid frameworks (Podsiadly et al., 24 Aug 2025).

Method	HSIC Principle Used	Domain
Barlow Twins	Linear-kernel	Images, vision
Mixed Barlow Twins	Linear-kernel + MixUp	Images, vision
TwinBooster	Cross-modal kernel	Molecules, text

This diversity illustrates the flexibility of HSIC as a foundation for independence-based regularization.

5. Limitations and Modifications

Empirical findings reveal challenges and adaptations related to HSIC in deep learning:

• Overfitting in Large-Dimensional Spaces: Standard Barlow Twins (and, by extension, HSIC objectives) can overfit in high embedding dimensions, especially with insufficient sample interaction. Mixed-sample regularization (MixUp) ameliorates this by introducing infinite synthetic samples, ensuring robust feature learning across longer training intervals (Bandara et al., 2023).
• Batch Size Constraints: Correlation-based objectives require sufficiently large batch sizes to provide stable estimators of dependence; this can restrict adoption on memory-limited hardware (Podsiadly et al., 24 Aug 2025).
• Modalities and Negative Samples: In multimodal applications, the assumption of independence can break if the modalities are not strictly complementary or are poorly aligned. The mixed-modality “mixed Barlow Twins” architecture enforces shared projection to increase cross-modal predictivity and invariance, but may still degrade on domains out-of-distribution from the training data (Schuh et al., 9 Jan 2024).

A plausible implication is that dynamic regularization weights or decorrelation penalties across mini-batches, as well as explicit hard negative mining, could enhance the stability and generalization of HSIC-based objectives in diverse data regimes.

6. Connections to Broader Representation Learning

HSIC occupies a central position in the landscape of statistical independence measures:

• It is tightly connected to mutual information estimation, serving as a lower bound under certain kernel choices.
• Numerous contemporary self-supervised losses—including those targeting feature decorrelation, invariance, and redundancy minimization—can be interpreted as simplified or derived versions of the HSIC principle (Bandara et al., 2023).
• The HSIC framework enables integration with non-contrastive learning, multi-view coherence, and cross-modal fusion, offering a spectrum of flexibility for representation design and evaluation in both vision and molecular domains (Bandara et al., 2023, Schuh et al., 9 Jan 2024).

The breadth of recent work leveraging variants of Barlow Twins and HSIC-related criteria demonstrates the continued relevance of kernel-based independence in modern deep learning pipelines for feature, representation, and modality learning (Bandara et al., 2023, Schuh et al., 9 Jan 2024, Podsiadly et al., 24 Aug 2025).