Weight Space Correlation Analysis

• Weight Space Correlation Analysis is a family of methods that quantifies similarities among neural network weights by addressing symmetries and alignment challenges.
• The approach uses PCA projection and cosine similarity to differentiate between decodability and active feature utilization in model decision boundaries.
• It has practical applications in medical imaging for detecting shortcut learning and extends to broader studies in diffusion models, quantum states, and NAS.

Weight Space Correlation Analysis refers to methods that analyze statistical or geometric relations among neural-network parameters, parameter-derived directions, or classifier-head weights in order to quantify similarity, correspondence, or feature utilization. In its specific formulation for medical imaging, Weight Space Correlation Analysis (WSCA) measures the alignment between the classification heads of a primary clinical task and auxiliary metadata tasks, thereby distinguishing whether metadata merely exists in the embedding or is actively used by the decision boundary (Wong et al., 15 Dec 2025). In the broader Weight Space Learning literature, such analyses are situated within a structured view of weight space in which pretrained models form organized distributions and comparisons must account for symmetries, equivalence classes, and representation choices rather than relying on raw coordinate-wise inspection (Han et al., 10 Mar 2026).

1. Conceptual scope and relation to weight-space learning

Within the taxonomy of Weight Space Learning, weight space is treated as a meaningful domain for analysis and modeling rather than as the terminal by-product of optimization. The survey literature organizes this perspective into Weight Space Understanding (WSU), Weight Space Representation (WSR), and Weight Space Generation (WSG). For correlation analysis, WSU provides the geometric and symmetry conditions under which weight comparisons are meaningful, while WSR provides computational mechanisms for placing models into a shared comparison space, whether through direct weight embeddings or behavior-based signatures (Han et al., 10 Mar 2026).

A central implication is that “correlation” in weight space is not restricted to a single statistic. Depending on the setting, it may denote cosine similarity between projected classifier weights, cosine similarity between semantic directions in a PCA basis, rank correlation between model rankings, or gradient-alignment statistics along a trajectory in parameter space. This suggests that Weight Space Correlation Analysis is best understood as a family of symmetry-aware comparative procedures rather than a single universal formula.

In the medical-imaging formulation, the object of interest is feature utilization similarity: if two heads have aligned weight vectors, they are attending to similar directions in the shared embedding space; if they are weakly correlated or orthogonal, they are likely using different features (Wong et al., 15 Dec 2025). In broader settings, the same general logic reappears as model similarity, trajectory similarity, or domain-alignment analysis, but the relevant representation and invariance structure differ.

2. Symmetry, quotient structure, and the need for alignment

Weight-space comparisons are complicated by the fact that neural networks admit many symmetries. The broader framework formalizes this with a group action on weight space $\mathcal{W}$. If $G$ is the symmetry group, then weights related by symmetry belong to the same orbit, and the quotient space is

$$\mathcal{W}/G \coloneqq \{[\theta] \mid \theta \in \mathcal{W}\}, \qquad [\theta] = \{g \cdot \theta \mid g \in G\}.$$

This matters directly for correlation analysis because raw Euclidean similarity can be misleading when two parameter vectors differ only by neuron permutations, scaling symmetries, sign flips, or attention-head permutations (Han et al., 10 Mar 2026).

The alignment problem makes this issue concrete. For an $M$-layer MLP with weights

$$v=[W_m,b_m]_{m\in[M]},\qquad v'=[W'_m,b'_m]_{m\in[M]},$$

the goal is to find hidden-unit permutations that make the two networks comparable. The symmetry group is

$G=\Pi_{d_1}\times\cdots\times \Pi_{d_{M-1}},$

and alignment is posed as

$g(v,v')=\arg\min_{k\in G}\|v-v'\|_2^2.$

Because $v$ and $g_\# v$ represent the same function, direct parameter correlation without symmetry handling may confound functional equivalence with coordinate mismatch (Navon et al., 2023).

Equivariant Deep Weight Space Alignment sharpens this point by showing that the optimal alignment obeys two symmetries: equivariance under independent permutations of the two inputs and inversion under input swap,

$$g(gv,g'v') = g\cdot g(v,v')\cdot g'^T,\qquad g(v',v)=g(v,v')^T.$$

A symmetry-aware correlation analysis therefore often requires either explicit alignment, as in Deep-Align, or a representation that is already permutation-equivariant or function-based (Navon et al., 2023).

3. Canonical WSCA methodology for feature utilization

The specific WSCA methodology introduced for medical imaging is designed to answer a narrower question than generic embedding probing: not whether a confounder is decodable from the latent representation, but whether the classifier uses that confounder for its final prediction (Wong et al., 15 Dec 2025).

The setup distinguishes a primary task from auxiliary metadata tasks. A baseline single-task model maps an ultrasound image $G$0 to a primary clinical label $G$1, while a multi-task model shares the same image input but adds auxiliary heads for metadata targets such as scanner, hospital, pixel spacing, or maternal factors. The paper uses a ResNet50 backbone in this framework (Wong et al., 15 Dec 2025).

WSCA treats the final classification-layer weights as an attention vector over the shared embedding space. Let $G$2 denote the primary-task weight vector and $G$3 the auxiliary metadata-task weight vector. Because the embedding space may contain inactive directions, the method first projects the frozen-backbone embeddings of the training set $G$4 into the active data manifold by PCA. The projection matrix $G$5 is formed from the top principal components that explain 99% of variance, with a minimum of 50 components retained. The classification weights are then projected as

$G$6

After projection, WSCA computes pairwise correlation, implemented as cosine similarity, between projected weight vectors. High correlation is interpreted as similar feature usage; low correlation is interpreted as decoupled feature usage. The resulting statistic is a proxy for shortcut learning: if the decision boundary for the clinical task aligns with the decision boundary for a confounder, the model is likely relying on that confounder (Wong et al., 15 Dec 2025).

This methodology is deliberately different from linear probing. A linear probe establishes encodability or presence: it shows that scanner, ethnicity, or another variable can be predicted from the embedding. WSCA addresses utilization: the clinical head can remain largely orthogonal to a metadata direction even when that direction is strongly decodable from the embedding. The distinction between presence and utilization is the method’s central conceptual contribution (Wong et al., 15 Dec 2025).

4. Empirical validation in fetal ultrasound and spontaneous preterm birth prediction

The paper first validates WSCA in a controlled shortcut-learning setting using fetal ultrasound. Shortcut learning was artificially induced by curating the training data so that each fetal plane was strongly associated with a particular scanner model. In that setting, the correlation between fetal-plane head weights and scanner head weights increased markedly, especially in the multi-task regime where the shared embedding was explicitly encouraged to encode scanner information. This confirmed the intended behavior of WSCA: when shortcut learning is present, strong alignment emerges; when the shortcut is absent, alignment remains low (Wong et al., 15 Dec 2025).

The same dataset also illustrates the difference between decodability and use. Separate classifiers trained on embeddings of a plane-classification model showed that scanner, pixel spacing, hospital ID, year of study, and some maternal or demographic variables were strongly decodable. Despite this, WSCA showed that the plane classifier’s weights were only weakly correlated with scanner weights on the unbiased dataset. The interpretation given is that scanner information is encoded but not actively used for plane prediction, and that this decoupling remains visible even in the multi-task setting (Wong et al., 15 Dec 2025).

The method is then applied to SA-SonoNet for spontaneous preterm birth (sPTB) prediction. SA-SonoNet extends SonoNet by incorporating the raw cervix image, a segmentation map of relevant anatomy, and pixel spacing information repeated spatially into image dimensions. The original model uses a $G$7 feature map followed by average pooling. For WSCA, the feature map is flattened into a 252-dimensional vector, auxiliary metadata heads are attached, and the original average-pooling classifier is represented by an all-ones vector of length 252 so that all heads can be compared in the same weight space (Wong et al., 15 Dec 2025).

The resulting correlation matrix was clinically differentiated rather than uniformly confounded. Birth weight showed strong correlation with the sPTB classifier, cervical length showed moderate correlation, and pixel spacing also showed moderate correlation. Scanner type showed only weak correlation. The paper interprets this as evidence that, although the representation contains substantial metadata, the final sPTB decision boundary is aligned with clinically relevant factors and decoupled from clinically irrelevant acquisition factors such as scanner (Wong et al., 15 Dec 2025).

5. Related forms of weight-space correlation analysis in other domains

Outside the medical-imaging formulation, the same analytical logic appears in several adjacent settings. These methods differ in the object being correlated and the comparison geometry they assume.

Setting	Correlation object	Main finding
Customized diffusion models (Dravid et al., 2024)	Absolute cosine similarity between learned semantic directions in a PCA subspace of LoRA weights	As the number of fine-tuned models increases, the average cosine similarity between attribute directions decreases, and the space becomes more interpretable
Neural quantum states (Hernandes et al., 21 Mar 2025)	PCA structure of weights obtained by adiabatic fine-tuning across a phase diagram	The first principal component shows a pronounced minimum at the phase transition
Training-mixture recovery (Huang et al., 2 Jul 2026)	Alignment between negative sample gradients and the direction from pseudo-checkpoints to the final model	Domain-wise geometric footprints can recover training mixtures with average MAE as low as 0.046 for BERT and 0.104 for GPT-2-Small
Weight-sharing NAS (Yu et al., 2020)	Sparse Kendall-Tau between super-net rankings and ground-truth stand-alone rankings	Sparse Kendall-Tau correlates better with final search performance than super-net accuracy

In customized diffusion models, a large collection of identity-specific LoRA fine-tunes is modeled as a PCA subspace, “weights2weights,” and entanglement is quantified by absolute cosine similarity between attribute directions. Lower cosine similarity corresponds to better disentanglement, and the reported trend is that larger model collections yield less correlated directions, reducing spurious co-occurrence effects such as Male versus Beard or Young versus Bald (Dravid et al., 2024).

In neural quantum states, adiabatic fine-tuning across a control-parameter axis induces strongly correlated weight representations across neighboring parameter values. PCA is then used as the main structural diagnostic, and the first principal component exhibits a pronounced minimum at the known phase transition for both the transverse field Ising model and the $G$8-$G$9 Heisenberg model (Hernandes et al., 21 Mar 2025).

In WARP for training-data portfolio recovery, the correlation object is a geometric footprint matrix built from domain-wise averages of alignment scores

$$\mathcal{W}/G \coloneqq \{[\theta] \mid \theta \in \mathcal{W}\}, \qquad [\theta] = \{g \cdot \theta \mid g \in G\}.$$0

where $$\mathcal{W}/G \coloneqq \{[\theta] \mid \theta \in \mathcal{W}\}, \qquad [\theta] = \{g \cdot \theta \mid g \in G\}.$$1 is the direction from a pseudo-checkpoint $$\mathcal{W}/G \coloneqq \{[\theta] \mid \theta \in \mathcal{W}\}, \qquad [\theta] = \{g \cdot \theta \mid g \in G\}.$$2 to the final fine-tuned model. These trajectories are simulated by model merging, and the resulting footprint can be mapped to domain proportions either by a parameter-free softmax readout or by a 2-layer MLP (Huang et al., 2 Jul 2026).

A distinct but related line appears in weight-sharing NAS, where the issue is not semantic feature use but the fidelity of shared weights as a ranking proxy. There the key correlation statistic is sparse Kendall-Tau, defined by collapsing architectures whose ground-truth accuracies differ by less than 0.1% into the same rank group. The resulting rank correlation is reported to track final search performance better than super-net validation accuracy, especially on NASBench-101 and DARTS-NDS (Yu et al., 2020).

6. Interpretation, limitations, and recurrent misconceptions

The first recurrent misconception is to equate encodability with utilization. WSCA rejects that equivalence explicitly. A metadata variable can be strongly predictable from the embedding while remaining weakly correlated with the primary classifier head. In this case the information is present but ignored by the decision boundary. Conversely, high alignment suggests similar feature usage, but it does not by itself establish the causal direction of dependence (Wong et al., 15 Dec 2025).

The second misconception is to treat raw parameter similarity as intrinsically meaningful. Because weight space contains large symmetry-induced equivalence classes, naive parameter correlation may conflate neuron reindexing or other invariances with substantive functional difference. This is why alignment procedures, quotient-space reasoning, permutation-equivariant encoders, or behavior-based embeddings are often required before correlation values can be interpreted (Han et al., 10 Mar 2026).

A third limitation concerns dataset and representation bias. In the diffusion-model setting, the weight space inherits dataset biases and label limitations, and correlated traits can remain entangled when the model collection is small or biased. In the WARP setting, the usefulness of alignment statistics depends on the geometric footprint exposed by pseudo-checkpoints and the probing set. These examples suggest that weight-space correlation is representation-dependent: it reveals structure relative to a chosen basis, trajectory, or task decomposition rather than an invariant essence of the model (Dravid et al., 2024).

A related methodological caution comes from adjacent work on hidden-layer neuron correlation. That line studies a neuron-correlation proxy based on the mean vector space angle (“m-angle”) between neuron directions rather than a PAC-Bayesian weight-correlation measure. Large m-angle corresponds to weak neuron correlation, small m-angle to strong neuron correlation, and the reported trend is that persistently weak correlation is associated with entropy reduction and better generalization. This is closely related in spirit but not equivalent to classifier-head WSCA, underscoring that “correlation analysis” can refer to structurally different objects across the literature (Jin et al., 2020).

Taken together, these limitations indicate that Weight Space Correlation Analysis is most reliable when its correlation object, invariance assumptions, and interpretive target are stated explicitly. In its narrow medical-imaging form, it is an interpretable test of whether a model uses the “right reasons.” In the broader literature, it is a general strategy for extracting information from the geometry, alignment, and directional structure of learned weights.