---
title: 'Weight Similarity (WS): Methods and Applications'
url: https://www.emergentmind.com/topics/weight-similarity-ws
type: topic
---

# Weight Similarity (WS): Methods and Applications

Searching arXiv for the cited papers to ground the article in the literature and verify identifiers.
{"query":"id:1902.10409 OR id:1606.00715 OR id:2502.16724 OR id:2002.04289 OR id:2501.16650 OR id:2501.07025 OR id:2508.00043 OR id:1804.00069 OR id:1811.04633 OR id:1304.1084 OR id:2508.07489 OR id:2208.04369 OR id:1903.10520","max_results":12}
Weight Similarity (WS) is not a single standardized construct in the arXiv literature. Across recent work, the phrase and the abbreviation denote several non-equivalent ideas: weighted coefficients over observations or attributes, learnable similarity functions in representation learning, direct comparison of neural-network weight matrices, and architectural or hardware constraints that force weights to align, repeat, or share structure [1902.10409][2501.07025][2501.16650][2508.00043]. The common thread is that “weight” is either part of the similarity definition itself or the object whose similarity is being measured.

## 1. Terminological scope

In the literature considered here, WS is best understood as a family of related but distinct technical usages rather than a single canonical metric. Some usages define similarity between data objects, some define similarity functions for learned embeddings, and some compare or constrain the parameters of neural networks directly.

| Usage | Object compared | Representative form |
|---|---|---|
| Weighted coefficient over data | attributes, sparse vectors, or linked nodes | \( \mathrm{WJS}(S,O)=\frac{\sum \min}{\sum \max} \), \( \mathrm{WS}=\mathrm{sNum}+\mathrm{sNan}+\mathrm{sNon} \), \( r=t^2/s^2 \) |
| Learned similarity function | embeddings or graph nodes | \( \langle y,y' \rangle_\lambda=\sum_k \lambda_k y_k y'_k \) |
| Direct weight comparison | neural-network weight matrices | DOCS, chain-normalized weight representations |
| Constraint on weights | neighboring units, encoder branches, or bit-columns | local \(L_2\) regularization, hard parameter sharing, duplicate-column reuse |

The ambiguity is substantive. In graph representation learning, “weighted inner product similarity” refers to a learnable indefinite inner product over embedding coordinates [1902.10409]. In sparse NLP data, “weighted similarity” is an additive score that combines numerical overlap, shared missingness, and mismatched missingness [2501.07025]. In large language models, weight similarity is a direct comparison between trained matrices, formalized by the Distribution of Cosine Similarity (DOCS) [2501.16650]. In topographic CNNs, Weight Similarity is a local regularizer that pushes neighboring units to develop similar incoming weights [2508.00043].

A useful practical distinction is between **weighted similarity**, where weights appear in the similarity formula, and **similarity of weights**, where trained parameters themselves are the compared objects. Several papers move between these meanings, but they should not be conflated.

## 2. Weighted similarity over observations, attributes, and missingness

One long-standing lineage defines WS as a coefficient over data vectors or pairwise observations. In weighted-set similarity, the canonical quantity is the Weighted Jaccard Similarity, also called the min-max kernel,
\[
\mathrm{WJS}(S,O)= \frac{\sum_{z \in S \cup O} \min\bigl(w(S,z),w(O,z)\bigr)} {\sum_{z \in S \cup O} \max\bigl(w(S,z),w(O,z)\bigr)},
\]
or, in vector form,
\[
S(\mathbf{x},\mathbf{y})= \frac{\sum_i \min(x_i,y_i)}{\sum_i \max(x_i,y_i)}.
\]
This is the target similarity estimated by Consistent Weighted Sampling, ICWS, and SCWS, and it reduces to ordinary Jaccard similarity when all weights are equal [1804.00069][1811.04633].

A second formulation is designed for highly sparse vectors with many missing values. There, WS is explicitly decomposed into three components,
\[
\mathrm{WS}=\mathrm{sNum}+\mathrm{sNan}+\mathrm{sNon},
\]
where \(\mathrm{sNum}\) measures agreement on jointly observed entries, \(\mathrm{sNan}\) rewards jointly missing entries, and \(\mathrm{sNon}\) penalizes one-sided missingness. For more than one common observed entry, the numerical term is overlap-scaled cosine similarity; for exactly one, it becomes a min/max ratio; and for no overlap it is set to zero [2501.07025]. This formulation is notable because shared absence is treated as informative similarity rather than discarded nuisance.

Mixed-type data introduce a different weighting problem. For Gower’s dissimilarity, the weighted form is
\[
d_{wG,ij} = \frac{\sum_{t=1}^{p}\delta_{ijt}\, d_{ijt}\, w_t} {\sum_{t=1}^{p}\delta_{ijt}\, w_t},
\]
with \(\delta_{ijt}\) handling missing values and \(d_{ijt}\) defined variable by variable for numeric, binary, and categorical data [2401.17041]. The paper argues that the nominally “unweighted” choice \(w_t=1\) hides an unbalanced contribution of variables, because categorical mismatches frequently contribute a full \(1\) whereas range-scaled quantitative dissimilarities rarely do. The proposed remedy is automatic weight selection that minimizes the dispersion of the correlations between each variable-specific dissimilarity and the resulting weighted Gower dissimilarity [2401.17041].

A related context-dependent approach appears in binary attribute spaces. There the dissimilarity is a weighted Hamming form,
\[
d(C_i, C_k) = \sum_{j=1}^n h(p_j)e(C_{ij}, C_{kj}),
\]
with
\[
h(p) = -\log_2(1-2p+2p^2),
\]
so that attribute weights depend on the empirical frequency \(p_j\) of value \(1\) in the current context [1304.1084]. Attributes near balance \(p_j \approx 1/2\) receive larger weight than attributes that are almost always \(0\) or \(1\).

In network analysis, weighted similarity can also mean weighted homophily. The multilayer-network paper defines a weighted intraclass correlation coefficient
\[
r = \frac{t^2}{s^2},
\]
where each linked pair contributes proportionally to its edge weight \(w_{ij}\), and a tuning exponent \(\alpha\) controls whether weak or strong ties dominate the measure [1606.00715]. This is explicitly a node-attribute similarity measure, not a structural node-similarity score.

## 3. Learned similarity functions and weighted graph representations

A separate usage treats WS as a learnable similarity model. The clearest example is weighted inner product similarity (WIPS), defined for embeddings \(y,y' \in \mathbb{R}^K\) by
\[
\langle y, y' \rangle_{\lambda}:=\sum_{k=1}^{K}\lambda_k y_k y_k',
\]
with learnable coordinate weights \(\lambda_k \in \mathbb{R}\), including negative values [1902.10409]. In Siamese form,
\[
h_{\mathrm{WIPS}}(x,x';\theta,\lambda)=\left\langle f_\theta(x), f_\theta(x') \right\rangle_{\lambda}.
\]
This generalizes IPS, subsumes SIPS and IPDS through explicit reparameterizations, and inherits the universal approximation capability of general similarities from IPDS. The theoretical point is that allowing negative \(\lambda_k\) yields an indefinite inner product associated with a pseudo-Euclidean space, making it possible to represent PD, CPD, and indefinite similarities within one parameterization [1902.10409].

The same paper gives a matrix-decomposition interpretation. For a symmetric similarity matrix \(H_*\) with spectral decomposition
\[
H_* = U\Lambda U^\top,
\]
truncation to the leading \(K\) eigenpairs gives
\[
H_* \approx U_K \Lambda_K U_K^\top,
\]
and the resulting approximation can be read as a weighted inner product over \(K\)-dimensional embeddings. This connects WS to a relaxed learnable eigendecomposition without orthogonality constraints [1902.10409].

In graph embedding, a different WS-like operationalization measures whether original edge weights are preserved by embedding similarity. The relevant score is the Pearson correlation between original edge weights and cosine similarity between endpoint embeddings,
\[
\operatorname{Corr}\big(w_{uv}, \cos(\mathbf{z}_u,\mathbf{z}_v)\big),
\]
computed over connected node pairs [2508.07489]. Weight-aware random walks consistently outperform unweighted and strength-based walks on this criterion, achieving correlations above \(0.90\) in network models, although performance in real-world networks is much more heterogeneous [2508.07489].

Weighted graph similarity also appears in kernel form. For complete weighted graphs, one paper first computes a commute-time minimum spanning tree, then probes it with a discrete-time quantum walk initialized by a weighted Perron-Frobenius operator, and finally compares the induced probability vectors with either
\[
k_{DP}(G_a,G_b)=\langle \mathcal{P}_a,\mathcal{P}_b\rangle
\]
or
\[
K_{JS}(G_a,G_b)=\exp\{-D_{JS}(\mathcal{P}_a,\mathcal{P}_b)\}.
\]
Both resulting similarities are presented as positive-definite graph kernels [1904.13239]. Here, weight information is not compared edgewise; it is re-encoded through commute time, tree extraction, and quantum-walk dynamics.

## 4. Direct similarity of neural-network weights

A more recent strand uses WS to compare trained neural-network parameters directly. In large language models, DOCS defines similarity between two weight matrices \(X=[X_1,\dots,X_m]\) and \(Y=[Y_1,\dots,Y_m]\) by first computing the cosine-similarity matrix
\[
C_{jk} = \frac{X_j^\top Y_k}{\|X_j\|\,\|Y_k\|},
\]
then taking the maximum absolute cosine for each column,
\[
s_{X_j} = \max_k |C_{jk}|,\qquad s_{Y_k} = \max_j |C_{jk}|,
\]
fitting Gumbel distributions to the two maxima vectors, and defining
\[
S_{\mathrm{DOCS}}(X,Y)=\frac{u_X+u_Y}{2},
\]
where \(u_X\) and \(u_Y\) are the fitted location parameters [2501.16650]. DOCS is claimed to satisfy permutation transformation invariance, symmetry, isotropic scaling invariance, reflexivity, and discriminativity on orthogonal matrices [2501.16650].

Empirically, DOCS reveals depth-wise organization in open-source LLMs. Adjacent layers frequently exhibit high weight similarity and tend to form clusters; base and instruction-tuned variants remain highly similar, with all reported DOCS scores above \(0.7\); and some MoE experts appear as outliers with much lower similarity to the rest [2501.16650]. The paper also stresses that representation similarity and weight similarity are distinct, especially in residual architectures where similar activations can arise from different underlying weights [2501.16650].

A different direct-parameter approach handles permutation ambiguity explicitly. The chain normalization rule defines
\[
\phi(W_1,W_2,\dots,W_l)=W_1W_2\cdots W_lW_l^T\cdots W_2^TW_1^T,
\]
so that hidden-unit permutations cancel along the chain [2208.04369]. Using these normalized weight representations, together with a hypothesis-training-testing procedure, the paper reports that identical neural networks optimized with SGD converge to a similar local solution in a metric space across MLP, CNN, and RNN settings [2208.04369]. The crucial claim is not raw parameter equality, but equivalence after normalization.

## 5. Weight similarity as a constraint in architectures and hardware

Some papers use WS not as a measurement but as an explicit constraint on model structure. In topographic CNNs, Weight Similarity is a local spatial loss applied to the \(11 \times 11\) grid of 121 units in the first fully connected layer. For each grid cell, the Moore neighborhood is identified, the \(L_2\) norm is computed between the incoming weight vector of that cell and those of each neighboring cell, and the resulting average pairwise distance enters the joint loss
\[
\mathcal{L}_{\text{joint}} = \mathcal{L}_{\text{CE}} + \lambda \,\mathcal{L}_{\text{spatial}},
\]
with \(\lambda \in \{0.1,0.3,0.5,1,2,3\}\) [2508.00043]. Compared with Activation Similarity and standard CNNs, WS improved robustness to noise, produced higher activation variance, and yielded stronger functional localization, with similarly responding units positioned at closer distances [2508.00043].

A nearby but distinct usage is **weight sharing**. In variational graph autoencoders, WS denotes hard equality of hidden-layer parameters across the mean and variance encoders, for example
\[
W^{(0)}_{\mu} = W^{(0)}_{\Sigma} = W^{(0)}.
\]
This reduces parameter count and encoder expressivity simultaneously, but across 10 VGAE variants and 16 datasets the shared-weight models remain within one standard deviation of their non-shared counterparts, leading the authors to recommend WS as a practical default [2502.16724]. In neural architecture search, by contrast, WS denotes super-net parameter sharing across candidate architectures. There the evidence is more skeptical: moderately good proxy/standalone correlations do not reliably translate into better search than random search, and search-space-induced bias is reported as the dominant issue [2002.04289].

A hardware-oriented usage appears in RRAM-based accelerators. There, a bit-level weight reordering strategy based on column similarity operates on binary two’s-complement weight matrices. Similarity between two bit-columns is measured by
\[
sHD(V_a,V_b)=\sum_{i=0}^{m-1} XOR(V_{a_i},V_{b_i}),
\]
and the method “treats bit sparsity as a special case of bit similarity,” keeping only one column from a pair of columns that have identical bit values after row reordering [2511.14202]. The reported result is a \(61.24\%\) average performance improvement and \(1.51\times\)–\(2.52\times\) energy savings under different sparsity ratios, with slight overhead relative to the compared design [2511.14202].

## 6. Conceptual distinctions and recurring caveats

A first recurring issue is terminological drift. “WS” may denote Weight Similarity, weight sharing, or Weight Standardization, depending on the paper. In micro-batch vision training, for example, WS means Weight Standardization: convolutional filters are normalized to zero mean and unit variance,
\[
\hat{\mathbf{W}}_{i,j} = \frac{\mathbf{W}_{i,j} - \mu_{\mathbf{W}_{i,\cdot}}}{\sigma_{\mathbf{W}_{i,\cdot}}},
\]
and the method is about optimization smoothness rather than similarity measurement [1903.10520]. The abbreviation alone is therefore not semantically reliable.

A second issue concerns what counts as informative absence. In the sparse-data metric, jointly missing entries contribute positively through \(\mathrm{sNan}\), and one-sided missingness contributes negatively through \(\mathrm{sNon}\) [2501.07025]. In Gower’s framework, missing values are instead excluded by \(\delta_{ijt}\), while asymmetric binary variables explicitly set \(\delta_{ijt}=0\) for the \((0,0)\) case so that joint absence does not count as similarity [2401.17041]. This suggests that WS formulations differ not merely in weighting scheme but in the semantics assigned to absence and missingness.

A third distinction is geometric. In weighted inner product similarity, negative coordinate weights are essential because ordinary inner products are tied to positive-definite geometry; WIPS uses negative \(\lambda_k\) to realize an indefinite inner product and thereby approximate similarities beyond the PD and CPD classes [1902.10409]. In parameter-space analysis, conversely, the main difficulty is not kernel definiteness but invariance: weight similarity must be distinguished from representation similarity, and raw parameters may be incomparable because of permutation or residual-connection effects [2501.16650].

Taken together, these usages show that WS is best treated as a technical umbrella term. In some settings it is a coefficient over sparse or mixed-type observations; in others it is a learnable similarity family, a direct weight-matrix comparison, or a structural regularizer that enforces redundancy. The literature does not converge on one universal definition, but it does converge on a narrower principle: weighting, whether in data space, embedding space, parameter space, or hardware mapping, is used to expose structure that uniform or unweighted formulations would miss.

Source: https://www.emergentmind.com/topics/weight-similarity-ws