Layer-Wise Weight Difference Analysis

• Layer-Wise Weight Difference Analysis is a set of methods that quantify per-layer variations in neural network weights to diagnose structural and behavioral differences.
• Techniques such as cosine similarity, perturbation norms, and relative weight change reveal intra-model discrepancies, guiding localized interventions.
• These methods inform effective strategies in quantization, federated learning, and training stability by pinpointing layers that amplify small changes or degrade performance.

Layer-wise weight difference analysis denotes a class of methods that measure how network parameters differ across layers, checkpoints, perturbation regimes, adjacent layers, or decentralized replicas, and then use those measurements to explain or control model behavior. In the literature, the relevant difference may be expressed as a cosine similarity between a client layer and the preceding global layer in federated learning, a perturbation norm between full-precision and quantized weights, a residual between adjacent convolution layers, a relative checkpoint-to-checkpoint change during training, or an inter-layer delta in a transformer (Rehman et al., 2023, Gluska et al., 2020, Lee et al., 2019, Agrawal et al., 2020, Wietrzykowski, 19 Mar 2026). The common premise is that a model-level scalar often obscures layer-specific heterogeneity: some layers remain well aligned, some drift, some amplify small perturbations, and some are structurally redundant.

1. Mathematical formulations and measurement regimes

A central formulation appears in federated learning, where L-DAWA measures layer-wise alignment between the $n$-th layer of client $k$ and the previous global model by cosine similarity

$$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$

with angular divergence $\theta_n^k = \arccos(\delta_n^k)$. In practice, L-DAWA works directly with $\delta_n^k$ to avoid the cost of $\arccos$ (Rehman et al., 2023). In quantization analysis, the corresponding layer-wise weight difference is the perturbation norm

$$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$

paired with a degradation term

$$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$

Under standard “small, zero-mean, independent” quantization-noise assumptions, the overall degradation satisfies $\Delta \approx \sum_{i=1}^n \Delta_i$ (Gluska et al., 2020).

A second regime measures differences across training time. For a layer $\ell$ with parameters $k$0 at epoch or iteration $k$1, relative weight change is defined as

$k$2

with an $k$3 variant

$k$4

This metric normalizes for layer scale and exposes which layers are still moving and which are settling (Agrawal et al., 2020).

A third regime measures differences across neighboring layers rather than across time. In inter-layer weight prediction, the residual at layer $k$5 is

$k$6

with special cases based on full-search or collocated prediction. The underlying Smoothly-Varying Weight Hypothesis states that adjacent convolution-layer weights share strong similarity in shapes and values, so the residuals tend to be small and follow narrow Laplace distributions (Lee et al., 2019). In transformer analysis, the analogous inter-layer object is the adjacent-layer delta

$k$7

whose consecutive correlations quantify oscillatory structure in layer space (Wietrzykowski, 19 Mar 2026).

These formulations are not interchangeable, but they all instantiate the same analytical move: replace a single global discrepancy by a layer-indexed family of discrepancies. This suggests that “layer-wise weight difference analysis” is best understood as a methodological pattern rather than a single metric.

2. Quantization degradation, perturbation attribution, and local repair

In post-training quantization analysis, the core objective is to decompose global degradation into per-layer terms that can be measured directly and acted upon. The hybrid-network construction $k$8, in which only layer $k$9 and its local activations are quantized, yields $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$0, while $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$1 measures the corresponding weight perturbation. Empirically, the degradation exhibits a strong monotonic relationship

$$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$2

for a slowly-varying link function $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$3 (Gluska et al., 2020). The paper further distinguishes intrinsic factors, such as weight range, variance, outlier ratio, and activation dynamic range, from extrinsic factors that reflect downstream amplification or attenuation. Using the Jacobian $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$4, the layer contribution is approximated by

$$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$5

so equal $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$6 perturbations need not induce equal degradation.

The ResNeXt26 case study makes this distinction concrete. Quantizing ResNeXt26 from a full-precision top-1 of $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$7 to $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$8 bits for both weights and activations yields top-1 $$\delta_n^k = \frac{w_n^g \cdot w_n^k}{\|w_n^g\| \, \|w_n^k\|},$$9, so $\theta_n^k = \arccos(\delta_n^k)$0. Layer-wise analysis reports, among other entries, $\theta_n^k = \arccos(\delta_n^k)$1 with $\theta_n^k = \arccos(\delta_n^k)$2 for conv1, $\theta_n^k = \arccos(\delta_n^k)$3 with $\theta_n^k = \arccos(\delta_n^k)$4 for conv2, $\theta_n^k = \arccos(\delta_n^k)$5 with $\theta_n^k = \arccos(\delta_n^k)$6 for conv4, and $\theta_n^k = \arccos(\delta_n^k)$7 with $\theta_n^k = \arccos(\delta_n^k)$8 for conv10. Summing all $\theta_n^k = \arccos(\delta_n^k)$9 gives $\delta_n^k$0, in close agreement with the global degradation, while conv4 alone accounts for $\delta_n^k$1 or about $\delta_n^k$2 of the total even though its perturbation norm is not the absolute largest. That pattern implies an extrinsic amplification effect rather than a purely local fail case (Gluska et al., 2020).

The same framework supports local interventions. The layer-wise clipping operator

$\delta_n^k$3

reduces outlier-driven quantization range. In the ResNeXt26 example, setting $\delta_n^k$4 to the $\delta_n^k$5th percentile of $\delta_n^k$6 and leaving all other $\delta_n^k$7 improved top-1 to $\delta_n^k$8 and reduced $\delta_n^k$9 to $\arccos$0, whereas Global SAWB clipping on all layers dropped top-1 to $\arccos$1 and increased $\arccos$2 to $\arccos$3 (Gluska et al., 2020). A plausible implication is that layer-wise weight difference analysis is most informative when it is paired with layer-local remediation rather than uniform global heuristics.

Inter-layer weight prediction addresses compression from a different angle. Rather than comparing full-precision and quantized copies of the same layer, it predicts a convolution layer from an earlier or adjacent layer and quantizes the residual. Under ILWP–ILL, the residuals in MobileNet on CIFAR-100 fit a Laplace distribution with average scale $\arccos$4, compared with $\arccos$5 under ILWP–FSS, and about $\arccos$6 of all quantized residuals collapse to zero at $\arccos$7 bits versus approximately $\arccos$8 in the raw-weight baseline (Lee et al., 2019). The inter-layer loss

$\arccos$9

with $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$0, regularizes collocated adjacent-layer kernels toward equality and eliminates index bits in ILWP–ILL (Lee et al., 2019).

Method	total bits (KB)	top-1 acc
Baseline (no pred.)	70.192	92.125%
ILWP–FSS	78.893	92.225%
ILWP–LSS	75.955	92.235%
ILWP–ILL	33.373	91.480%

For MobileNet on CIFAR-10 at $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$1 bits, these results show that ILWP–ILL effectively halves the quantized model size at less than $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$2 drop in accuracy (Lee et al., 2019). The broader significance is that small residuals between adjacent layers can be operationalized as a storage primitive, not merely as a descriptive statistic.

3. Federated aggregation, divergence awareness, and adaptive shrinking

In federated learning, layer-wise weight difference analysis is used to counter client drift under non-IID data. L-DAWA begins from the observation that heterogeneous client data drives different layers to drift in different directions, so a single scalar per client, as in FedAvg, cannot capture which layers remain aligned and which have drifted. Its aggregation weight is defined per layer and per client as

$$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$3

leading to the global update

$$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$4

If a baseline such as FedAvg would normally use $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$5, L-DAWA can instead set

$$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$6

so that the per-layer weights still sum to one (Rehman et al., 2023). High-alignment layers receive greater weight, while low or negative alignment is down-weighted or reversed.

The reported empirical findings emphasize optimization geometry as well as downstream accuracy. On both SimCLR and Barlow Twins in cross-silo CIFAR-10, L-DAWA reaches a given loss or accuracy in fewer communication rounds, and the global model is driven into a wider, smoother valley than FedAvg, reducing zig-zag behavior. The average $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$7 across layers remains steadier under L-DAWA, whereas FedAvg shows large oscillations. In downstream linear probing under non-IID CIFAR-10 with $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$8 in the cross-silo setting, SimCLR with FedAvg achieved $$\delta_{i,p} \coloneqq \|W_i - W_i^q\|_p,\qquad p\in\{2,\infty\},$$9 while L-DAWA achieved $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$0, a $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$1 absolute improvement; on CIFAR-100 SimCLR, the result was $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$2; on Tiny ImageNet SimCLR, approximately $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$3. Gains also held under $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$4 semi-supervised linear probes, cross-device settings with $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$5, and transfer from Tiny ImageNet to CIFAR (Rehman et al., 2023). An ablation comparing full-model and layer-wise divergence reported that “M-DAWA” improves accuracy but costs about $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$6 longer per aggregation, whereas L-DAWA has approximately $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$7 overhead versus FedAvg and provides the best trade-off (Rehman et al., 2023).

FedLWS uses an adjacent but distinct quantity: variance of layer-wise client updates. Writing client $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$8’s local model after $$\Delta_i \coloneqq \mathbb E_x[L(f(x;W)) - L(f^{(i)}(x))].$$9 epochs as

$\Delta \approx \sum_{i=1}^n \Delta_i$0

and the baseline aggregate as

$\Delta \approx \sum_{i=1}^n \Delta_i$1

the introduction of a shrink factor $\Delta \approx \sum_{i=1}^n \Delta_i$2 yields

$\Delta \approx \sum_{i=1}^n \Delta_i$3

FedLWS interprets the last term as weight-shrinking regularization and ties its strength to the variance

$\Delta \approx \sum_{i=1}^n \Delta_i$4

of client gradients at layer $\Delta \approx \sum_{i=1}^n \Delta_i$5. The adaptive layer-wise shrink factor is

$\Delta \approx \sum_{i=1}^n \Delta_i$6

and the final model concatenates $\Delta \approx \sum_{i=1}^n \Delta_i$7 across layers (Shi et al., 19 Mar 2025). No proxy data is needed, because the method uses only server-available global weights and client updates.

Across CIFAR-10, CIFAR-100, TinyImageNet, and AG News, FedLWS reported consistent accuracy gains of $\Delta \approx \sum_{i=1}^n \Delta_i$8–$\Delta \approx \sum_{i=1}^n \Delta_i$9 absolute over FedAvg, FedProx, FedDisco, and FedLAW, with especially strong gains under low Dirichlet $\ell$0. On ResNet20/CIFAR-10 with $\ell$1, model-wise shrinking achieved $\ell$2 whereas layer-wise shrinking achieved $\ell$3. The shrink factors vary more in early rounds and converge toward $\ell$4 as training stabilizes, and ablations show that adjusting only the regularization term or only the gradient scale does not yield the full benefit (Shi et al., 19 Mar 2025). Taken together, L-DAWA and FedLWS demonstrate two distinct uses of layer-wise difference information in FL: directional reweighting and adaptive regularization.

4. Relative weight change and depth-dependent learning dynamics

Relative weight change provides a direct per-layer view of how training unfolds in standard supervised networks. In the CNN study, weights were snapshotted once per epoch over $\ell$5 epochs, and RWC was computed for AlexNet, VGG-19 with BatchNorm, and ResNet-18 across MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100, with five random seeds per architecture-dataset pair (Agrawal et al., 2020). The broad trend was that “middle” or “late” layers move more per epoch than “early” layers on every dataset. For the simplest task, MNIST, RWC curves decay quickly within the first $\ell$6–$\ell$7 epochs, with early layers leveling near $\ell$8 and the deepest layers plateauing around $\ell$9–$k$00. On CIFAR-100, the deepest layers continue to change substantially late in training: in ResNet-18 at epoch $k$01, block $k$02 had RWC approximately $k$03 versus approximately $k$04 for block $k$05 (Agrawal et al., 2020).

The WeightScale framework extends this analysis from raw trajectories to clusterable representations. Over $k$06 checkpoints, each layer produces an RWC time series $k$07, collected into a layer-by-time matrix $k$08. PCA on row-centered $k$09 yields a reduced embedding $k$10, and K-Means with k-means++ groups layers with similar learning dynamics (Agrawal et al., 2021). The method optionally removes outlier RWC entries beyond $k$11 of a layer’s distribution and replaces them with the layer mean. Applied to ResNet-50 and EfficientNet-B4 on CIFAR-10, CIFAR-100, and SVHN, the results showed that on simpler tasks such as CIFAR-10 and SVHN, early and mid-network layers exhibit large RWC early in training and then decay quickly, while later layers show much smaller RWC throughout; on CIFAR-100, the pattern flips and later layers undergo substantially more RWC (Agrawal et al., 2021). EfficientNet-B4, with about $k$12 layers, showed the same qualitative pattern, though its depthwise and pointwise building blocks produced sub-clusters within primitive-type groups.

A more mechanistic account appears in the analysis of deep normalized networks. There, each normalized layer $k$13 has expected squared Frobenius norm $k$14 and base gradient magnitude $k$15. Orthogonality induced by normalization leads to the discrete update

$k$16

and, in continuous time,

$k$17

The effective learning rate is

$k$18

and the ratio between any two layers satisfies $k$19 under constant learning rate (Mehmeti-Göpel et al., 2023). The critical threshold for flipping the ordering of two ELRs is

$k$20

The proposed sub-critical warm-up computes current ELRs, finds the pair with the largest ELRs, and sets the step size to the corresponding $k$21; after at most $k$22 warm-up steps, all pairwise ratios reach $k$23 (Mehmeti-Göpel et al., 2023). In very deep networks without residuals, reducing ELR spread restored trainability, whereas smaller residual networks already had much milder spread (Mehmeti-Göpel et al., 2023).

5. Anatomical heterogeneity and inter-layer structure in transformer LLMs

In transformer LLMs, layer-wise weight difference analysis has been used to challenge the assumption that layers are anatomically homogeneous. The study of SmolLM2-135M, a $k$24-layer, $k$25M-parameter causal LLM, evaluated five diagnostics: weight predictability, ablation degradation, recovery speed, weight manipulation robustness, and structural analysis (Wietrzykowski, 19 Mar 2026). Weight predictability fits Ridge regression on polynomial and sinusoidal basis functions of the layer index and evaluates

$k$26

Average $k$27 values were $k$28 for mlp.gate_proj, $k$29 for mlp.down_proj, $k$30 for self_attn.q_proj, $k$31 for self_attn.k_proj, $k$32 for mlp.up_proj, $k$33 for self_attn.o_proj, and $k$34 for self_attn.v_proj, with best values ranging from $k$35 to $k$36 except for self_attn.v_proj, whose best value was $k$37 (Wietrzykowski, 19 Mar 2026).

The same work defined adjacent-layer deltas $k$38 and found that the Pearson correlation between $k$39 and $k$40 is approximately $k$41 with $k$42 variation across all components, indicating a universal oscillatory delta pattern. A fitted model

$k$43

with a minimal fit $k$44 captures the observed period-$k$45 sign inversion (Wietrzykowski, 19 Mar 2026). Yet the paper also reported that predicted weights cause catastrophic failure due to nonlinear error accumulation. The text attributes part of that sensitivity to softmax attention, with sensitivity approximately $k$46 for small per-layer prediction errors $k$47 (Wietrzykowski, 19 Mar 2026). High regularity in weight space therefore does not imply functional interchangeability.

Layer importance was measured by replacing a layer with the neighbor average

$k$48

and computing percent degradation

$k$49

The range extended from $k$50 at layer $k$51, where removal improved perplexity, to $k$52 at layer $k$53, spanning more than a $k$54 range in importance (Wietrzykowski, 19 Mar 2026). The critical core comprised layers L8–L11, while anti-layers included L14 and L17. Recovery-speed experiments added Gaussian noise with standard deviation $k$55 to a single layer, froze all other layers, and fine-tuned that layer alone. Critical-core layers often failed to recover within $k$56 steps, with L11 remaining at $k$57 baseline perplexity, whereas L14 and L17 recovered instantly and even improved (Wietrzykowski, 19 Mar 2026).

Weight manipulation robustness further sharpened the distinction. Across five tested strategies on redundant layers, only “Scale × 0.9” preserved model quality, with $k$58; all other manipulations produced millions to tens of millions of percent degradation (Wietrzykowski, 19 Mar 2026). The associated Growth Transformer Training, which allocates computational budget by layer importance, achieved approximately $k$59 cost reduction, and a proof-of-concept experiment reported $k$60 lower validation loss than uniform training at identical parameter count while being $k$61 faster (Wietrzykowski, 19 Mar 2026). These findings extend layer-wise weight difference analysis from descriptive diagnostics to resource-allocation policy.

6. Recurring themes, interpretive cautions, and methodological significance

Several themes recur across these otherwise different settings. First, local magnitude and global effect are distinct. In quantization, conv4 in ResNeXt26 dominated degradation without having the absolute largest perturbation norm, because downstream amplification mattered (Gluska et al., 2020). In federated learning, client disagreement is not uniform across layers, so per-layer angular alignment or per-layer gradient variance provides a finer control signal than a client-level scalar or a model-level shrink factor (Rehman et al., 2023, Shi et al., 19 Mar 2025). In transformer analysis, high inter-layer predictability coexists with catastrophic failure under direct weight substitution, showing that regularity in parameter space can be functionally fragile (Wietrzykowski, 19 Mar 2026).

Second, layer-wise analysis repeatedly favors local intervention over global intervention. Local clipping on conv4 only reduced ResNeXt26 degradation to $k$62, whereas Global SAWB clipping increased degradation to $k$63 (Gluska et al., 2020). L-DAWA’s per-layer cosine weighting offered approximately $k$64 overhead relative to FedAvg, while a whole-model cosine variant cost about $k$65 longer per aggregation (Rehman et al., 2023). FedLWS reported $k$66 for layer-wise shrinking versus $k$67 for model-wise shrinking on ResNet20/CIFAR-10 with $k$68 (Shi et al., 19 Mar 2025). These results suggest that a principal value of layer-wise weight difference analysis lies in discriminating where an intervention should be applied, not merely in quantifying aggregate instability.

Third, the measurements are portable across objectives and architectures. The cited works apply them to self-supervised and supervised federated learning, post-training quantization, inter-layer compression, CNN training dynamics, deep normalized networks, EfficientNet-B4, and causal LLMs (Rehman et al., 2023, Lee et al., 2019, Agrawal et al., 2021, Mehmeti-Göpel et al., 2023, Wietrzykowski, 19 Mar 2026). The metrics themselves differ—cosine similarity, perturbation norms, residuals, RWC, ELR ratios, Pearson correlations, perplexity degradation—but the operational logic is stable: identify a layer-wise discrepancy, relate it to downstream behavior, and use that relation to diagnose, regularize, compress, or reallocate compute.

A common misconception is that layer-wise weight difference analysis is reducible to inspecting which layers have the largest absolute changes. The evidence does not support that simplification. Extrinsic gain can make a modest perturbation disproportionately harmful, anti-layers can improve performance when removed, and global regularization can underperform selective layer-wise control (Gluska et al., 2020, Wietrzykowski, 19 Mar 2026, Shi et al., 19 Mar 2025). A plausible implication is that the method is most informative when layer-wise statistics are interpreted jointly with architecture, training regime, and downstream sensitivity rather than as standalone magnitudes.