Layerwise Exponentially Weighted Connectivity (LEWC)

• LEWC is defined as the property where each network layer’s output in a weight interpolation is an exponentially weighted combination of the endpoint outputs, effectively mimicking an ensemble.
• Empirical results show that wide, low-rank ReLU networks achieve high cosine similarity and reciprocal orthogonality across layers, supporting direct weight averaging.
• Temperature calibration is used to adjust logit magnitudes at midpoints, ensuring low-loss paths and preserving classification accuracy in permutation-free merging.

Searching arXiv for the primary paper and closely related work mentioned in the provided data. Search query: "(Ito et al., 9 Oct 2025) Do We Really Need Permutations? Impact of Width Expansion on Linear Mode Connectivity" Layerwise Exponentially Weighted Connectivity (LEWC) is a layerwise property of linearly merged neural networks introduced to explain why sufficiently wide models can exhibit linear mode connectivity (LMC) even without neuron or channel permutations (Ito et al., 9 Oct 2025). In the setting of two independently trained $L$-layer networks with parameters $\bm\theta^{(a)}$ and $\bm\theta^{(b)}$, LEWC states that the output of each layer in the weight-space interpolation $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$ can be written as an exponentially weighted combination of the corresponding layer outputs of the endpoint models. The concept is used to interpret direct weight averaging as an ensemble-like computation in function space, and thereby to account for low-loss linear paths between independently trained solutions in wide ReLU networks (Ito et al., 9 Oct 2025).

1. Formal definition

LEWC is defined for an $L$-layer network whose layer-$\ell$ output is denoted $f_\ell(x;\bm\theta)$, with $f_L$ the logits or output layer. For two models with parameters $\bm\theta^{(a)}$ and $\bm\theta^{(b)}$, and for the merged model

$\bm\theta^{(a)}$0

the models are said to be layerwise exponentially weighted connected if, for every layer $\bm\theta^{(a)}$1 and any $\bm\theta^{(a)}$2,

$\bm\theta^{(a)}$3

This definition appears as Definition 4.1 in "Do We Really Need Permutations? Impact of Width Expansion on Linear Mode Connectivity" (Ito et al., 9 Oct 2025).

The term “exponentially weighted” refers to the dependence of the coefficients on the layer index $\bm\theta^{(a)}$4. At depth $\bm\theta^{(a)}$5, model $\bm\theta^{(a)}$6 contributes with weight $\bm\theta^{(a)}$7 and model $\bm\theta^{(a)}$8 with weight $\bm\theta^{(a)}$9. Near the input, these weights remain close to the global interpolation coefficients $\bm\theta^{(b)}$0 and $\bm\theta^{(b)}$1, whereas near the output they can become very small when $\bm\theta^{(b)}$2 is not close to $\bm\theta^{(b)}$3 or $\bm\theta^{(b)}$4, particularly around $\bm\theta^{(b)}$5. There is no extra per-layer mixing parameter; the depth dependence arises from repeated application of the same global interpolation ratio across layers (Ito et al., 9 Oct 2025).

At the final layer,

$\bm\theta^{(b)}$6

Because $\bm\theta^{(b)}$7 and $\bm\theta^{(b)}$8 may be small for large $\bm\theta^{(b)}$9, especially at midpoints of the path, the paper later interprets this expression after a normalization by $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$0 (Ito et al., 9 Oct 2025).

2. Ensemble interpretation and the connection to linear mode connectivity

The principal significance of LEWC lies in its direct implication for the output logits of the merged model. Under LEWC, the logits satisfy

$$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$1

For classification, the predicted label depends on the $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$2 of the logits and is invariant to multiplication by a positive scalar. The paper therefore introduces the rescaled quantity

$$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$3

which is exactly a logit-level ensemble of the two original models, with weights

$$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$4

Accordingly, the merged model’s logits are proportional to ensemble logits, and the predictions of the merged model coincide with those of the ensemble (Ito et al., 9 Oct 2025).

This establishes the functional link between LEWC and LMC. If LEWC holds along the entire interpolation path $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$5, then the merged model behaves like an ensemble throughout that path. Since the endpoint models are assumed to have low test loss, and since the ensemble is not worse and is usually slightly better, the interpolating path inherits low loss. In the framework of the paper, this is the mechanism by which LEWC explains a low-loss linear path in weight space between independently trained solutions (Ito et al., 9 Oct 2025).

A key qualification is that raw cross-entropy is not invariant to positive rescaling of logits. LEWC can preserve the prediction rule while still reducing logit magnitudes substantially, especially at $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$6 when $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$7 is large. This creates a distinction between preserving classification accuracy and preserving cross-entropy loss. The paper treats this distinction through softmax temperature calibration rather than by modifying the LEWC definition itself (Ito et al., 9 Oct 2025).

3. Temperature calibration and loss barriers

To address the sensitivity of cross-entropy to logit norm, the paper introduces softmax temperature calibration via an inverse temperature $$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$8:

$$\bm\theta^{(c)}(\lambda)=\lambda\bm\theta^{(a)}+(1-\lambda)\bm\theta^{(b)}$$9

The inverse temperature is chosen to minimize cross-entropy on a held-out subset comprising $L$0 of the test data, and is then applied to the remaining $L$1 (Ito et al., 9 Oct 2025).

The role of calibration is especially clear at midpoint interpolation. Under LEWC,

$L$2

This scaling leaves the $L$3 unchanged but can substantially alter cross-entropy by producing logits with very small magnitude. Optimizing $L$4 compensates for this effect by restoring an appropriate effective logit scale. The paper therefore adopts the convention that, in this setting, “LMC holds” includes the case where logits are calibrated by an optimal inverse temperature (Ito et al., 9 Oct 2025).

Empirically, the distinction is consequential. Without calibration, the loss barrier for the merged model remains non-negligible even when accuracy is already high. With the best inverse temperature, the calibrated test loss barrier approaches zero as width increases. This result is central to the paper’s claim that widening plus temperature calibration is sufficient for LMC even in the absence of any permutation alignment (Ito et al., 9 Oct 2025).

A plausible implication is that LEWC should be understood primarily as a statement about layerwise functional composition and logit direction, while temperature scaling restores agreement at the calibration level required by cross-entropy. The paper states this point directly in terms of “logit direction level” versus “logit norm / calibration level” (Ito et al., 9 Oct 2025).

4. Sufficient conditions and the proposed mechanism

The paper proves a sufficient-condition theorem for LEWC in bias-free ReLU networks. The theorem states that if two bias-free models satisfy weak additivity for ReLU activations and reciprocal orthogonality, then the LEWC identity holds for all layers (Ito et al., 9 Oct 2025). The assumptions are: biases are neglected, activations are ReLU, and the two structural conditions hold. The neglect of biases is justified for ResNet and VGG with batch normalization.

The proof proceeds inductively. The base case at $L$5 is trivial. For layer $L$6, one writes

$L$7

uses the inductive form of LEWC for $L$8, removes cross terms such as $L$9 and $\ell$0 via reciprocal orthogonality, and then applies weak additivity to recover

$\ell$1

This theorem supplies the formal bridge from internal representation geometry to LMC (Ito et al., 9 Oct 2025).

The paper then links the emergence of these sufficient conditions to width expansion and low-rank weights. Its conceptual picture is fourfold. First, wider models tend toward low-rank weight matrices, with relative rank decreasing as width grows; stronger weight decay accelerates this tendency, whereas weaker weight decay raises rank and disrupts LEWC and LMC. Second, low-rank weights imply that only a limited low-dimensional subspace is active at each layer, as many coordinates of the pre-activations $\ell$2 have extremely small standard deviation. Third, across independently trained models, this high-dimensional low-rank structure yields weak additivity of ReLU and reciprocal orthogonality. Fourth, high dimensionality itself makes ReLU approximately linear on most directions: Theorem 5.2 states that for high-dimensional Gaussian $\ell$3, the cosine similarity between $\ell$4 and $\ell$5 concentrates around approximately $\ell$6 (Ito et al., 9 Oct 2025).

The paper argues that combining high dimension with low rank improves on this Gaussian baseline. High dimension yields approximate additivity, and low-rank non-overlap of active coordinates across models pushes the cosine similarity closer to $\ell$7. This is presented as the structural mechanism through which width induces LEWC without any explicit alignment procedure (Ito et al., 9 Oct 2025).

5. Layerwise diagnostics and empirical signatures

The empirical evidence for LEWC is organized around layerwise diagnostics of intermediate representations. At $\ell$8, LEWC reduces to

$\ell$9

Since cosine similarity is scale-invariant, the paper tests whether the merged layer output is directionally close to the arithmetic mean of the original layer outputs. For each test input $f_\ell(x;\bm\theta)$0, it computes

$f_\ell(x;\bm\theta)$1

and averages the cosine similarity between $f_\ell(x;\bm\theta)$2 and $f_\ell(x;\bm\theta)$3 over the test data. The reported cosine similarity increases with width at every layer and approaches $f_\ell(x;\bm\theta)$4 at the last layer for sufficiently wide models (Ito et al., 9 Oct 2025).

Weak additivity is examined with an analogous cosine-similarity diagnostic applied to ReLU inputs:

$f_\ell(x;\bm\theta)$5

The paper reports high cosine similarity for all layers, increasing with width and often exceeding the Gaussian-limit value of $f_\ell(x;\bm\theta)$6. This is taken as evidence that the low-rank structure of trained wide networks strengthens the linear-like behavior of ReLU beyond what would be expected from high-dimensional Gaussian inputs alone (Ito et al., 9 Oct 2025).

Reciprocal orthogonality is probed by two diagnostics. The first is the norm ratio

$f_\ell(x;\bm\theta)$7

which should be small when $f_\ell(x;\bm\theta)$8. The paper finds that $f_\ell(x;\bm\theta)$9 decreases with width and is small for all hidden layers. The second diagnostic compares the directions of

$f_L$0

under the expectation that the merged input as seen by $f_L$1 should resemble the original input if cross terms vanish. The cosine similarity again increases with width and approaches $f_L$2 (Ito et al., 9 Oct 2025).

A further layerwise signature concerns variance concentration. For each layer and neuron, the standard deviation of the ReLU input $f_L$3 over the data is computed. The paper reports that most components have very small standard deviation, only a small subset of dimensions are substantially active, the proportion of small-standard-deviation dimensions grows with width, and the overlap of high-variance dimensions between independently trained models decreases with width. This pattern is used to support the interpretation that independently trained wide models occupy largely different active subspaces, reinforcing weak additivity and reciprocal orthogonality (Ito et al., 9 Oct 2025).

6. Width expansion, permutation-free merging, and comparison with related connectivity notions

The paper’s empirical starting point is that, for independently trained MLP, VGG-11, and ResNet-20 models at various width multipliers, simple weight interpolation without any permutation can already produce strong interpolation performance when the models are sufficiently wide (Ito et al., 9 Oct 2025). Accuracy along the interpolation path improves monotonically with the width multiplier, and for sufficiently large width the midpoint accuracy is comparable to that of the endpoint models. Uncalibrated loss still shows a barrier, but calibrated loss approaches zero with increasing width. The paper therefore concludes that widening plus temperature calibration is sufficient for LMC without permutation search (Ito et al., 9 Oct 2025).

This position contrasts with earlier permutation-based accounts of LMC. In those accounts, width is useful because it enlarges the space of candidate permutations, making alignment methods such as weight matching or optimal transport more effective. The paper acknowledges this prior view and notes that such methods often required very wide networks, such as $f_L$4 ResNet-20, before LMC emerged reliably. Against that background, it proposes a distinct mechanism: width itself induces low-rank weights and orthogonality properties that make direct interpolation behave like an ensemble in function space, even in the absence of any matching procedure (Ito et al., 9 Oct 2025).

The paper also places LEWC in relation to Layerwise Linear Feature Connectivity (LLFC), associated with Zhou et al. LLFC states that the merged layer feature is a linear average of features from the original models, contingent on weak additivity and a commutativity condition,

$f_L$5

According to the paper, this commutativity condition is violated in the non-permuted setting, but improves when permutations found by weight matching are applied. LEWC is introduced precisely because LLFC is not the appropriate concept for the widened, non-aligned regime under study (Ito et al., 9 Oct 2025).

An additional empirical observation reinforces this distinction: random permutations before merging do not significantly degrade accuracy for sufficiently wide models. This suggests that, once width is large enough, explicit alignment is not crucial for preserving merged performance. The paper uses this result to argue that the central role of width in its setting is not to facilitate permutation search, but to induce the representational geometry captured by LEWC (Ito et al., 9 Oct 2025).

7. Scope, limitations, and open questions

The empirical scope of LEWC as presently documented is limited to relatively simple datasets and moderate-scale architectures. The experiments are reported on MNIST, Fashion-MNIST, CIFAR-10, and, in the appendix, CIFAR-100, using an MLP with three hidden layers, VGG-11, and ResNet-20 with large width multipliers (Ito et al., 9 Oct 2025). The paper explicitly notes that LEWC-based LMC typically requires larger width multipliers than permutation-based merging, which makes extension to larger architectures such as ImageNet-scale ResNet-50 or transformers more costly. It states that an important direction for future work is to test whether these phenomena persist in large-scale settings and other modalities (Ito et al., 9 Oct 2025).

A second limitation is the dependence on low-rank structure induced by the training regime. The paper reports that strong weight decay, for example $f_L$6, produces relatively low-rank weights, stronger weak additivity and reciprocal orthogonality, high LEWC diagnostic scores, and LMC after temperature calibration. By contrast, weak weight decay, for example $f_L$7, produces higher-rank weights, degrades the layerwise cosine similarity associated with LEWC, causes weak additivity and reciprocal orthogonality to fail, and leaves both accuracy and calibrated loss barriers large (Ito et al., 9 Oct 2025). Width alone is therefore not presented as a universal guarantee; the phenomenon also depends on implicit or explicit regularization that promotes low-rank solutions.

The paper identifies several open questions. One concerns scalability: whether LEWC and width-induced LMC persist in large-scale CNNs, transformers, LLMs, and non-vision domains such as NLP, speech, or multimodal learning. Another concerns theory: whether one can prove, under assumptions on initialization, optimization, and regularization, that wide ReLU networks trained by SGD converge to low-rank, reciprocal-orthogonal solutions so that LEWC holds with high probability. A further question is whether the spectral perspective emphasized here can be unified with spectral observations in permutation-based LMC research. Finally, the relationship between LEWC and LLFC remains open: LEWC is sufficient for LMC in the widened non-aligned regime, whereas LLFC is sufficient in permutation-aligned models, and the precise boundary between these regimes has not been characterized (Ito et al., 9 Oct 2025).

These limitations delimit the current status of LEWC. It is a formal and empirically supported explanation for permutation-free LMC in wide, low-rank ReLU networks, but its generality beyond the architectures, datasets, and regularization regimes studied so far remains an unresolved research question (Ito et al., 9 Oct 2025).