---
title: Shared-Weights Extender (SWE)
url: https://www.emergentmind.com/topics/shared-weights-extender-swe
type: topic
---

# Shared-Weights Extender (SWE)

Shared-Weights Extender (SWE) denotes a family of mechanisms that reuse a shared parameter substrate and extend it into concrete model components through coupling, composition, slicing, or lightweight adaptation. In the narrowest sense, SWE is the name of a neural network expansion method that adds neurons by coupling them to already trained neurons so that the new units do not become inactive [2509.18842]. In a broader and increasingly common interpretive sense, the term describes systems that learn a reusable weight template, super-network, learngene, or encoder and then instantiate task- or architecture-specific weights from that shared source, rather than pre-training every target model independently [2601.19694][2312.01274][2601.02092].

## 1. Scope, terminology, and lineage

The literature is not terminologically unified. One strand uses “Shared-Weights Extender” explicitly for network expansion during training [2509.18842]. Several other papers do not introduce the term themselves, but describe mechanisms that are explicitly framed as “a concrete realization” or “a conceptual lens” for a shared-weight extender: a common weight pool is learned once and then reused across layers, depths, widths, clients, or tasks [2601.19694][2312.01274][2404.16897].

Earlier architectural precedents established the core intuition that substantial redundancy exists in repeated structures. ShaResNet shares convolutional weights between residual blocks operating at the same spatial scale, reducing a 152-layer-deep residual network to 106 convolutional layers, with a parameter gain of 39\% while losing less than 0.2\% accuracy on ImageNet [1702.08782]. Multi-Scale Weight Sharing Network reuses one convolutional kernel over multiple dilation rates within the same layer, achieving approximately 25\% fewer parameters while maintaining similar performance on ImageNet and Places365-Standard [2001.02816]. Conditional super-network weights then extended one-shot NAS super-networks by making operation weights depend on predecessor choices, with nearly negligible resource overhead for sequential network designs [2104.11522].

Taken together, these systems suggest a common abstraction: a shared parameter basis is trained under structural constraints, and specialization is deferred to a smaller set of conditional parameters, routing rules, or adaptation steps.

| Representative system | Shared object | Extension target |
|---|---|---|
| SWE | Existing neuron weights plus temporary couplings | Newly added neurons during width expansion |
| SWEET | Low-rank template $\mathcal{G}$ and scalers $(X,U,V)$ | Vision models with varying depth and width |
| SuperWeight Networks | Weight templates and SuperWeights | Layers and architectures under a parameter budget |
| SWS | Stage-wise learngene layers | Variable-depth transformer descendants |
| SuperSFL | Global super-network $\theta$ | Client-specific subnetworks |
| XLM-SWCM | Multilingual encoder attention and FFN weights | Decoder for low-resource generation |

## 2. Explicit SWE for neural network expansion

In its explicit formulation, SWE addresses progressive network growth in trained ReLU networks, where newly inserted neurons often become inactive. The method introduces auxiliary coupling parameters between every existing neuron and each new neuron. During a short adjustment phase, the effective parameters are defined as
$$
\mathbf{w}^{\text{eff}}_{\text{new}} = \mathbf{w}_{\text{new}} + \sum_{i=1}^{n_l} \mathbf{w}_{c,i}, \qquad
b^{\text{eff}}_{\text{new}} = b_{\text{new}} + \sum_{i=1}^{n_l} b_{c,i},
$$
for the new neuron, and
$$
\mathbf{w}^{\text{eff}}_{i} = \mathbf{w}_i - \mathbf{w}_{c,i}, \qquad
b^{\text{eff}}_{i} = b_i - b_{c,i}
$$
for each existing neuron [2509.18842].

This reparameterization makes each existing neuron donate a small, learnable portion of its representation to the new neuron. Only the coupling parameters and the parameters of the next layer are updated during this phase; the existing neurons’ base weights and the new neuron’s base weights are frozen. Because the couplings appear in both the new and old neurons, the coupling gradients remain non-zero as long as either side is active. After one forward-backward pass, the couplings are merged into the base weights and discarded, and training resumes normally. The paper reports that a single forward-backward pass is sufficient to initialize the couplings effectively [2509.18842].

The companion allocator, Steepest Voting Distributor (SVoD), determines where to add neurons in deep networks. It inserts temporary probe neurons with output
$$
h_{\text{probe}}(z) = \phi\big((1+z)u\big),
$$
and uses the first-order sensitivity
$$
\mathcal{L}(z) \approx \mathcal{L}(0) + \frac{\partial \mathcal{L}}{\partial z}\Big|_{z=0} z
$$
to count negative-gradient votes for each layer; layers with more negative probes receive more added neurons [2509.18842].

Empirically, the defining result is inactivity suppression. After adding neurons and training only for 5 epochs, common initializations produced very high fractions of inactive new neurons, whereas SWE reported 0.0\% inactive new neurons in all tested MNIST and FashionMNIST settings. For MNIST \(20 \rightarrow 40\), random initialization yielded 81.7\% inactive neurons, Frobenius-preserving initialization 90.0\%, Firefly 6.7\%, and SWE 0.0\% [2509.18842]. The same paper reports better reconstruction and classification performance than alternative growth methods and baselines across single-hidden-layer and three-hidden-layer MLP settings.

## 3. Template-based and architecture-scalable extenders

A distinct SWE pattern appears in scalable initialization for variable-sized models. SWEET learns a Tucker-factorized shared weight template
$$
\mathcal{W} \Leftarrow \mathcal{G} \times_1 X \times_2 U \times_3 V,
$$
where the low-rank core tensor $\mathcal{G}$ acts as a reusable template and $(X,U,V)$ are size-specific scalers. Instead of pre-training a fixed architecture, SWEET pre-trains this structured template under a masked autoencoding objective and later instantiates target models by composing the frozen template with target-specific scalers. Width-wise stochastic scaling regularizes the template across width-related dimensions and improves cross-width generalization [2601.19694].

The empirical profile of SWEET is unusually broad. It reports state-of-the-art initialization performance across classification, detection, segmentation, and generation. Relative to the strongest baseline, average gains are +1.60\% Top-1 accuracy in classification, +2.04 AP\(^\text{box}\) and +1.59 AP\(^\text{mask}\) in detection, +2.76 mIoU and +3.43 mAcc in segmentation, and FID ↓ 2.19 with IS ↑ 1.76 in image generation [2601.19694]. The same paper emphasizes that the target-specific scalers are often only a few thousand parameters and can be adapted with approximately 0.16 epoch, described as a few hundred iterations.

SuperWeight Networks address neural parameter allocation search through another extender form. Small weight templates are linearly combined into SuperWeights,
$$
\theta^{(k)}_g = \sum_{i=1}^{N} \alpha^{(k)}_i T^g_i,
$$
and layer weights are then built by concatenating one or more SuperWeights. The system learns not only which layers should share templates, but also whether they should share coefficients, using cosine similarity of gradients on shared SuperWeights and on coefficient vectors. On ImageNet at 10\% parameter budget, SWN improves over Shapeshifter Networks from 70.39\% to 71.14\% top-1; in ensemble settings it outperforms deep ensembles with 17\% fewer parameters [2312.01274].

Stage-wise Weight Sharing (SWS) learns a compact transformer learngene through an auxiliary network whose layers are shared within stages, then expands those stage-specific shared layers to initialize variable-depth descendants. The key rule is not mere copying but stage-aware expansion: learned early, middle, and late stage parameters are repeated into corresponding stages of the descendant model using Cyclic Initialization. On ImageNet-1K, SWS reports around 6.6x reduction in total training costs relative to training ten descendant models from scratch, around 20x fewer parameters stored for initialization, and around 10x lower pre-training costs than pre-training and fine-tuning each target model independently [2404.16897].

These systems differ in parameterization, but they share a structural thesis: reusable knowledge is concentrated into a compact shared basis, while the target model is generated or specialized by a smaller, architecture-aware mechanism.

## 4. Resource-heterogeneous and low-resource extensions

In distributed training, the extender idea appears as a shared super-network that is sliced into client-specific subnetworks. SuperSFL defines a global model
$$
\theta = \{\theta_1, \theta_2, \ldots, \theta_L\},
$$
and assigns client \(i\) a contiguous prefix
$$
\theta_i = \{\theta_1, \theta_2, \ldots, \theta_{d_i}\},
$$
with depth \(d_i\) chosen by a resource-aware rule based on memory capacity and communication latency. Three-Phase Gradient Fusion (TPGF) then fuses local and server-side gradients using both depth and inverse-loss weighting before collaborative client-server aggregation updates the shared layers [2601.02092].

The reported systems-level gains are substantial: on CIFAR-10 and CIFAR-100 with up to 100 heterogeneous clients, SuperSFL converges \(2\text{--}5\times\) faster in communication rounds than baseline SFL, with up to \(20\times\) lower total communication cost and \(13\times\) shorter training time [2601.02092]. Here the extender is not generating a new architecture from scratch; it is partitioning one shared parameter space into resource-feasible subnetworks while keeping the shared weights globally coherent.

A second extension regime appears in multilingual generation for extremely low-resource languages. XLM-SWCM starts from the multilingual encoder CINO-base-v2 and reuses encoder self-attention and FFN parameters inside the decoder. CustomDecoderLayers inherit encoder weights for decoder self-attention, cross-attention, and two FFN blocks, while NormalDecoderLayers remain randomly initialized to preserve generation-specific flexibility. The model is then trained with denoising autoencoding and auxiliary machine translation [2502.10852].

This approach uses weight sharing as a transfer mechanism from multilingual understanding to generation. On Tibetan summarization, XLM-SWCM reports ROUGE-L F1 25.7, compared with 16.1 for MC2-LLaMA-13B and 8.6 for mBART-CM; on Tibetan MRC it reaches 16.4 versus 13.2 and 7.9; on Chinese–Tibetan machine translation it reaches 24.5 versus 15.1 and 11.5 [2502.10852]. The ablations identify removal of weight sharing as the largest performance drop among the tested objective and structure changes.

## 5. Acronym ambiguity in software engineering literature

A persistent source of confusion is that many recent arXiv papers use “SWE” to mean **software engineering**, not Shared-Weights Extender. “From SWE-ZERO to SWE-HERO” states explicitly that “SWE” refers to software engineering agents rather than a new model family name, even though its two-stage SFT pipeline can be viewed as extending Qwen2.5-Coder shared weights through 300k execution-free trajectories and 13k execution-based trajectories; the resulting SWE-HERO-32B reaches 62.2\% resolution on SWE-bench Verified and 44.1\% on SWE-bench Multilingual despite Python-only training [2604.01496]. “SWE-Edit” makes the same clarification, explicitly noting that SWE there means software engineering, not Shared-Weights Extender [2604.26102].

The same acronym usage appears in open-weight SWE-agent infrastructure and diagnostic benchmarks. R2E-Gym introduces 8.7K procedurally curated executable environments for open-weight SWE agents and reaches 51\% on SWE-Bench Verified through hybrid execution-based and execution-free verification [2504.07164]. SWE-rebench V2 builds 32,000+ executable tasks spanning 20 languages and 3,600+ repositories, plus 120,000+ additional tasks with installation instructions and fail-to-pass tests [2602.23866]. ORACLE-SWE analyzes oracle information signals such as reproduction tests, regression tests, edit location, execution context, and API usage for SWE agents [2604.07789]. SWE-Interact converts 75 single-turn SWE tasks into user-driven multi-turn sessions and finds that models resolving roughly 50\% of baseline tasks solve only about 25\% of their interactive counterparts [2606.30573].

This ambiguity matters because the software-engineering papers are not primarily about parameter tying. A plausible implication is that “SWE” in contemporary arXiv discourse now names two different research objects: a shared-weight mechanism in model construction, and a benchmark-and-agent ecosystem for software engineering.

## 6. Common patterns, limitations, and research directions

Across the surveyed literature, several constraints recur. The explicit SWE method is evaluated primarily on ReLU MLPs and small CNN-backed heads, and the paper notes that it is not directly analyzed for other activations such as GELU or Swish [2509.18842]. SWEET assumes a ViT-like organization of attention and MLP weights and notes that extremely different architectures, radically different patch sizes, or different positional encodings may require modifying how the unified weight tensor is constructed [2601.19694]. SuperWeight Networks require layer-shape compatibility and depend on similarity thresholds such as \(\tau\) and \(\beta\) for cluster formation and hard-vs-soft sharing decisions [2312.01274]. SWS mainly addresses varying depth with fixed width, and its performance depends on balanced stage design [2404.16897]. SuperSFL assumes contiguous prefix subnetworks and, in its reported form, does not incorporate differential privacy or secure aggregation [2601.02092].

Despite these differences, the surveyed works suggest three durable design patterns. First, the shared component is usually structurally constrained: a low-rank core tensor, a template bank, a stage-wise learngene, a shared encoder, or a global super-network. Second, specialization is cheap relative to full retraining: target-specific scalers, coefficient vectors, replicated stage assignments, client depth slices, or temporary couplings. Third, the best-performing systems often add a mechanism that prevents harmful collapse of the shared representation: width-wise stochastic scaling in SWEET, gradient-similarity clustering in SuperWeight Networks, balanced stage partitioning in SWS, TPGF in SuperSFL, and explicit coupling in SWE [2601.19694][2312.01274][2404.16897][2601.02092][2509.18842].

The forward-looking directions are correspondingly clear in the source papers. The explicit SWE paper proposes extensions toward continual learning and even pruning-by-redistribution [2509.18842]. SWS points to richer distillation schemes, dynamic stage design, and hypernetwork-based extenders [2404.16897]. Conditional super-network weights identify the problem of predicting the optimal splitting epoch without benchmark-side oracle knowledge [2104.11522]. SuperSFL highlights privacy extensions and broader heterogeneity modeling [2601.02092]. This suggests that the central open question is no longer whether shared weights can be extended effectively, but how much structure should remain shared, when specialization should occur, and how to make that specialization stable across domains, scales, and execution regimes.

Source: https://www.emergentmind.com/topics/shared-weights-extender-swe