Shared-Weights Extender (SWE)

• Shared-Weights Extender (SWE) is a method that reuses a common weight template by coupling new neurons with pre-trained ones, preventing inactivity.
• The approach, along with variants like SWEET and SuperWeight Networks, applies structural constraints and lightweight adaptations to tailor models for specific tasks or architectures.
• Empirical studies highlight SWE's effectiveness in reducing inactive neurons and improving performance across applications such as image classification, segmentation, and multilingual generation.

Shared-Weights Extender (SWE) denotes a family of mechanisms that reuse a shared parameter substrate and extend it into concrete model components through coupling, composition, slicing, or lightweight adaptation. In the narrowest sense, SWE is the name of a neural network expansion method that adds neurons by coupling them to already trained neurons so that the new units do not become inactive (Chatzis et al., 23 Sep 2025). In a broader and increasingly common interpretive sense, the term describes systems that learn a reusable weight template, super-network, learngene, or encoder and then instantiate task- or architecture-specific weights from that shared source, rather than pre-training every target model independently (Xie et al., 27 Jan 2026, Teterwak et al., 2023, Asif et al., 5 Jan 2026).

1. Scope, terminology, and lineage

The literature is not terminologically unified. One strand uses “Shared-Weights Extender” explicitly for network expansion during training (Chatzis et al., 23 Sep 2025). Several other papers do not introduce the term themselves, but describe mechanisms that are explicitly framed as “a concrete realization” or “a conceptual lens” for a shared-weight extender: a common weight pool is learned once and then reused across layers, depths, widths, clients, or tasks (Xie et al., 27 Jan 2026, Teterwak et al., 2023, Xia et al., 2024).

Earlier architectural precedents established the core intuition that substantial redundancy exists in repeated structures. ShaResNet shares convolutional weights between residual blocks operating at the same spatial scale, reducing a 152-layer-deep residual network to 106 convolutional layers, with a parameter gain of 39\% while losing less than 0.2\% accuracy on ImageNet (Boulch, 2017). Multi-Scale Weight Sharing Network reuses one convolutional kernel over multiple dilation rates within the same layer, achieving approximately 25\% fewer parameters while maintaining similar performance on ImageNet and Places365-Standard (Aich et al., 2020). Conditional super-network weights then extended one-shot NAS super-networks by making operation weights depend on predecessor choices, with nearly negligible resource overhead for sequential network designs (Laube et al., 2021).

Taken together, these systems suggest a common abstraction: a shared parameter basis is trained under structural constraints, and specialization is deferred to a smaller set of conditional parameters, routing rules, or adaptation steps.

Representative system	Shared object	Extension target
SWE	Existing neuron weights plus temporary couplings	Newly added neurons during width expansion
SWEET	Low-rank template $\mathcal{G}$ and scalers $(X,U,V)$	Vision models with varying depth and width
SuperWeight Networks	Weight templates and SuperWeights	Layers and architectures under a parameter budget
SWS	Stage-wise learngene layers	Variable-depth transformer descendants
SuperSFL	Global super-network $\theta$	Client-specific subnetworks
XLM-SWCM	Multilingual encoder attention and FFN weights	Decoder for low-resource generation

2. Explicit SWE for neural network expansion

In its explicit formulation, SWE addresses progressive network growth in trained ReLU networks, where newly inserted neurons often become inactive. The method introduces auxiliary coupling parameters between every existing neuron and each new neuron. During a short adjustment phase, the effective parameters are defined as

$$\mathbf{w}^{\text{eff}}_{\text{new}} = \mathbf{w}_{\text{new}} + \sum_{i=1}^{n_l} \mathbf{w}_{c,i}, \qquad b^{\text{eff}}_{\text{new}} = b_{\text{new}} + \sum_{i=1}^{n_l} b_{c,i},$$

for the new neuron, and

$$\mathbf{w}^{\text{eff}}_{i} = \mathbf{w}_i - \mathbf{w}_{c,i}, \qquad b^{\text{eff}}_{i} = b_i - b_{c,i}$$

for each existing neuron (Chatzis et al., 23 Sep 2025).

This reparameterization makes each existing neuron donate a small, learnable portion of its representation to the new neuron. Only the coupling parameters and the parameters of the next layer are updated during this phase; the existing neurons’ base weights and the new neuron’s base weights are frozen. Because the couplings appear in both the new and old neurons, the coupling gradients remain non-zero as long as either side is active. After one forward-backward pass, the couplings are merged into the base weights and discarded, and training resumes normally. The paper reports that a single forward-backward pass is sufficient to initialize the couplings effectively (Chatzis et al., 23 Sep 2025).

The companion allocator, Steepest Voting Distributor (SVoD), determines where to add neurons in deep networks. It inserts temporary probe neurons with output

$h_{\text{probe}}(z) = \phi\big((1+z)u\big),$

and uses the first-order sensitivity

$$\mathcal{L}(z) \approx \mathcal{L}(0) + \frac{\partial \mathcal{L}}{\partial z}\Big|_{z=0} z$$

to count negative-gradient votes for each layer; layers with more negative probes receive more added neurons (Chatzis et al., 23 Sep 2025).

Empirically, the defining result is inactivity suppression. After adding neurons and training only for 5 epochs, common initializations produced very high fractions of inactive new neurons, whereas SWE reported 0.0\% inactive new neurons in all tested MNIST and FashionMNIST settings. For MNIST $20 \rightarrow 40$, random initialization yielded 81.7\% inactive neurons, Frobenius-preserving initialization 90.0\%, Firefly 6.7\%, and SWE 0.0\% (Chatzis et al., 23 Sep 2025). The same paper reports better reconstruction and classification performance than alternative growth methods and baselines across single-hidden-layer and three-hidden-layer MLP settings.

3. Template-based and architecture-scalable extenders

A distinct SWE pattern appears in scalable initialization for variable-sized models. SWEET learns a Tucker-factorized shared weight template

$$\mathcal{W} \Leftarrow \mathcal{G} \times_1 X \times_2 U \times_3 V,$$

where the low-rank core tensor $\mathcal{G}$ acts as a reusable template and $(X,U,V)$0 are size-specific scalers. Instead of pre-training a fixed architecture, SWEET pre-trains this structured template under a masked autoencoding objective and later instantiates target models by composing the frozen template with target-specific scalers. Width-wise stochastic scaling regularizes the template across width-related dimensions and improves cross-width generalization (Xie et al., 27 Jan 2026).

The empirical profile of SWEET is unusually broad. It reports state-of-the-art initialization performance across classification, detection, segmentation, and generation. Relative to the strongest baseline, average gains are +1.60\% Top-1 accuracy in classification, +2.04 AP$(X,U,V)$1 and +1.59 AP$(X,U,V)$2 in detection, +2.76 mIoU and +3.43 mAcc in segmentation, and FID ↓ 2.19 with IS ↑ 1.76 in image generation (Xie et al., 27 Jan 2026). The same paper emphasizes that the target-specific scalers are often only a few thousand parameters and can be adapted with approximately 0.16 epoch, described as a few hundred iterations.

SuperWeight Networks address neural parameter allocation search through another extender form. Small weight templates are linearly combined into SuperWeights,

$(X,U,V)$3

and layer weights are then built by concatenating one or more SuperWeights. The system learns not only which layers should share templates, but also whether they should share coefficients, using cosine similarity of gradients on shared SuperWeights and on coefficient vectors. On ImageNet at 10\% parameter budget, SWN improves over Shapeshifter Networks from 70.39\% to 71.14\% top-1; in ensemble settings it outperforms deep ensembles with 17\% fewer parameters (Teterwak et al., 2023).

Stage-wise Weight Sharing (SWS) learns a compact transformer learngene through an auxiliary network whose layers are shared within stages, then expands those stage-specific shared layers to initialize variable-depth descendants. The key rule is not mere copying but stage-aware expansion: learned early, middle, and late stage parameters are repeated into corresponding stages of the descendant model using Cyclic Initialization. On ImageNet-1K, SWS reports around 6.6x reduction in total training costs relative to training ten descendant models from scratch, around 20x fewer parameters stored for initialization, and around 10x lower pre-training costs than pre-training and fine-tuning each target model independently (Xia et al., 2024).

These systems differ in parameterization, but they share a structural thesis: reusable knowledge is concentrated into a compact shared basis, while the target model is generated or specialized by a smaller, architecture-aware mechanism.

4. Resource-heterogeneous and low-resource extensions

In distributed training, the extender idea appears as a shared super-network that is sliced into client-specific subnetworks. SuperSFL defines a global model

$(X,U,V)$4

and assigns client $(X,U,V)$5 a contiguous prefix

$(X,U,V)$6

with depth $(X,U,V)$7 chosen by a resource-aware rule based on memory capacity and communication latency. Three-Phase Gradient Fusion (TPGF) then fuses local and server-side gradients using both depth and inverse-loss weighting before collaborative client-server aggregation updates the shared layers (Asif et al., 5 Jan 2026).

The reported systems-level gains are substantial: on CIFAR-10 and CIFAR-100 with up to 100 heterogeneous clients, SuperSFL converges $(X,U,V)$8 faster in communication rounds than baseline SFL, with up to $(X,U,V)$9 lower total communication cost and $\theta$0 shorter training time (Asif et al., 5 Jan 2026). Here the extender is not generating a new architecture from scratch; it is partitioning one shared parameter space into resource-feasible subnetworks while keeping the shared weights globally coherent.

A second extension regime appears in multilingual generation for extremely low-resource languages. XLM-SWCM starts from the multilingual encoder CINO-base-v2 and reuses encoder self-attention and FFN parameters inside the decoder. CustomDecoderLayers inherit encoder weights for decoder self-attention, cross-attention, and two FFN blocks, while NormalDecoderLayers remain randomly initialized to preserve generation-specific flexibility. The model is then trained with denoising autoencoding and auxiliary machine translation (Su et al., 15 Feb 2025).

This approach uses weight sharing as a transfer mechanism from multilingual understanding to generation. On Tibetan summarization, XLM-SWCM reports ROUGE-L F1 25.7, compared with 16.1 for MC2-LLaMA-13B and 8.6 for mBART-CM; on Tibetan MRC it reaches 16.4 versus 13.2 and 7.9; on Chinese–Tibetan machine translation it reaches 24.5 versus 15.1 and 11.5 (Su et al., 15 Feb 2025). The ablations identify removal of weight sharing as the largest performance drop among the tested objective and structure changes.

5. Acronym ambiguity in software engineering literature

A persistent source of confusion is that many recent arXiv papers use “SWE” to mean software engineering, not Shared-Weights Extender. “From SWE-ZERO to SWE-HERO” states explicitly that “SWE” refers to software engineering agents rather than a new model family name, even though its two-stage SFT pipeline can be viewed as extending Qwen2.5-Coder shared weights through 300k execution-free trajectories and 13k execution-based trajectories; the resulting SWE-HERO-32B reaches 62.2\% resolution on SWE-bench Verified and 44.1\% on SWE-bench Multilingual despite Python-only training (Ludwig et al., 2 Apr 2026). “SWE-Edit” makes the same clarification, explicitly noting that SWE there means software engineering, not Shared-Weights Extender (Zhang et al., 28 Apr 2026).

The same acronym usage appears in open-weight SWE-agent infrastructure and diagnostic benchmarks. R2E-Gym introduces 8.7K procedurally curated executable environments for open-weight SWE agents and reaches 51\% on SWE-Bench Verified through hybrid execution-based and execution-free verification (Jain et al., 9 Apr 2025). SWE-rebench V2 builds 32,000+ executable tasks spanning 20 languages and 3,600+ repositories, plus 120,000+ additional tasks with installation instructions and fail-to-pass tests (Badertdinov et al., 27 Feb 2026). ORACLE-SWE analyzes oracle information signals such as reproduction tests, regression tests, edit location, execution context, and API usage for SWE agents (Li et al., 9 Apr 2026). SWE-Interact converts 75 single-turn SWE tasks into user-driven multi-turn sessions and finds that models resolving roughly 50\% of baseline tasks solve only about 25\% of their interactive counterparts (Raghavendra et al., 29 Jun 2026).

This ambiguity matters because the software-engineering papers are not primarily about parameter tying. A plausible implication is that “SWE” in contemporary arXiv discourse now names two different research objects: a shared-weight mechanism in model construction, and a benchmark-and-agent ecosystem for software engineering.

6. Common patterns, limitations, and research directions

Across the surveyed literature, several constraints recur. The explicit SWE method is evaluated primarily on ReLU MLPs and small CNN-backed heads, and the paper notes that it is not directly analyzed for other activations such as GELU or Swish (Chatzis et al., 23 Sep 2025). SWEET assumes a ViT-like organization of attention and MLP weights and notes that extremely different architectures, radically different patch sizes, or different positional encodings may require modifying how the unified weight tensor is constructed (Xie et al., 27 Jan 2026). SuperWeight Networks require layer-shape compatibility and depend on similarity thresholds such as $\theta$1 and $\theta$2 for cluster formation and hard-vs-soft sharing decisions (Teterwak et al., 2023). SWS mainly addresses varying depth with fixed width, and its performance depends on balanced stage design (Xia et al., 2024). SuperSFL assumes contiguous prefix subnetworks and, in its reported form, does not incorporate differential privacy or secure aggregation (Asif et al., 5 Jan 2026).

Despite these differences, the surveyed works suggest three durable design patterns. First, the shared component is usually structurally constrained: a low-rank core tensor, a template bank, a stage-wise learngene, a shared encoder, or a global super-network. Second, specialization is cheap relative to full retraining: target-specific scalers, coefficient vectors, replicated stage assignments, client depth slices, or temporary couplings. Third, the best-performing systems often add a mechanism that prevents harmful collapse of the shared representation: width-wise stochastic scaling in SWEET, gradient-similarity clustering in SuperWeight Networks, balanced stage partitioning in SWS, TPGF in SuperSFL, and explicit coupling in SWE (Xie et al., 27 Jan 2026, Teterwak et al., 2023, Xia et al., 2024, Asif et al., 5 Jan 2026, Chatzis et al., 23 Sep 2025).

The forward-looking directions are correspondingly clear in the source papers. The explicit SWE paper proposes extensions toward continual learning and even pruning-by-redistribution (Chatzis et al., 23 Sep 2025). SWS points to richer distillation schemes, dynamic stage design, and hypernetwork-based extenders (Xia et al., 2024). Conditional super-network weights identify the problem of predicting the optimal splitting epoch without benchmark-side oracle knowledge (Laube et al., 2021). SuperSFL highlights privacy extensions and broader heterogeneity modeling (Asif et al., 5 Jan 2026). This suggests that the central open question is no longer whether shared weights can be extended effectively, but how much structure should remain shared, when specialization should occur, and how to make that specialization stable across domains, scales, and execution regimes.