Shared-Weights Architecture

Updated 14 May 2026

Shared-weights architecture is a neural network design pattern that reuses the same parameters across different model parts to reduce redundancy and improve performance.
It achieves significant efficiency by cutting memory and storage needs while preserving, or even enhancing, generalization through parameter tying.
It is implemented via various strategies including layer-wise, block-wise, and head-wise sharing, each balancing model expressiveness with computational stability.

A shared-weights architecture is a neural network design pattern in which one or more sets of learnable weights (parameters) are reused across different parts of the model or across multiple subnetworks. This reuse can be applied across layers, blocks, attention heads, time steps, scales, domains, or even between separate networks (e.g., encoder and decoder). The principal goals of shared-weights architectures are to reduce memory/storage requirements, improve generalization via parameter tying, and exploit inductive structure or redundancy present in the problem domain. Shared-weights designs appear throughout contemporary deep learning models across modalities and tasks, yielding significant theoretical and practical advantages.

1. Core Principles and Mathematical Formulation

In the most generic sense, let $f(x; \theta)$ be a neural network with parameter set $\theta$ . In a shared-weights model, $\theta$ contains parameters that are accessed (via referencing, slicing, or functional mapping) in multiple locations in $f$ . Examples include:

All convolutional layers in CNNs: the same filter $\theta$ applied at every spatial location.
Models leveraging shared supernetworks in neural architecture search: a single parameter tensor $\theta$ is projected/sliced/masked differently for each sampled subarchitecture $a$ via $\theta_a \subseteq \theta$ (Yu et al., 2020).
BERT and related Transformers: layer parameters are sometimes tied across all $L$ layers (phase 1 in share-then-unshare training) as $W^{(l)} = W_{\text{shared}}$ for $\theta$ 0 (Yang et al., 2021).

Mathematically, for a set of modules $\theta$ 1 and a set of shared weights $\theta$ 2, the forward propagation is

$\theta$ 3

where $\theta$ 4 is an (optional) transformation—identity for direct sharing, slicing/masking for subnetwork selection, or other linear/nonlinear mappings.

In weight-shared optimization (as in one-shot NAS), the objective is typically to minimize an average loss over architectures or configurations: $\theta$ 5 where $\theta$ 6 are shared weights, $\theta$ 7 indexes child models, $\theta$ 8 defines the sampling (Yu et al., 2020).

2. Variants and Taxonomy

The literature presents a diverse range of shared-weights strategies:

A. Layer-wise/block-wise sharing.

Entire blocks/layers share identical parameters (e.g., all residual blocks at the same spatial resolution in a ShaResNet share the $\theta$ 9 conv) (Boulch, 2017).

B. Head-wise/attention sharing.

Fine-grained sharing across attention heads, enabling selected heads throughout a transformer to point to the same $\theta$ 0 (query/key) matrices (Cao et al., 2024).

C. Scale-wise/parameterized sharing.

Multi-scale CNNs share filter weights across dilation rates, so the same kernel is applied at different receptive field sizes (Aich et al., 2020).

D. Temporal and recurrence sharing.

Classical RNNs and LSTMs intrinsically use the same weight matrices at each timestep. LiteLSTM further collapses multiple gating matrices into a single shared set (Elsayed et al., 2023).

E. Cross-domain/cross-task sharing.

Domain adaptation often uses partially or softly shared weights across source and target processing streams, with regularization to maintain some alignment but allow domain specificity (Rozantsev et al., 2016).

F. Inter-network/module sharing.

In encoder-decoder architectures, decoder layers may directly reuse (via identity mapping) encoder weights and embeddings, dramatically reducing parameter counts (Su et al., 15 Feb 2025).

G. Low-rank and "delta" sharing.

Block parameters are shared via an "anchor" plus per-block low-rank deltas, as in DeltaLLM, where $\theta$ 1, with $\theta$ 2 low-rank (Mikaelyan et al., 30 Jan 2025).

H. Superweight factorization.

Layers are composed dynamically as linear combinations of global "superweight" templates; layers can share subsets or vary coefficients (Teterwak et al., 2023).

3. Methodologies, Training Algorithms, and Regularization

In neural architecture search (NAS), a shared-weights supernet parameterizes a space of child architectures; each child is a subgraph/subnetwork whose parameters are projected slabs of the supernet weights. The core one-shot recipe:

During each training step, sample one (or several) subarchitectures $\theta$ 3.
Forward/backward pass on $\theta$ 4, updating only the relevant subset of $\theta$ 5.
Optionally, include teacher-student distillation, regularization, or batchnorm adaptation (Yu et al., 2020).

For highly modular architectures (e.g., Transformers), training can begin with full sharing (all layers tied), then at a pre-determined schedule (e.g., after 10% of training), switch to individualized weights per layer, initialized from the shared template, and resume training (Yang et al., 2021). This approach yields rapid convergence (theoretical $\theta$ 6-fold speedups for $\theta$ 7 layers), and empirically boosts final accuracy.

Layer weights are parameterized as $\theta$ 8; only the base $\theta$ 9 is stored densely, each $f$ 0 is low-rank, yielding significant parameter compression with minimal loss if $f$ 1 (Mikaelyan et al., 30 Jan 2025).

Pseudocode Snippet (Supernet Sampling in BigNAS Style)

$f$ 7 (Yu et al., 2020)

Regularization and Initialization

Selective application of dropout and weight decay often only to the largest child model/path to prevent over/underfitting (Yu et al., 2020).
Residual and batchnorm initializations are adapted (e.g., final BN scaling parameter $f$ 2) to stabilize learning under dynamic layer slicing and sharing.
In multi-stream/domain designs, "soft" parameter tying is imposed using an additional regularizer $f$ 3 (Rozantsev et al., 2016).

4. Empirical Impact, Trade-offs, and Representative Results

Compression and Efficiency

Block-wise sharing can yield up to 40% parameter reduction in deep residual networks with <0.2% ImageNet accuracy drop for the deepest ShaResNet variants (Boulch, 2017).
Multi-scale/shared filter CNNs achieve ≈25% parameter savings in ResNet101/152 with no significant loss on ImageNet, and equivalent transfer learning results (Aich et al., 2020).
In LLMs, low-rank delta-based sharing achieves 12–24% parameter reduction while retaining 90% of the original performance; e.g., DeltaPhi-2.9B at 24% reduction matches SlicedPhi 3.3B with $f$ 4M fewer parameters and no finetuning (Mikaelyan et al., 30 Jan 2025).
Encoder-decoder weight reusing in XLM-SWCM yields ~17% global savings and large accuracy gains versus baseline mBART on low-resource summarization and QA (Su et al., 15 Feb 2025).
Head-wise attention parameter sharing in LLMs with a 30% sharing ratio removes ≈9% of total model weights and preserves >99% of reasoning accuracy (Llama 2-7B, DirectShare) (Cao et al., 2024).

Generalization and Regularization

In convolutional deep nets, shared-weight designs can be theoretically shown to radically improve the optimization landscape, turning intractable high-frequency target learning into a poly-time solvable problem by leveraging shared filters for low- and high-frequency harmonics (Shalev-Shwartz et al., 2017).
Shared-weights recurrent designs (LiteLSTM) halve parameter and compute loads while maintaining or slightly exceeding LSTM accuracy across domains (Elsayed et al., 2023).

NAS-Specific Findings

Weight-sharing NAS supernets are extremely compute-efficient but induce high variance in ranking and model selection, due to strong interference among child models via shared parameters. Grouped or partial sharing reduces this variance and improves correlation to ground-truth (Zhang et al., 2020).
Superweight-based NPAS enables hierarchical, gradient-informed parameter reuse, supporting efficient anytime prediction and ensembling with 17% fewer parameters than naive methods for the same accuracy (Teterwak et al., 2023).
Shared normalization-only deep ensembles achieve uncertainty estimates as sharp as full ensembles, but with near-single-model memory/latency costs due to all major weights being shared (Ahmed et al., 2024).

5. Controversies, Limitations, and Open Challenges

Optimization and Stability

In NAS, the main challenge remains the optimization gap between supernet (shared) weights and stand-alone performance—shared-weight proxies can suffer low rank correlation ( $f$ 5– $f$ 6 between supernet and real rankings for full-sharing), leading to unreliable search (Zhang et al., 2020, Xie et al., 2020).
Recent advances, such as orthogonal gradient updates, have partially addressed interference: disturbance-immune weight sharing projects subnet-specific gradients to prevent accretive drift, stabilizing both rewards and rank consistency (Niu et al., 2020).
There is still no global theory for when subarchitectures will interfere destructively under heavy sharing; only partial empirical predictors are available (Xie et al., 2020).

Expressiveness and Fine-grained Coordination

Extremely aggressive sharing (e.g., all layers, all heads in Transformers) inevitably reduces model expressiveness or results in instability, especially on "diverse" tasks or domains (Boulch, 2017, Cao et al., 2024).
In cross-domain adaptation, fully shared weights can force artificial invariances and restrict capture of low-level domain-specific cues. Layer-wise soft tying via light regularization and learned per-layer linear transforms is consistently superior (Rozantsev et al., 2016).
Block repetition and low-rank factorization show that effective "virtual" depth or diversity can be achieved with minimal unique parameter overhead, but per-component analysis is essential; e.g., unsharing certain convolutional subcomponents in speech models yields large WER drops (Hernandez et al., 2023).

Training, Search, and Compute

Bayesian or gradient-driven strategies are needed to determine which layers or modules should be shared. Many approaches still require costly search, clustering, or multi-phase schedule/tuning to maintain the optimal balance between tied and untied parameters (Teterwak et al., 2023, Yang et al., 2021).
Share-then-unshare is limited to systems with highly repetitive modularity (e.g., fixed-depth Transformers).

6. Future Directions and Broader Implications

Advanced methods for parameter selection (learned, gradient-based clustering, or meta-learned sharing masks) promise further efficiency while avoiding the instability/flatness tradeoffs that plague full-sharing (Teterwak et al., 2023).
Fine-grained sharing mechanisms (e.g., head-wise, channel-wise, module-level) combined with regularization, quantization, and low-rank adaptation may supplant traditional pruning and drop-based compressions, especially in LLM and edge-AI settings (Mikaelyan et al., 30 Jan 2025, Cao et al., 2024, Hernandez et al., 2023).
The unification of weight-sharing NAS, meta-learning, and AutoML is anticipated, where a single global superweight bank can serve as a factory for diverse architectures under any compute or deployment constraint (Xie et al., 2020, Teterwak et al., 2023).
Theoretical understanding of the optimization and generalization advantages of shared weights—especially as they relate to frequency spectra and inductive bias—remains an active research area (Shalev-Shwartz et al., 2017).

7. Representative Table: Shared-Weights Architectures Across Domains

Model/Family	Sharing Mechanism	Reported Benefit
BigNAS, OFA-Supernet	Sliced weights for NAS	Universal weights, no retraining
ShaResNet	Block-wise layer sharing	20–40% params cut, <0.2% Δacc.
Conformer Tiny-ASR	Block/module/subcomponent	5M params, SOTA WER
LiteLSTM	Single gate share	~40% less compute, same accuracy
Head-wise LLM attention	Fine (head) sharing	10% param cut, <1% Δ on reasoning
DeltaLLM	Block anchor + low-rank Δ	12–24% smaller, ~90% performance
XLM-SWCM	Encoder↔decoder full share	+199% Summ F1 (low-resource NLU)
Tiny Deep Ensemble	Norm-only share	1%+ better acc, no memory cost

These prime exemplars demonstrate the flexible, efficient, and increasingly principled role played by shared-weights architectures in state-of-the-art machine learning systems.