Weight-Sharing Networks

• Weight-Sharing Networks are neural architectures that enforce identical parameter usage across spatial positions or structural components, reducing model complexity.
• They leverage shared convolution kernels and multi-scale applications to decrease parameter count, improve optimization, and maintain competitive performance.
• Applications in hardware, graph networks, and compression use weight sharing to boost efficiency and reduce resource demand while preserving accuracy.

A weight-sharing network is a neural architecture in which multiple nominal parameters are constrained to take identical values, or are generated from a smaller shared parameterization. In the canonical convolutional setting, the same kernel is reused across spatial positions; later work extends the same principle to sharing across scales within a layer, across candidate architectures in a supernet, across equivalence classes of graph relations, across channel groups, and across repeated hardware mappings. Across these settings, weight sharing functions both as an inductive bias and as a device for reducing parameter count, imposing symmetry, stabilizing optimization, or amortizing computation (Ott et al., 2019, Zhang et al., 2020, Seiffarth, 25 May 2026, Xu et al., 2024).

1. Classical formulation and optimization role

In the standard convolutional formulation, weight sharing means that the same parameter vector is reused at every output position. For a single-channel input $x$ and kernel $w$ with finite spatial support $V$, the convolution is

$(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$

In the multi-channel case, with $C_{\mathrm{in}}$ input channels and $C_{\mathrm{out}}$ output channels, the shared operator is

$$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$

The defining property is that the parameters used at any two spatial locations $u$ and $u'$ are identical. This yields translation-equivariant feature extraction and reduces parameters from a fully local operator to approximately $$K_hK_w\times C_{\mathrm{in}}\times C_{\mathrm{out}}$$, rather than scaling additionally with $w$0 (Ott et al., 2019).

The complementary, non-shared construction is the locally connected or free convolutional operator,

$w$1

with a distinct kernel at each output location. Local connectivity and finite receptive fields are preserved, but parameter tying is removed. In the multi-channel case, the free operator has shape $w$2, making it much larger and slower on digital hardware (Ott et al., 2019).

Weight sharing also has a formal optimization role. In a theoretical study of convolution-like architectures, weight sharing is shown to be crucial to successful optimization for certain target functions with low-frequency structure. The shared parameterization reduces the effective dimensionality from $w$3 parameters to $w$4, eliminates exponentially flat regions that appear in a non-sharing comparator, and can yield a strongly convex objective in the analyzed setting. The same work separates necessity of low-frequency components from sufficiency: low-frequency signal alone is not enough unless the architecture can exploit it through sharing (Shalev-Shwartz et al., 2017).

2. Relaxed, emergent, and multi-scale sharing in vision

Relaxing exact tying does not necessarily destroy recognition performance. Free Convolutional Networks (FCNs), which use position-dependent filters without weight sharing, can match standard CNNs on vision tasks when training includes sufficient translational augmentation. Under translationally augmented data, FCNs learn translationally invariant representations and exhibit an approximate form of weight sharing: filters at different positions become similar, and the average Euclidean distance $w$5 between filters decreases during training, especially at small spatial radius. Translation, rather than rotation or noise, is the augmentation that reliably induces this effect (Ott et al., 2019).

The same literature shows that weight sharing is therefore not strictly necessary for model accuracy in vision tasks, but is a pragmatic optimization and storage strategy. Without augmentation, FCNs overfit more than CNNs. With sufficient random translations, FCNs can approach CNN validation accuracy, and on MNIST the translated validation gap nearly closes at moderate augmentation levels. A plausible implication is that exact tying can be replaced by data-driven emergence of symmetry when the training distribution itself strongly enforces translated views (Ott et al., 2019).

A different generalization of the idea is multi-scale sharing inside a single convolutional layer. In the multi-scale weight-sharing network for image recognition, the same $w$6 kernel is applied at multiple dilation rates within the same bottleneck layer, and the branch outputs are concatenated. On ResNet101 and ResNet152, this scheme reduces total parameters by about $w$7 while maintaining similar Top-1 and Top-5 performance on ImageNet and Places365-Standard. The reported ResNet101 parameter count drops from $w$8 to $w$9, which the authors interpret as evidence of substantial redundancy in deeper networks and of the value of increasing receptive field per parameter (Aich et al., 2020).

Weight sharing can also be learned rather than fixed. In weight-sharing networks with doubly stochastic tensors, a collection of learnable doubly stochastic matrices acts as soft permutation matrices on a canonical weight tensor. Because doubly stochastic matrices are convex combinations of permutation matrices, the method can interpolate between exact group-equivariant sharing and softer, partial symmetry. When the dataset exhibits strong symmetries, the learned matrices are reported to converge toward regular group representations, effectively recovering regular group convolutions; when symmetries are partial, the softer operators remain useful (Linden et al., 2024).

3. Structural sharing in equivariant and graph networks

In equivariant architectures, weight sharing is often formulated as sharing across input pairs that should be treated identically under a symmetry group. For $V$0-equivariant message passing, one formalization defines an equivalence relation on point-pairs,

$V$1

A message function then shares parameters across an entire equivalence class by conditioning only on class-identifying invariant attributes:

$V$2

On homogeneous spaces such as $V$3, $V$4, and $V$5, these attributes recover group-convolution kernel constraints while remaining compatible with sparse message passing. The position–orientation space $V$6 is presented as a middle ground: more expressive than isotropic $V$7 methods, but less expensive than full $V$8 lifting (Bekkers et al., 2023).

Invariant-based weight sharing for message passing pushes the idea further by indexing weights directly with user-chosen graph invariants. In ShareGNN, a node-pair signature is

$V$9

where $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$0 is a node label and $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$1 is shortest-path distance. The learned adjacency is then assembled from invariant-indexed scalars,

$(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$2

and the layer update becomes

$(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$3

This construction is permutation aware by design, offers explicit control over model complexity through the chosen invariant set, and is proved to have discriminative power at least as strong as the chosen invariants. The reported experiments include consistent improvements over standard MPNNs, competitive behavior beyond the 1-WL test on selected tasks, and strong results on subgraph counting and ZINC regression (Seiffarth, 25 May 2026).

These graph and equivariant formulations differ from classical convolutional tying in one crucial respect. The reuse pattern is not attached to a fixed grid alone; it is induced by equivalence classes of relations, distances, orientations, or structural signatures. This suggests that weight sharing can be viewed more generally as a mechanism for quotienting the parameter space by a symmetry or invariant structure, whether that structure is spatial, geometric, or combinatorial.

4. Supernets and architecture search

In neural architecture search, a weight-sharing network usually means a supernet: a single over-parameterized network that contains an entire search space of candidate architectures as subgraphs. If $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$4 denotes an architecture and $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$5 the shared weights, the supernet is trained to minimize

$(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$6

typically by sampling one architecture per minibatch and updating only the active path. This makes architecture evaluation far cheaper than training each candidate from scratch, but introduces a mismatch between supernet optimization and the true NAS objective (Zhang et al., 2020).

The central empirical finding is that ranking fidelity is unstable and search-space dependent. On a controlled 64-architecture CIFAR-10 search space, GT-Tau at epoch 200 is reported as $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$7 across different seeds, and the best architecture under shared weights can vary substantially across runs and even across consecutive epochs. Across larger benchmarks, the behavior remains heterogeneous: Spearman $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$8 reaches about $(x * w)(u)=\sum_{v\in V} w(v)\,x(u+v).$9 on NAS-Bench-201, about $C_{\mathrm{in}}$0 to $C_{\mathrm{in}}$1 on NAS-Bench-101, declines during prolonged training on DARTS-CIFAR10, and can be non-positive on DARTS-PTB. These studies attribute the variance to interference among child models, operator preference bias, model-size bias, BatchNorm effects, and the basic fact that inherited weights are compromise solutions rather than per-architecture optima (Zhang et al., 2020, Zhang et al., 2020).

Subsequent work treats supernet training itself as the main object of analysis. A large heuristic study over NAS-Bench-101, NAS-Bench-201, and DARTS-NDS shows that proper BatchNorm handling, lower learning rates than in stand-alone training, sufficiently long schedules, and careful treatment of dynamic channels materially improve ranking correlation, while dropout, OFA-style super-kernels, and several commonly used shortcuts degrade it. The same study introduces sparse Kendall-Tau as a more informative fidelity metric than raw supernet accuracy and reports that, once the supernet is properly trained, simple random search becomes competitive with far more elaborate NAS procedures (Yu et al., 2021).

Several methods attempt to reduce or correct the noise introduced by sharing. Group sharing and prefix sharing reduce interference by limiting which architectures reuse the same weights; in width search, CafeNet uses locally free channel sharing and FLOPs-sensitive bins to make widths more distinguishable while preserving one-shot evaluation efficiency (Su et al., 2021). Another line models the noisy supernet landscape directly: a graph convolutional network is trained on sampled architectures and their inherited accuracies to suppress random estimation error and improve the ranking of candidates, especially in MobileNetV2-like search spaces (Chen et al., 2020).

5. Compression-oriented weight sharing

Compression research treats weight sharing as a replacement of many distinct parameters by a small codebook of shared values. In the most general formulation, ArbNets define a hash or assignment function $C_{\mathrm{in}}$2 from parameter indices to shared buckets,

$C_{\mathrm{in}}$3

so that the gradient on a shared parameter is the sum of gradients over all tied entries:

$C_{\mathrm{in}}$4

Within this framework, standard MLPs correspond to the identity hash, CNNs correspond to deterministic balanced hashes induced by Toeplitz structure, and RNNs correspond to sharing across time. Two explicit hash families are studied: a Dirichlet hash that controls balance through the entropy of bucket occupancies, and a Neighborhood hash that controls stochasticity while preserving expected balance. The reported result is that balanced and deterministic sharing improves test accuracy relative to more random or unbalanced assignments (Chang et al., 2023).

Bayesian formulations reinterpret weight sharing as stochastic parameter tying. In stochastic weight sharing for Bayesian neural networks, each variational weight is represented by its posterior mean and variance, and inlier weights are clustered in the $C_{\mathrm{in}}$5 plane by a small Gaussian mixture model. Inlier weights then share stochastic representatives, while outliers retain individual Gaussian posteriors. The method uses Wasserstein-based merging and alpha blending between clusters, and is reported to scale Bayesian training to large models such as ResNet-101 and ViT while compressing parameters by approximately $C_{\mathrm{in}}$6 and achieving compression rates around $C_{\mathrm{in}}$7–$C_{\mathrm{in}}$8 in several tabulated settings (Lin et al., 23 May 2025).

A structure-agnostic multi-objective compression approach uses a single global codebook over all layers, uniformly sized bins, and a Pareto search over validation loss and memory cost. The uniform quantization step has $C_{\mathrm{in}}$9 complexity, in contrast to the $C_{\mathrm{out}}$0 complexity of k-means-based non-uniform quantization, and no retraining of shared weights is performed. Reported compression ratios are $C_{\mathrm{out}}$1 on CIFAR-10, $C_{\mathrm{out}}$2 on CIFAR-100, and $C_{\mathrm{out}}$3 on ImageNet (Khosrowshahli et al., 6 Jan 2025).

Across these compression-oriented formulations, the notion of a weight-sharing network shifts from architectural prior to post hoc parameter economy. The shared entities are not necessarily spatial kernels or symmetry classes; they may be codebook entries, hash buckets, or mixture components.

6. Hardware and systems realizations

In hardware-oriented work, weight sharing is often motivated less by statistical regularization than by the cost of storing, programming, or updating large parameter sets. In a hardware-friendly unsupervised memristive neural network, all synapses associated with black pixels and the same output category share one weight, and all synapses associated with white pixels and the same category share another. For a task with $C_{\mathrm{out}}$4 categories, this reduces the input-to-output parameter count to $C_{\mathrm{out}}$5 shared weights, independent of the number of input pixels. The reported implementation attains 743 correct classifications out of 750 noisy test images on a $C_{\mathrm{out}}$6 alphabet task, i.e. $C_{\mathrm{out}}$7 accuracy, while greatly reducing logic usage and increasing clock frequency relative to earlier FPGA realizations (Tang et al., 2019).

Optical neural networks on micro-ring resonator arrays introduce a different systems bottleneck: repeated programming and calibration of small photonic weight banks. The reuse-and-blend architecture addresses this by reusing a programmed matrix across layers or blocks and altering its effective action through optical transpose and electronic shuffle. In this context, weight sharing is temporal and layer-wise rather than spatial. The reported system maintains comparable accuracy while achieving about $C_{\mathrm{out}}$8 energy savings and $C_{\mathrm{out}}$9 latency improvement, and reduces the need for frequent MRR rewriting by holding a basic weight block over a reuse window (Xu et al., 2024).

These hardware realizations broaden the meaning of weight sharing beyond parameter tying in software training. The shared object may be a synaptic state register, a photonic matrix held across several logical layers, or any reusable hardware-programmed block whose repeated use lowers memory traffic, calibration overhead, or device wear.

7. Selective and domain-specific sharing

Continual learning introduces the question of not only whether to share, but which weights to share. In a pruning-based continual-learning framework, each task obtains a structured subnetwork and previously learned subnetworks can be frozen and shared as feature extractors for later tasks. The evidence shows that sharing all prior weights is often sub-optimal: on the MPC six-task setting with ResNet-18, mean accuracy is $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$0 for share-all, $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$1 for share-none, and $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$2 for a manual policy that shares according to task similarity. The paper relates effective sharing to task similarity and task complexity, and proposes connectivity-based criteria for deciding which subnetworks to reuse (Andle et al., 2023).

Channel-wise sharing can also be injected into attention-based image models. In hypercomplex-inspired axial-attention networks, quaternion and PHM layers enforce cross-channel tying so that each $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$3 channel-mixing block is determined by four parameters rather than sixteen. Replacing the stem, bottleneck projections, or classifier with these representationally coherent variants improves classification on ImageNet300k. In the reported RepAA configuration, validation accuracy reaches $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$4 with $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$5 parameters, compared with $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$6 for the baseline Axial-ResNet-50 at $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$7 parameters (Shahadat et al., 2021).

Application-specific sharing can encode physical structure directly. In three-dimensional turbulent-flow reconstruction from sparse planar measurements, a weight-sharing network applies the same 2D subnetwork to every $$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$8–$$y_{c^{\mathrm{out}}}(u)=\sum_{c^{\mathrm{in}}=1}^{C_{\mathrm{in}}}\sum_{v\in V} w_{c^{\mathrm{out}},c^{\mathrm{in}}}(v)\,x_{c^{\mathrm{in}}}(u+v).$$9 slice along a homogeneous $u$0 direction, then refines the stacked volume with 3D convolutions. This architectural tying reduces the parameter count to 271,805, compared with 134,255,824 for the adapted PC-DualConvNet baseline, and yields better generalization under noisy measurements: validation sensor loss decreases with training sensor loss for the shared model but not for the baseline (Mo et al., 23 Sep 2025).

Taken together, these application-driven studies show that weight-sharing networks are not defined by a single canonical construction. They are defined by a design decision: identifying transformations, relations, channels, tasks, or hardware contexts that should reuse a common parameterization, and then encoding that decision directly into the network. The resulting models trade raw flexibility for structure, but the structure can serve very different ends: symmetry, efficiency, robustness, transfer, or physical plausibility.