---
title: GATE_S Weight-Sharing Variant
url: https://www.emergentmind.com/topics/gate_s-weight-sharing-variant
type: topic
---

# GATE_S Weight-Sharing Variant

The GATE_S weight-sharing variant refers to a class of neural network architectures that impose explicit parameter-sharing schemes within and across neural layers—particularly in gated recurrent models (e.g., LSTMs, highway networks) and local-receptive-field constructions on graphs—in order to achieve substantial reductions in both memory footprint and computational overhead while retaining expressiveness and accuracy. Distinct implementations of the GATE_S variant are discussed in "Semi-tied Units for Efficient Gating in LSTM and Highway Networks" [1806.06513] and "Learning Local Receptive Fields and their Weight Sharing Scheme on Graphs" [1706.02684]. Both works formalize general methods for dynamically learning the parameter-sharing structure itself (as opposed to imposing static sharing), enabling adaptability to varied topologies and learning tasks.

## 1. Weight-Sharing in Gates: Motivation and Rationale

In conventional gated architectures, subunits such as the input, forget, and output gates in LSTMs, as well as analogous gates in highway networks, each possess independent, full-rank affine transformation matrices. For an LSTM layer with hidden size $H$ and input size $X$, this results in four separate sets of weights: $W_1,U_1,\ldots,W_4,U_4$ and corresponding biases, entailing $O(4XH + 4H^2 + 4H)$ parameters and quadrupling matrix-vector computations per step. As $X$ and $H$ scale or as models deepen, this becomes prohibitively expensive.

The central observation motivating GATE_S is that these affine transformations across the subunits have identical form and thus can be decomposed into a shared affine transformation followed by lightweight, gate-specific, parametric adapters. This decomposition drastically reduces model size and computational cost, while retaining subunit diversity through per-gate scaling operations.

## 2. GATE_S Formulation in Gated Architectures

### 2.1 Shared Affine Transform and Parametric Nonlinearities

A GATE_S (also called semi-tied unit, or STU) gated layer replaces all gate-specific affine mappings with a single shared projection $e_t$:
$$
e_t = W x_t + U h_{t-1} + b
$$
($W \in \mathbb{R}^{H \times X}$, $U \in \mathbb{R}^{H \times H}$, $b \in \mathbb{R}^H$).

Each gate, denoted $g$ (input, forget, output, candidate), then uses a unique pair of vectors $\gamma_g, \eta_g \in \mathbb{R}^H$ to define a parametric nonlinearity:
- Parametric sigmoid: $\sigma_{\eta,\gamma}(a) = \eta \odot \text{sigmoid}(\gamma \odot a)$
- Parametric tanh: $\tanh_{\eta,\gamma}(a) = \eta \odot \tanh(\gamma \odot a)$
- Parametric ReLU: $\text{ReLU}_\eta(a) = \eta \odot \max(a,0)$

The full STU-LSTM equations integrate these as follows (with a shared peephole vector $V$):
\[
\begin{align*}
e_t &= W x_t + U h_{t-1} + b \\
i_t &= \sigma_{\eta_i,\gamma_i}(e_t + V \odot c_{t-1}) \\
f_t &= \sigma_{\eta_f,\gamma_f}(e_t + V \odot c_{t-1}) \\
o_t &= \sigma_{\eta_o,\gamma_o}(e_t + V \odot c_t) \\
\tilde{c}_t &= \tanh_{\eta_c,\gamma_c}(e_t) \\
c_t &= f_t \odot c_{t-1} + i_t \odot \tilde{c}_t \\
h_t &= o_t \odot \tanh(c_t)
\end{align*}
\]
This framework similarly extends to highway networks, with transform and carry gates, as well as candidate activations, all receiving distinct $\gamma$ and $\eta$.

### 2.2 Parameter Count and Computational Complexity

For LSTM layers, standard parameterization requires $4 (XH + H^2 + H)$ parameters. GATE_S reduces this to $(XH + H^2 + H) + 8H + H = XH + H^2 + 10H$, representing approximately a $4\times$ reduction in storage and large matrix-vector multiplications. For highway networks, the reduction achieves a $3\times$ factor, as each component (transform/carry/candidate) is tied to the shared weights with lightweight adapters [1806.06513].

## 3. GATE_S Weight-Sharing on Graphs

The GATE_S variant is generalized to arbitrary graph-structured data in [1706.02684], introducing a learnable, soft-parameterized weight-sharing scheme over local receptive fields. Given an adjacency matrix $A \in \{0,1\}^{n \times n}$ (directed or undirected), the layer maintains:
- A weight pool $W \in \mathbb{R}^{\omega}$ (single-channel) or $W \in \mathbb{R}^{\omega \times p_\text{in} \times p_\text{out}}$ (multi-channel), for filter size $\omega$
- A sharing tensor $S \in \mathbb{R}^{n \times n \times \omega}$, where for each $(i,j)$ edge, $S_{ij:}$ is a soft assignment vector over $\omega$ slots, with $S_{ij:k} \geq 0$ and $\sum_k S_{ij:k} = 1$ when $A_{ij} = 1$

The effective weight matrix for signal propagation is built via:
\[
\Theta_{ij}^{\,\alpha\beta} = \sum_{k=1}^{\omega} S_{ij\,k} W_{k,\alpha,\beta}
\]
so that output features are aggregated as:
\[
y_{i,\beta} = f\!\left( \sum_{j=1}^{n} \sum_{\alpha=1}^{p_\text{in}} \Theta_{ij}^{\,\alpha\beta} x_{j,\alpha} + b_\beta \right)
\]

$S$ is trained jointly with $W$ under convex constraints (simplex projection), enabling the model to learn arbitrary soft parameter-sharing patterns across the graph.

## 4. Implementation Aspects and Optimization Procedures

### 4.1 Initialization and Regularization

In the STU/LSTM case, scaling vectors $\gamma$ and $\eta$ are initialized to 1.0. For graph-based GATE_S, $S$ can be initialized by one-hot assignment (e.g., circulant for grids) or uniform randomization projected onto the simplex per edge.

Regularization employs weight-decay on $W$, and optionally $S$, with standard values ($\lambda_W \approx 10^{-5}$). For stochastic optimization, gradients through the parameterized activations are carefully computed:
- For $\sigma_{\eta,\gamma}(a)_j$, $\frac{\partial \sigma}{\partial a_j} = \eta_j \gamma_j \text{sigmoid}(\gamma_j a_j)(1-\text{sigmoid}(\gamma_j a_j))$

Gradients for shared weights are normalized by the number of subunits (e.g., divided by 4 in LSTM; for recurrent layers, further divided by unroll steps).

### 4.2 Computational Notes

On practical hardware (TensorFlow/CuDNN for graphs), the extra cost of building the composite weight matrix $\Theta$ in the GATE_S graph layer results in runtimes approximately $2$-$2.5\times$ slower than standard matrix-multiplied CNN layers of similar size, due to the flexibility of the learned weight-sharing step [1706.02684].

## 5. Empirical Evaluations

Speech recognition experiments on the British-English MGB dataset demonstrate that STU-LSTM (GATE_S) achieves performance within $0.1$–$0.3$% absolute word error of standard LSTM and highway baselines, while reducing parameter count and computation by $3$-$4\times$. For instance, with a standard LSTM (hidden size $H = 500$, 55h training), WER is $32.2$\%; STU-LSTM matches this with $31.9$\%, using $4\times$ fewer hidden-layer parameters [1806.06513].

In image understanding, experiments on MNIST and CIFAR-10 demonstrate that the GATE_S variant on graphs nearly matches (and in certain graph constructions, exceeds) the accuracy of conventional convolutional or fixed-topology GCN baselines, even when pixel order is scrambled or the feature graph is non-Euclidean. Notably, with the underlying grid known, GATE_S recovers standard convolutional results; with structure unknown, it discovers a near-optimal weight-sharing scheme, thus maintaining or exceeding the performance of vanilla conv, MLP, GCN, and GAT models [1706.02684].

## 6. Relation to Other Models and Generalization

When specialized to grid-structured data and initialized with circulant constraints, the GATE_S scheme exactly reproduces standard convolutions (Toeplitz weight sharing). Unlike GATs, which distribute attention scores over edges, GATE_S imposes explicit, learnable assignment of shared filters across neighborhoods, without reliance on positional or coordinate-based translation definitions, and generalizes to arbitrary graph structures.

The key distinction is the replacement of implicit translation invariance (Euclidean convolutions) with a flexible, explicitly parameterized weight-sharing structure (soft assignment tensor $S$ in graph-based layers, parametric nonlinearities in STUs), enabling the architecture to adapt to non-Euclidean topologies or data where spatial locality or translation symmetry is unknown or irrelevant.

## 7. Summary and Implications

The GATE_S weight-sharing variant offers a principled mechanism to achieve major reductions in storage and computation for both sequence modeling (via semi-tied units in gated architectures) and general graph-structured data (via soft parameter-sharing across arbitrary receptive fields). When the underlying domain supports usual convolutional weight sharing, GATE_S collapses to standard models; when not, it learns problem-specific sharing patterns with minimal loss in accuracy. This suggests that GATE_S architectures are particularly advantageous when model efficiency is crucial or when the underlying topology is unknown or complex [1706.02684, 1806.06513].

Source: https://www.emergentmind.com/topics/gate_s-weight-sharing-variant