HyperLSTM: Dynamic Hypernetwork for LSTM

• HyperLSTM is a dynamic hypernetwork that generates context-dependent weights for LSTM cells to improve sequence modeling.
• It employs a small auxiliary LSTM to modulate main LSTM parameters through element-wise scaling, balancing expressiveness with efficiency.
• Empirical benchmarks show enhanced accuracy and lower perplexity across tasks such as language modeling, translation, and handwriting generation.

HyperLSTM is a dynamic hypernetwork architecture that extends the standard long short-term memory (LSTM) model by allowing its weights to vary in a data-dependent manner at each time step. The approach employs a small recurrent neural network—the “hypernetwork” or “HyperLSTM cell”—which generates embedding vectors used to modulate the weights and biases of a main LSTM at every step. Unlike traditional LSTM models that use a fixed set of parameters shared across all time steps, HyperLSTM introduces relaxed weight-sharing, increasing expressiveness while incurring only modest computational and memory cost. This mechanism is trained end-to-end using backpropagation through time, making it compatible with standard deep learning workflows and enabling dynamically adapted parameterization for a variety of sequence modeling tasks (Ha et al., 2016).

1. Motivation and Conceptual Overview

HyperLSTM introduces the abstraction of a hypernetwork—a network that generates some or all parameters for another network, analogous to a genotype-phenotype relationship in biological systems. Specifically, it challenges the fixed weight paradigm in recurrent neural networks (RNNs), exploring whether dynamically generated, context-dependent weights can enhance expressiveness without a prohibitive increase in learnable parameters. The use of a small, auxiliary RNN to influence the weight and bias dynamics of the primary LSTM enables the model to implement a form of context-dependent adaptation, situated between full parameter sharing and full parameter untied per timestep (Ha et al., 2016).

2. Architectural Structure

HyperLSTM Cell (Hypernetwork Core)

The HyperLSTM cell is a small LSTM receiving as input the concatenation $$\hat{\mathbf{x}}_t = [\mathbf{h}_{t-1}; \mathbf{x}_t]$$, where $\mathbf{h}_{t-1}$ is the main LSTM’s previous hidden state and $\mathbf{x}_t$ is the current input. This cell, potentially layer-normalized within its gates, produces a hidden state $\hat{\mathbf{h}}_t \in \mathbb{R}^{\hat N}$.

From $\hat{\mathbf{h}}_t$, the hypernetwork generates for each gate $y \in \{i, f, g, o\}$ of the main LSTM, three embedding vectors:

• $$\mathbf{z}_h^y = W_{\hat h h}^y \hat{\mathbf{h}}_t + \mathbf{b}_{\hat h h}^y$$
• $$\mathbf{z}_x^y = W_{\hat h x}^y \hat{\mathbf{h}}_t + \mathbf{b}_{\hat h x}^y$$
• $\mathbf{z}_b^y = W_{\hat h b}^y \hat{\mathbf{h}}_t$

Main LSTM with Dynamic Parameterization

Base parameters for the main LSTM are:

• Gate weights: $$W_h^y \in \mathbb{R}^{N\times N},\ W_x^y \in \mathbb{R}^{N\times M}$$
• Bias: $\mathbf{h}_{t-1}$0

Dynamic adjustments generated by the hypernetwork include:

• Row-scaling vectors: $\mathbf{h}_{t-1}$1, $\mathbf{h}_{t-1}$2
• Bias correction: $\mathbf{h}_{t-1}$3

The pre-activation for each gate $\mathbf{h}_{t-1}$4 at time $\mathbf{h}_{t-1}$5 is given by: $\mathbf{h}_{t-1}$6 where $\mathbf{h}_{t-1}$7 denotes element-wise multiplication and LN is optional layer normalization.

3. Mathematical Formulation

The standard LSTM cell computes gate pre-activations as: $\mathbf{h}_{t-1}$8 with cell and hidden updates: $\mathbf{h}_{t-1}$9

HyperLSTM dynamically modulates the gate pre-activations using the hypernetwork-generated vectors as described above. Element-wise scaling vectors replace outright generation of full weight matrices to save memory and computation. Layer normalization may be applied to both the hypernetwork LSTM and main LSTM gates. Recurrent dropout is applied to the $\mathbf{x}_t$0 activations as in “Recurrent Dropout without Memory Loss”.

4. Training Regimen and Regularization

The combined system—comprising both hypernetwork and main LSTM—is trained end-to-end using backpropagation through time. Gradients flow through both the hypernetwork and main LSTM parameter sets. Task-specific loss functions are employed:

• Cross-entropy for next-character prediction in language modeling.
• Negative log-likelihood for Mixture Density outputs in handwriting generation.
• Cross-entropy or log-perplexity for neural machine translation (NMT).

Adam optimizer is typically used with learning rates around $\mathbf{x}_t$1, tuned per task. For large-scale NMT, per-step learning rate schedules or mixed Adam+SGD are sometimes utilized.

Regularization strategies include input/output dropout (0.1–0.2), recurrent dropout (~0.1), L2 weight decay ($\mathbf{x}_t$2–$\mathbf{x}_t$3), and optional layer normalization. Gradient clipping (norm capped at 1 or 5) is routinely applied to stabilize learning in the recurrent modules (Ha et al., 2016).

5. Empirical Performance and Benchmarks

HyperLSTM’s sequence modeling capacity has been assessed across language modeling, handwriting generation, and machine translation. Results demonstrate systematic gains over comparable LSTM and Layer Norm LSTM models, while maintaining modest computational overhead.

Task	Baseline Model	Score	HyperLSTM (LN/emb/2-layer)	Score
Penn Treebank (char LM, 1000u)	LSTM	1.312 bpc	HyperLSTM	1.265 bpc
	Layer Norm LSTM	1.267 bpc	HyperLSTM+LN	1.250 bpc
			Larger emb	1.233 bpc
			2-layer	1.219 bpc
enwik8 (char LM, 1800u)	LSTM	1.470 bpc	HyperLSTM	1.391 bpc
	Layer Norm LSTM	1.402 bpc	HyperLSTM+LN	1.353 bpc
			Larger HyperLSTM (2048u)	1.340 bpc
Handwriting Generation (IAM, 900u)	LSTM	-1,055 nats	HyperLSTM	-1,162 nats
	Layer Norm LSTM (1000u)	-1,106 nats
WMT’14 En→Fr NMT (GNMT, single model)	GNMT LSTM	BLEU 38.95	HyperLSTM	BLEU 40.03

HyperLSTM achieves higher accuracy or lower perplexity (and negative log-likelihood) compared to standard or layer-normalized LSTM baselines in all main evaluated tasks. On WMT’14 En→Fr machine translation, HyperLSTM surpasses the GNMT LSTM single model baseline, achieving a BLEU score of 40.03 (Ha et al., 2016).

6. Significance and Implications

The HyperLSTM approach demonstrates that generating parameter modulations on-the-fly via a hypernetwork can enhance the modeling power of recurrent architectures with relatively low computational and memory overhead. The results challenge the fixed-weight paradigm in recurrent neural networks and advocate for the potential of relaxed, context-sensitive parameter sharing strategies. A plausible implication is that such modular, multi-network systems could be extended to other sequence and structured prediction tasks, or further generalized to architectures beyond LSTMs.

Empirical findings establish that dynamic hypernetworks applied to LSTMs can achieve near state-of-the-art or state-of-the-art results across standard sequence modeling benchmarks. These improvements occur without requiring a full generation of all weight tensors, relying instead on elementwise scaling and modest hypernetwork size. This suggests broad applicability for dynamic weight-generation mechanisms in deep sequence models, optimizing capacity versus parameter/compute tradeoffs (Ha et al., 2016).