Hierarchical Multiscale LSTM

• Hierarchical Multiscale LSTM is a recurrent neural network architecture that discovers latent hierarchical structure through adaptive multiscale segmentation.
• It employs discrete boundary variables that enable layers to update, copy, or flush their states, promoting efficient temporal abstraction and improved interpretability.
• Empirical studies demonstrate that HMLSTM lowers bits-per-character and aligns discovered segments with linguistic units, validating its effectiveness over traditional LSTMs.

The Hierarchical Multiscale Long Short-Term Memory (HMLSTM) is a class of recurrent neural architectures developed to discover latent hierarchical structure in sequential data through adaptive multiscale processing. HMLSTM models implement layerwise segmentation and gating mechanisms, allowing each layer to update or copy its recurrent state at different timescales, thus enabling efficient temporal abstraction. Originally introduced by Chung et al. (2016) as the Hierarchical Multiscale Recurrent Neural Network (HM-LSTM), the approach has been analyzed and refined in subsequent studies, most prominently by Kádár et al. (2018), which provided detailed ablation and segmentation analysis (Chung et al., 2016, Kádár et al., 2018).

1. Architecture and Update Dynamics

The HMLSTM consists of an $L$-layer stack, with each layer $\ell$ ($\ell=1 \ldots L$) maintaining its own cell state $c_t^\ell$, hidden state $h_t^\ell$, and a binary boundary variable $z_t^\ell \in \{0,1\}$ at each timestep $t$. These boundary variables act as learnable segment detectors, enabling each layer to decide dynamically when to update, copy, or reset its internal state. The segmentation occurs via the following mechanism:

• Bottom-up input: Each layer receives either the embedded input (layer 1) or the hidden state of the lower layer, gated by the boundary variable of the layer below ($h_t^{\ell-1} \odot z_{t-1}^{\ell-1}$ for $\ell > 1$).
• Top-down input: Gated by the previous boundary variable of the current layer, with $x_t^{\ell+1}$ including $h_t^\ell \odot z_{t-1}^\ell$.

The update at each layer is governed by discretized gating:

• Boundary detection: $z_t^\ell = 1$ if $u_t^\ell > 0.5$, where $u_t^\ell$ is produced by a sigmoid nonlinearity on the input.
• Flush decision: If $z_{t-1}^\ell=1$, the cell state is reset: $c_t^\ell = i_t^\ell \odot g_t^\ell$. Otherwise, the usual LSTM update applies: $$c_t^\ell = f_t^\ell \odot c_{t-1}^\ell + i_t^\ell \odot g_t^\ell$$. The hidden state is always $h_t^\ell = o_t^\ell \odot \tanh(c_t^\ell)$ (Kádár et al., 2018).

The HM-LSTM version formalizes these choices as COPY, UPDATE, and FLUSH operations, conditionally applied based on boundary variables at the current and previous timesteps (Chung et al., 2016).

In HM-LSTM, pre-activation vectors are constructed through affine combinations of recurrent, bottom-up, and top-down signals, followed by nonlinearities and boundary thresholding. The COPY operation preserves information during segments, whereas the FLUSH operation resets the memory upon boundary detection.

2. Training Protocol and Implementation

For language modeling, HMLSTM is commonly trained on the character-level Penn Treebank (PTB) and Text8 datasets. Training procedures include:

• Sequence lengths: 100 for PTB, 80 for Text8.
• Optimization: Stochastic Gradient Descent (SGD) with an initial learning rate of 1.0, halved upon validation loss stall.
• Regularization: Gradient clipping (≤1.0), dropout (0.5 on outputs, 0.2 on embeddings), and input/output embedding tying for marginal gains ($\sim$0.02 bpc on PTB).
• Initialization: Weights uniformly within $[-0.05, 0.05]$; forget-gate bias initialized to +1. Boundary-bias $b_z^\ell$ is set to $\sim$3.0 to encourage initial inactivity (z=0) for stable segmentation emergence.

The crucial discretization of boundary variables is performed using a straight-through estimator for gradient propagation. An auxiliary $z$-loss penalty ($\lambda \cdot \sum_t z_t^\ell$ with $\lambda$ annealed from 1.0 to 0.1 over 50,000 steps) is applied to prevent degenerate boundary allocations (all-0 or all-1) (Kádár et al., 2018). In HM-LSTM, a slope annealing trick for the hard-sigmoid in boundary computation is applied, typically ramping up from 1 to 5 during training (Chung et al., 2016). Layer normalization is not employed in the main HMLSTM replications by Kádár et al., though it was used in some subsequent works.

3. Empirical Performance and Ablation Studies

HMLSTM demonstrates superior character-level language modeling performance compared to standard LSTM baselines, albeit with moderate margins:

• Full HMLSTM on PTB: 1.27 bits per character (bpc)
• 3-layer standard LSTM (matched parameter count): 1.32 bpc
• HMLSTM (with top-down omitted, "no-TD"): 1.30 bpc
• HMLSTM (no flush/reset, "no-flush"): 1.33 bpc
• HMRNN (ReLU RNNs replace LSTMs): 1.38 bpc
• HMLSTM without $z$-loss: collapses to $\sim$1.50 bpc due to degenerate segmentation (Kádár et al., 2018)

On Text8, HMLSTM achieves a gain of $\sim$0.04 bpc over flat LSTM. On the Hutter Prize Wikipedia dataset, it achieves 1.32 bpc. These results establish the architecture as state-of-the-art among RNN models for unsupervised hierarchical processing at the time of publication (Chung et al., 2016).

Ablation findings clarify component importance:

• Top-down connections and the FLUSH mechanism yield modest gains ($0.03{-}0.06$ bpc).
• Multiscale gating via binary boundaries is essential; removing LSTM cells in favor of simple RNNs yields a substantial performance drop (0.11 bpc loss).
• The $z$-loss penalty and correct boundary initialization are vital for non-degenerate segmentation.

The table below summarizes ablation impacts on PTB bpc:

Model Variant	bpc	Δ over Full Model
Full HMLSTM	1.27	0.00
no-TD	1.30	+0.03
no-flush	1.33	+0.06
HMRNN	1.38	+0.11
no-z-loss	~1.50	collapse

4. Segmentation and Hierarchical Structure Discovery

HMLSTM and HM-LSTM architectures were explicitly designed to discover interpretable boundaries in sequence data. Boundary analysis focuses primarily on layer 2 activations, which were aligned with gold token, morpheme, and syntactic chunk boundaries:

• F1-score against word boundaries: $\sim$44%
• F1-score against morpheme boundaries: $\sim$32%
• F1-score against syntactic chunks: $\sim$28%
• Layer-2 boundaries qualitatively align with spaces and punctuation but display high false alarm rates.

Pearson correlation between segmentation quality (word-boundary F1) and language modeling performance (bpc) is weakly negative: $\rho \approx -0.12$. This suggests that alignment with linguistic units does not reliably predict perplexity improvements, and the model can optimize the language modeling objective even when the discovered segments do not coincide with gold annotations (Kádár et al., 2018).

Qualitative visualizations in (Chung et al., 2016) show that in text, layer 1 boundaries (z's) nearly always fire at word ends, layer 2 is sensitive to multi-word or phrasal breaks, and layer 3 often detects sentence or paragraph-level boundaries. In handwriting modeling, discovered boundaries correlate closely with pen-up (stroke ends), revealing an ability to recover domain-relevant hierarchical structure from raw streams.

5. Interpretability, Computational Efficiency, and Practical Considerations

A core advantage of HMLSTM/HM-LSTM is the interpretability of learned boundaries and the efficient allocation of computation across time and hierarchy. For example, in a 270-character PTB sample, layer 1 performs 270 updates, layer 2 only 56, and layer 3 just 9—compared to a flat LSTM requiring 810 updates. The COPY operation preserves memory during intra-chunk steps, preventing information leakage, while the FLUSH operation limits backpropagation depth per segment and potentially improves gradient flow (Chung et al., 2016).

Further, ablation analysis by Kádár et al. demonstrates that simplified versions of HMLSTM, such as omitting top-down feedback or using a smaller flush bias, incur less than 0.03 bpc in performance loss, suggesting that such streamlining yields models that are more practical to implement while preserving most of the original interpretability and capacity for segmentation (Kádár et al., 2018).

6. Limitations, Lessons, and Implications for Representation Learning

Despite improved bpc over standard LSTMs and the emergence of nontrivial segmentation, HMLSTM segment boundaries do not strongly correspond to human-annotated token, morpheme, or syntactic chunk boundaries; nor does improved segmentation F1 translate into better language modeling performance. This indicates a decoupling between representations optimized for prediction and those corresponding to explicit linguistic units.

HMLSTM established the feasibility of learning multi-timescale, interpretable representations in deep recurrent models without supervision. The approach provides a framework for investigating segmentation phenomena in cognitive computational linguistics and for constructing more efficient sequence models that adaptively allocate computation—and may form the basis for future architectures incorporating similar segmental gating or hierarchical state control (Chung et al., 2016, Kádár et al., 2018).