Hierarchical Bi-GRU with Multi-Level Attention

• The paper introduces a Hierarchical Bi-GRU with Multi-Level Attention mechanism that integrates bidirectional GRUs and layered attention outputs through learnable weights.
• Empirical results demonstrate notable improvements in tasks like reading comprehension, document classification, and sequence generation compared to conventional attention models.
• The architecture leverages norm-boundedness and monotonic convergence properties, ensuring improved representation generalization and stable training.

A Hierarchical Bi-GRU with Multi-Level Attention is a neural architecture that organizes sequential modeling at multiple granularity levels and enhances it with structured attention mechanisms, enabling the model to capture and leverage information from various hierarchical depths within language data. This approach integrates bidirectional gated recurrent units (Bi-GRUs) for contextual encoding and applies attention mechanisms at either multiple structural levels (e.g., word, sentence, utterance) or network depths, with a focus on aggregating their outputs for improved representation and generalization.

1. Motivation and Key Principles

Conventional attention mechanisms, such as vanilla attention (VAM), typically capture only low-level interactions with a single pass over the encoded sequence. Stackings of multiple attention layers can extract deeper feature representations, but many models use only the final layer’s outputs, thereby discarding informative intermediate contexts. This restricts the model from simultaneously exploiting low-, mid-, and high-level features that are often critical for complex sequence understanding tasks.

The introduction of a hierarchical scheme—such as the Hierarchical Attention Mechanism (Ham)—overcomes these limitations by computing attention outputs at each depth and combining them via a learned, task-adaptive weighted sum. This approach permits finer control over representational granularity and enhances generalization by retaining information from each depth (Dou et al., 2018).

2. Mathematical Formulation

Formally, let $$H = [h_1, h_2, ..., h_n] \in \mathbb{R}^{d_h \times n}$$ denote the sequence of encoder states from a Bi-GRU. For multi-level (depth-$L$) attention, the mechanism operates as follows:

For $\ell = 1 \ldots L$:

• Compute level-$\ell$ alignment:

$$e_i^{(\ell)} = {u^{(\ell)}}^\top h_i,\quad a_i^{(\ell)} = \frac{\exp(e_i^{(\ell)})}{\sum_{j=1}^n \exp(e_j^{(\ell)})}$$

• Form context vector:

$c^{(\ell)} = \sum_{i=1}^n a_i^{(\ell)} h_i$

• Optionally update the query for the next level:

$u^{(\ell+1)} = W_u [u^{(\ell)}; c^{(\ell)}] + b_u$

Trainable scalar weights $w_1,\dots,w_L$ (softmax-normalized to $\alpha_1,\ldots,\alpha_L$) determine the influence of each context vector: $$c_{\text{Ham}} = \sum_{\ell=1}^L \alpha_\ell c^{(\ell)}$$ where $$\alpha_\ell = \frac{\exp(w_\ell)}{\sum_{j=1}^L \exp(w_j)}$$ (Dou et al., 2018).

This flexible weighted summation mechanism allows the model to choose, adaptively and via training, which depth(s) of attention deliver the most relevant features for a given task.

3. Integration with Bidirectional GRUs

The bidirectional GRU encoder encodes sequential data with forward and backward passes: $$\overrightarrow{h}_t = \text{GRU}_{\text{fw}}(x_t, \overrightarrow{h}_{t-1}),\quad \overleftarrow{h}_t = \text{GRU}_{\text{bw}}(x_t, \overleftarrow{h}_{t+1})$$

$h_t = [\overrightarrow{h}_t;\overleftarrow{h}_t]$

This concatenation forms $H=\{h_1,\dots,h_n\}$, which feeds into the multi-level attention mechanism as described.

In hierarchical document and dialogue models, two-stage Bi-GRU encoding is customary: an initial Bi-GRU processes tokens within a sentence or utterance, and a higher-level Bi-GRU operates across sentence or utterance vectors, yielding a hierarchy of representations (Abreu et al., 2019, Xing et al., 2017).

4. Hierarchical Attention Variants and Multi-Level Fusion

There are two principal strategies for hierarchical and multi-level attention:

• Depth-Wise Attention (e.g., Ham): Computes attention at each layer in a stacked attention chain and aggregates their outputs via learnable weights. This approach has theoretical backing—norm-boundedness of each layer’s output and monotonic convergence (deeper networks do not increase minimal achievable loss) (Dou et al., 2018).
• Structural Hierarchical Attention: Applies attention at different linguistic levels (e.g., word-level within sentences, sentence-level within documents, utterance-level within conversations). For instance, in document classification, word-level attention produces sentence representations, and sentence-level attention aggregates these into a document vector (Abreu et al., 2019). In multi-turn response generation, word and utterance attention are cascaded to encode context for the decoder (Xing et al., 2017).

Some architectures combine both, employing multi-level mechanisms within each layer and fusing information horizontally and vertically (multi-granularity fusion) to leverage both global and local alignments, as in SLQA+ for reading comprehension (Wang et al., 2018).

5. Empirical Results and Applications

Models employing hierarchical Bi-GRU with multi-level attention have demonstrated state-of-the-art or superior empirical results in various tasks:

• In machine reading comprehension (DuReader, SQuAD, MS-MARCO), replacing single-level attention with Ham yielded up to 7.7% absolute gains in EM/F1, with a consistent ∼6.5% average improvement over prior architectures (Dou et al., 2018).
• For sequence generation, such as Chinese poem composition, augmenting a Bi-GRU decoder with Ham improved BLEU from 0.192 to 0.246, a 28% relative increase (Dou et al., 2018).
• In document classification, hierarchical Bi-GRU plus two-level attention models outperformed both deep CNNs and standard HANs; for example, the HAHNN with a TCN encoder achieved 95.17% accuracy (IMDB), surpassing HAN and deep CNN benchmarks (Abreu et al., 2019).
• In multi-turn dialogue, the HRAN model with hierarchical Bi-GRU encoding and dual-level attention reduced test perplexity and received higher human preference relative to HRED and VHRED (Xing et al., 2017).

6. Theoretical Properties and Implementation Considerations

Two notable theorems were established for the Ham mechanism (Dou et al., 2018):

• Norm-Boundedness: Each attention layer's output remains within the 2-norm of its input keys, thus avoiding vanishing or exploding representations regardless of depth.
• Monotonic Convergence: The optimal achievable loss does not increase with hierarchical depth, guaranteeing that augmenting attention levels cannot hinder model performance.

Training stability is enhanced via initialization (uniform softmax over weights), and recurrent update connections ensure robust gradient propagation. Practical implementations find that depth $L=5$–$10$ yields favorable trade-offs between computational cost and performance.

7. Comparative Overview and Architectural Variations

The following table summarizes core distinctions among representative hierarchical attention architectures utilizing Bi-GRU encoders:

Model	Hierarchy	Attention Depth/Strategy	Task(s)
Ham (Dou et al., 2018)	Sequence or encoder	Depth-wise, weighted layer sum	Reading comprehension, generation
HAHNN (Abreu et al., 2019)	Word → Sentence → Doc	Two-level, context vector at each	Document classification
HRAN (Xing et al., 2017)	Word → Utterance	Two-level, generation step-dependent	Dialogue response gen
SLQA+ (Wang et al., 2018)	Embedding → Encoding	Multi-granularity, fusion at each	Reading comprehension

Each model exploits the Bi-GRU's capacity for bidirectional context propagation, but varies in whether multi-level attention is structured by depth (Ham), by linguistic unit (HAHNN, HRAN), or by joint horizontal/vertical fusion (SLQA+).

Empirical results across these architectures demonstrate that hierarchical Bi-GRU with multi-level attention is a robust paradigm for structured sequential modeling, yielding consistent gains across natural language inference, comprehension, generation, and classification tasks (Dou et al., 2018, Abreu et al., 2019, Wang et al., 2018, Xing et al., 2017).