Focused Hierarchical RNNs (FHE)

• Focused Hierarchical RNNs (FHE) are a sequence processing mechanism that uses learned, question-conditioned discrete gates to update higher recurrent layers.
• The architecture features a two-layer LSTM where the lower layer processes every token and the upper layer updates only at contextually relevant boundaries, reducing computational cost.
• Demonstrated on synthetic tasks and QA benchmarks like SearchQA and MS MARCO, FHE improves long-horizon generalization while controlling update sparsity.

Focused Hierarchical RNNs, introduced as the Focused Hierarchical Encoder (FHE), are a mechanism for conditional sequence processing in which a recurrent encoder does not propagate every token uniformly through a stacked hierarchy. Instead, a multi-layer conditional sequence encoder reads one token at a time and makes a discrete, question- or context-conditioned decision about whether the current token is relevant enough to update a higher recurrent layer. In the formulation reported in "Focused Hierarchical RNNs for Conditional Sequence Processing," the resulting hierarchy is intended to concentrate computation and attention on key parts of the input, yielding a compact concept-level representation of long sequences while reducing memory and attention-search cost (Ke et al., 2018).

1. Problem formulation and conceptual position

The model is motivated by a limitation of standard attention-based recurrent encoders: many such architectures use an encoder with attention that looks over the entire sequence and assigns a weight to each token independently. FHE reformulates this setting by introducing a conditional hierarchy in which the encoder itself is focused before downstream attention is applied. The conditioning signal is the question or context embedding, and the central decision variable is whether the current token should trigger an update in the upper recurrent layer (Ke et al., 2018).

Within this formulation, the hierarchy is not merely a conventional stacked RNN. The lower layer processes every token, whereas the upper layer updates only when a learned discrete boundary gate opens. This distinction is operationally important because the comparison baselines include both a single-layer model, equivalent to gates always closed, and a two-layer model with always open gates. FHE differs from both by making sequence compression input- and query-dependent rather than structurally fixed.

A plausible implication is that FHE shifts part of the burden of relevance estimation from post hoc attention to online segmentation. The paper states this directly in functional terms: focusing via learned, question-conditioned gates induces a compact, concept-level summary of long passages, reduces the burden on the attention mechanism, and improves long-horizon generalization (Ke et al., 2018).

2. Encoder architecture and discrete boundary mechanism

The encoder is a two-layer LSTM. The lower, token-level layer processes every input token $x_t$ sequentially, producing hidden and cell states

$$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$

The upper, focused-concept layer is updated conditionally. When the boundary gate is closed, it carries over its previous state:

$h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$

At the end of encoding, the model has

$$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$

but only $k \approx \sum_t \hat g_t$ of the upper-layer states are unique updates. Downstream attention and decoding attend only over $H^u$, not over the full token-level sequence, which reduces both memory and attention-search cost (Ke et al., 2018).

The boundary-gate mechanism is discrete. At each time step $t$, a scalar gate-opening probability is computed as

$$b_t = \sigma\!\left(w_b^\top\, \mathrm{LReLU}(W_b z_t + b_b)\right),$$

where $z_t$ concatenates features conditioned on the question/context $q$ and the lower-layer state $$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$0. In the simplest QA setup,

$$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$1

A Bernoulli sample then determines whether the upper LSTM updates:

$$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$2

If $$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$3,

$$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$4

If $$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$5,

$$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$6

This architecture makes the upper representation sparse in time. In effect, the lower layer remains token-synchronous, while the upper layer becomes event-synchronous, where events are defined by learned, question-conditioned boundaries.

3. Optimization, policy gradient, and sparsity control

The training objective maximizes the log-likelihood of the answer $$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$7 given passage $$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$8 and question $$h_t^l, c_t^l = \mathrm{LSTM}_l(x_t, h_{t-1}^l, c_{t-1}^l).$$9:

$h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$0

Because the gate decisions are discrete, the model uses a policy-gradient estimator. For the gate policy $h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$1,

$h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$2

where $h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$3 is a control-variate baseline used to reduce variance. In practice, the reward is

$h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$4

and a small entropy bonus $h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$5 is added to encourage exploration, with $h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$6 in QA (Ke et al., 2018).

The model also introduces explicit sparsity regularization to avoid the degenerate solution in which all gates open:

$h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$7

$h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$8

Here, $h_t^u = h_{t-1}^u,\qquad c_t^u = c_{t-1}^u.$9 specifies the allowed fraction of opens without penalty and $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$0 controls the penalty strength.

The paper concludes that policy-gradient training with a sparse-opening penalty $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$1 yields a controllable trade-off between accuracy and number of opens. This is significant because it makes the compression ratio not merely an emergent property but a tunable aspect of the model. At the same time, the reported limitations note that training discrete gates remains more complex than fully-differentiable soft gating (Ke et al., 2018).

4. Synthetic tasks and generalization behavior

The synthetic evaluations were designed to probe generalization and gate behavior in controlled settings. The first task, the "Picking Task," uses a random digit sequence of length $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$2 and a question $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$3 asking which digit is most frequent among the first $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$4 tokens, with ties broken by choosing the largest digit. The baselines are LSTM1, a single-layer model equivalent to gates always closed, and LSTM2, a two-layer model with always open gates.

With fixed hyper-parameters $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$5 and $$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$6, FHE achieves the following accuracies on the Picking Task (Ke et al., 2018):

Setting	LSTM1 / LSTM2 / FHE
$$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$7	99.4% / 99.7% / 99.5%
$$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$8	97.0% / 99.2% / 99.4%
$$H^l = \{h_1^l,\ldots,h_n^l\},\qquad H^u = \{h_1^u,\ldots,h_n^u\},$$9	92.9% / 97.5% / 96.9%

The more consequential result is out-of-distribution length generalization. When trained on $k \approx \sum_t \hat g_t$0, $k \approx \sum_t \hat g_t$1, and tested up to $k \approx \sum_t \hat g_t$2, the reported accuracies are:

Test length	LSTM1 / LSTM2 / FHE
$k \approx \sum_t \hat g_t$3	$k \approx \sum_t \hat g_t$4 / $k \approx \sum_t \hat g_t$5 / $k \approx \sum_t \hat g_t$6
$k \approx \sum_t \hat g_t$7	$k \approx \sum_t \hat g_t$8 / $k \approx \sum_t \hat g_t$9 / $H^u$0

The paper attributes this to boundary behavior: FHE learns to open gates approximately only at $H^u$1, after which attention focuses on that single upper-state. This observation is central to the model’s interpretation. Rather than maintaining a uniformly dense temporal abstraction, the upper layer appears to localize the decisive computation around the query-dependent boundary.

A second controlled task treats each MNIST image as a sequence of 784 pixels and asks a binary question of the form "is this digit $H^u$2?". On validation, the reported accuracies are LSTM1 97.3%, LSTM2 98.4%, and FHE-fixed, with $H^u$3 and $H^u$4, 99.1% (Ke et al., 2018). The gate visualization shows that gates open mostly around the white foreground pixels of the digit. This suggests that the learned boundaries can align with semantically informative structure even when the input is a rasterized sequence rather than text.

5. Large-scale question answering benchmarks

The model is also evaluated on SearchQA and MS MARCO, both described as large-scale QA tasks. On SearchQA, which uses a Jeopardy! question with approximately 50 snippets, the metrics are Exact Match (EM) for single-word answers and F1 for multi-word answers. The reported validation and test results are as follows (Ke et al., 2018):

| Model | Validation F1 | EM | Test F1 | EM | |---|---|---| | LSTM1+PtrSoftmax | 52.8 | 41.9 | 48.7 | 39.7 | | LSTM2+PtrSoftmax | 55.3 | 44.7 | 51.9 | 41.7 | | FHE | 56.7 | 49.6 | 53.4 | 46.8 | | Prior AQA model [Buck ‘18] | 47.7 | 40.5 | 45.6 | 38.7 |

On MS MARCO, described as web queries with passages and human answers, the generative models are evaluated with BLEU-1 and ROUGE-L. The reported validation and test results are:

| Model | Validation BLEU-1 | ROUGE-L | Test BLEU-1 | ROUGE-L | |---|---|---| | Seq2Seq (Nguyen ‘16) | – | 8.9 | – | – | | Memory Net (Nguyen ‘16) | – | 11.9 | – | – | | LSTM1+PtrSoftmax | 24.8 | 26.5 | 28 | 28 | | LSTM2+PtrSoftmax | 24.3 | 23.3 | 27 | 28 | | FHE | 27.3 | 26.7 | 30 | 30 |

The validation ablations on MS MARCO identify three components as consequential. Removing the elementwise product between $H^u$5 and $H^u$6 yields 18.5 | 19.3, removing pointer-softmax yields 20.5 | 18.7, and removing learned boundaries yields 23.5 | 24.0 (Ke et al., 2018). These ablations indicate that the gains are not attributable to a single isolated modification; rather, the conditional gating, the multiplicative conditioning term, and the output mechanism all contribute to performance.

Across both benchmarks, the paper reports consistent improvements over prior work and over the recurrent baselines. In that sense, the large-scale QA results complement the synthetic findings: the same mechanism that improves long-horizon generalization in controlled settings also improves benchmark accuracy in realistic QA.

6. Interpretation, limitations, and relation to adjacent design choices

The principal interpretive claim made for FHE is that learned, question-conditioned focusing induces a compact summary of long passages at the concept level. In the reported experiments, this has two stated consequences: it reduces the burden on the attention mechanism and improves long-horizon generalization (Ke et al., 2018). The synthetic gate visualizations support this interpretation by showing targeted opens at decisive positions, while the MNIST experiment suggests that the learned boundaries may align with task-relevant structure outside textual data.

FHE should therefore be distinguished from a standard stacked recurrent encoder with attention. The LSTM2 baseline demonstrates that simply adding an upper recurrent layer with always open gates does not reproduce the same behavior. The critical operation is conditional updating under a discrete gate, not depth alone. This is an important architectural distinction because it locates the model’s compression mechanism in learned temporal sparsification rather than in post-encoding attention weights.

The paper also states several limitations and future directions. Training discrete gates remains more complex than fully-differentiable soft gating. Extending the architecture to deeper hierarchies, self-supervised boundary pretraining, or adaptive $H^u$7 may further improve performance. Integration with span-based mechanisms could combine generative flexibility with pointer accuracy (Ke et al., 2018). These statements delimit the contribution carefully: FHE is presented not as a universal replacement for attention-based sequence models, but as a specific recurrent design in which discrete, query-conditioned boundaries improve both efficiency-relevant representation structure and empirical QA performance.

A plausible implication is that FHE occupies an intermediate position between token-level encoding and explicit segment-level modeling. It does not require externally supplied segmentation, yet it does produce an upper representation whose unique states correspond to learned update points. That property is the defining feature of Focused Hierarchical RNNs as formulated in the original work.