---
title: Hierarchical Token Prepending in Transformers
url: https://www.emergentmind.com/topics/hierarchical-token-prepending-htp
type: topic
---

# Hierarchical Token Prepending in Transformers

Hierarchical Token Prepending (HTP) is a methodology for augmenting transformer architectures—principally in decoder-only Large Language Models (LLMs)—to enhance information flow through explicit introduction of multiple summary tokens at different hierarchical levels of the input. HTP addresses the inherent information bottlenecks imposed by causal attention, improving representation quality, especially for long-context embeddings used in tasks such as retrieval or classification. It generalizes and supersedes single-token prepending by coupling blockwise summary generation with hierarchical token placement and mean-pooling, resulting in more robust, scalable, and interpretable embeddings for both zero-shot and finetuned settings [2511.14868]. Related approaches in the vision domain leverage hierarchical prompts for morphological discrimination in hierarchical image classification [2304.06385]. The following sections detail core mechanisms, theoretical motivations, architectural workflows, practical implications, and empirical results.

## 1. Theoretical Motivation and Bottlenecks

Decoder-only transformer models with causal masking restrict every token to attend only to preceding tokens. The mask
$$
\alpha_{ij}^{(\ell)} = 0 \quad \forall j > i
$$
prevents any token $i$ from accessing future context, so early token representations $h_i^{(L)}$ are deprived of later input information. This architectural constraint degrades embedding quality as sequence lengths increase. A standard solution is Token Prepending (TP): inserting a single summary token, which is typically fed the last-layer hidden state of the final token, to the front of the sequence. However, this approach creates a compression bottleneck, as information from an entire long input must be condensed into a single vector, causing “over-squashing” of semantic content and limiting downstream performance [2511.14868].

## 2. Hierarchical Token Construction and Placement

HTP improves upon TP by segmenting the input sequence
$$
x = (x_1, x_2, \dots, x_L)
$$
into $N$ contiguous blocks $B_i$ of approximately equal size $K$:
$$
B_i = (x_{(i-1)K+1}, x_{(i-1)K+2}, \dots, x_{iK}), \quad i=1,\dots,N
$$
For each block $B_i$, HTP introduces a “placeholder” summary token $\langle\mathrm{PST}\rangle_i$ directly into the token sequence. This summary token is dynamically populated at each transformer layer (for layers $\ell>1$) by copying the final hidden state of its corresponding block:
$$
s_i^{(\ell)} = h^{(\ell)}_{\mathrm{end}(B_i)}, \quad \tilde h^{(\ell)}_{\mathrm{pos}(\langle\mathrm{PST}\rangle_i)} = s_i^{(\ell)}
$$
Additionally, a higher-level of $N$ “global” summary tokens, $\langle\mathrm{B\!-\!PST}\rangle_1, \ldots, \langle\mathrm{B\!-\!PST}\rangle_N$, is prepended to the front of the sequence. These global tokens are themselves populated by copying information from the local blockwise summaries, ensuring that all block-level summaries are accessible from the very beginning of the sequence to every subsequent token, including those in later blocks. This architecture introduces $2N$ backward “edges” per sequence, rather than a single prepended summary, thereby offering much richer pathways for backward information flow [2511.14868].

## 3. Attention Mask Modification and Information Flow

The original causal mask is a strict lower-triangular matrix. HTP modifies this mask to enable every downstream token to attend to both local and global summary tokens regardless of their chronological position. Specifically, for each $\langle\mathrm{PST}\rangle_i$ and global $\langle\mathrm{B\!-\!PST}\rangle_i$,
$$
M_{j,\;\pos(\langle\mathrm{PST}\rangle_i)} = 1, \qquad M_{j,\;\pos(\langle\mathrm{B\!-\!PST}\rangle_i)} = 1 \qquad \forall\, j > \pos(\langle\mathrm{PST}\rangle_i)
$$
This reconfiguration ensures that representations can propagate not only along the default causal path but also through multiple explicit hierarchical shortcuts, mitigating vanishing transport and the loss of global information in deep transformers. *This suggests improved long-range dependency modeling with limited additional computation (≈1.1–1.3× speed/memory overhead)* [2511.14868].

## 4. Readout Strategy and Robustness

Standard last-token pooling, which extracts the embedding from the final token $h_n^{(L)}$, is highly sensitive to the attention mask and vulnerable to depth-induced attenuation. HTP replaces this step with mean-pooling across all (rewired) output tokens at a chosen exit layer $L'$:
$$
e = \frac{1}{L'} \sum_{t=1}^{L'} h^{(L')}_t
$$
Theoretical analysis demonstrates that, whereas last-token sensitivity decays rapidly with network depth (due to the lower-triangular propagation matrix $A$), mean-pooling sustains high sensitivity, as gradients sum over all paths:
$$
\left\|\frac{\partial \bar y}{\partial v_i^{(0)}}\right\| \leq \frac{K_L}{L'} \sum_{j=1}^{L'} A_{j,i}
$$
This pooling enhancement is essential for robust representation learning in long-context scenarios [2511.14868].

## 5. End-to-End Algorithmic Workflow

The HTP workflow is summarized as follows:

- Partition input tokens into $N$ blocks of size $K$
- Build the input sequence by interleaving global and local summary tokens:
  $$
  [\, \mathrm{B\!-\!PST}_1,\ldots,\mathrm{B\!-\!PST}_N,\, \mathrm{PST}_1,\, B_1,\, \mathrm{PST}_2,\, B_2,\,\ldots,\, \mathrm{PST}_N,\, B_N \,]
  $$
- Randomly initialize all summary token embeddings
- For each transformer layer:
  - Apply attention and MLP as usual
  - For $1 < \ell \leq L'$ (where $L'$ is the “early exit” layer):   
    - Update each $\langle\mathrm{PST}\rangle_i$ with the final embedding of $B_i$
    - Update each $\langle\mathrm{B\!-\!PST}\rangle_i$ with its corresponding $\langle\mathrm{PST}\rangle_i$
- At the output, mean-pool the final hidden states of all tokens up to layer $L'$

The process remains architecture-agnostic and is compatible with both zero-shot and finetuned transformer models [2511.14868].

## 6. Empirical Performance and Ablation Results

HTP demonstrates significant performance gains across diverse retrieval tasks and embedding benchmarks. In BEIR retrieval (context $\leq$512), HTP improves NDCG@10 scores to 30.4 (Mistral-7B) vs. 27.8 (vanilla mean) and 17.7 (TP), and to 27.4 (Gemma2-9B) vs. 25.5 (vanilla mean). On LongEmbed datasets with input lengths up to 8192, HTP maintains or surpasses vanilla performance (e.g., 49.29 on Gemma2-9B vs. 44.06 vanilla mean), while TP collapses under long-context (18.15) [2511.14868].

Ablative studies on block size $K$ indicate:
| Input Length | Optimal $K$ | Notes                                    |
|--------------|-------------|------------------------------------------|
| ≤ 512        | 1           | Dense shortcutting preferred             |
| Up to 16k    | 2 or 4      | Larger blocks reduce summary OOD effects |

Additionally, the memory and run-time overheads remain moderate (1.1–1.3×), and robustness to block partitioning indicates the flexibility of HTP for different context lengths [2511.14868].

## 7. Extensions to Vision Transformers and Other Domains

A related instantiation of hierarchical token mechanisms in vision is presented in "TransHP: Image Classification with Hierarchical Prompting," which leverages hierarchical prompt embedding and injection for hierarchical image classification. There, a prompt-pool for coarse class labels is predicted and injected at a specified transformer block, conditioning fine-class discrimination on explicit coarse-class hints. Empirical gains in classification accuracy (e.g., +2.83% for ViT-B/16 on ImageNet) demonstrate the effectiveness of hierarchical token strategies across modalities [2304.06385]. *A plausible implication is that explicit hierarchical token conditioning generalizes across language and vision contexts to facilitate fine-grained discrimination and robust representation learning.*

---

**References**

- "Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings" [2511.14868]
- "TransHP: Image Classification with Hierarchical Prompting" [2304.06385]

Source: https://www.emergentmind.com/topics/hierarchical-token-prepending-htp