Hierarchical and Path-Aware Transformers

• Hierarchical and path-aware transformers are models that incorporate multi-level structure and explicit path dependencies to improve attention mechanisms in graphs and hierarchical texts.
• They utilize techniques like HDSE for encoding hierarchical node distances in graphs and path-adaptive masking in text, achieving measurable gains in tasks such as node classification and deep label prediction.
• This approach boosts expressivity and generalization by enabling transformers to capture internal hierarchy and structural biases, though it requires careful implementation and regularization.

Hierarchical and path-aware transformers are transformer architectures designed to incorporate hierarchical or multi-level structure and explicit path dependencies into the self-attention mechanism. These models provide improved inductive biases for data with inherent hierarchy, such as graphs with multiscale structure or texts with tree-organized labels. Enhancements include Hierarchical Distance Structural Encoding (HDSE) for graphs, path-adaptive masking for hierarchical label sequences, and theoretical analysis of how transformers internally infer position and depth without explicit encoding.

1. Hierarchical Distance Structural Encoding (HDSE) in Graph Transformers

HDSE provides a formal, trainable mechanism for encoding hierarchical node distances into the attention layers of graph transformers. Starting from an input graph $G^0=(V^0,E^0)$, HDSE builds a hierarchy of coarsened graphs $G^k$ via any graph-coarsening algorithm, each with a mapping $\phi_k: V^k \to V^{k+1}$. For each pair of nodes $u, v \in V^0$, the graph-hierarchy distance at level $k$ is defined as

$$\mathrm{GHD}^k(u,v) = \mathrm{SPD}\bigl(\phi_{k-1}\circ\cdots\circ\phi_0(u),\;\phi_{k-1}\circ\cdots\circ\phi_0(v)\bigr),$$

where $\mathrm{SPD}$ is the shortest-path distance in the $k$-th level coarsened graph, with $\mathrm{GHD}^0$ recovering standard path distance. The multi-level HDSE vector for each node pair is

$$D_{i,j} = [\mathrm{GHD}^0(i,j),\;\mathrm{GHD}^1(i,j),\;\dots,\;\mathrm{GHD}^K(i,j)]$$

with $G^k$0 the maximum hierarchy depth used (Luo et al., 2023).

In the transformer, each distance is clipped and embedded, and for each attention pair $G^k$1:

• Embeddings for all $G^k$2 levels are concatenated,
• Passed through an MLP ("Bias") to yield a scalar bias $G^k$3,
• Which is additively integrated into the attention logits.

For scalability on large graphs ($G^k$4), coarsened node representations are used in place of full-keyed attention, matching Hierarchical Linformer-style efficiency. The approach is strictly additive to existing positional/structural encodings and is compatible with all transformer backbones.

2. Path-Aware Masking in Hierarchical Text Transformers

In hierarchical text classification tasks, path-aware transformers such as PAMM-HiA-T5 address label dependencies by flattening the label tree for each sample into a sequence via breadth-first search (BFS), using separator tokens to mark hierarchy levels. The decoder receives the multi-level sequence, embedding label hierarchy directly into sequence order. A binary path-adaptive mask $G^k$5 ensures that at each decode step, attention can only target true ancestors (and separators) along the path to the current label:

• For label tokens, only their ancestors and level separators are permitted.
• The mask is applied multiplicatively to the regular attention weights.

An explicit regularization term,

$G^k$6

encourages the model's attention to concentrate on valid hierarchical paths.

This structure leads to explicit conditional generation of lower-level labels on true upper-level ones, yielding measurable gains in Macro-F1, especially on deep labels where label sparsity is a challenge (Huang et al., 2021).

3. Internal Hierarchy Reasoning without Explicit Positional Encoding

Recent theoretical analysis demonstrates that causal-masked transformers, given a start-of-sequence token (BOS), invent internal representations for both position and depth in hierarchical languages (e.g., Dyck$G^k$7 bracket languages), even without hand-crafted positional encodings. The architecture leverages:

• First-layer: using BOS as a positional anchor, the transformer computes continuous angular encodings $G^k$8 for position via attention to BOS and preceding tokens.
• Next layers: accumulate depth/stack count via special bracket token embeddings, producing angular depth encodings $G^k$9 mapping the current parenthesis depth.
• Higher layers: perform matching (e.g., matching open-close pairs, retrieving most recent open at same depth).

The suite of proofs establishes that for hierarchical recognition and generation:

• Recognition: A 5-layer, $\phi_k: V^k \to V^{k+1}$0-width, 1-head causal transformer (no PE) computes membership in Dyck$\phi_k: V^k \to V^{k+1}$1 [(Hayakawa et al., 2024), Thm 4.1].
• Generation: A 3-layer, $\phi_k: V^k \to V^{k+1}$2-width transformer (no PE) generates Dyck$\phi_k: V^k \to V^{k+1}$3 [(Hayakawa et al., 2024), Thm 4.2].
• Explicit positional encodings can reduce generalization to longer sequences by coupling representation scale to seen example lengths.

A plausible implication is that path- and hierarchy-aware attention may be best induced through architecture and masking, not merely positional embeddings.

4. Theoretical Expressivity and Generalization Analyses

Hierarchical encoding strategies enhance the expressivity and generalization of transformers over purely distance-based or flat positional schemes. Specifically:

• Graph HDSE enables transformers to simulate the Graph Distance–Weisfeiler–Lehman (GD-WL) test with hierarchical bias, outperforming shortest-path only methods in graph isomorphism distinction (e.g., distinguishing Dodecahedral from Desargues graphs) (Luo et al., 2023).
• There exist fixed-parameter HDSE-based transformers strictly more expressive than any SPD-based variant.
• For node classification, if label is determined by a particular HDSE-core, a suitably constructed transformer with HDSE can generalize arbitrarily well; pure SPD encoding cannot isolate the “core” ring.

Theoretical size bounds for hierarchical sequence tasks exhibit optimality: The least width needed to process the shuffle of $\phi_k: V^k \to V^{k+1}$4 Dyck languages grows as $\phi_k: V^k \to V^{k+1}$5, demonstrating the necessity of increased model dimension in parallel-depth tracking (Hayakawa et al., 2024).

5. Empirical Performance and Model Ablations

Graph Transformers with HDSE

In graph-level and node-level benchmarks, integration of HDSE yields consistent improvements:

• On ZINC (regression), GT achieves a reduction from $\phi_k: V^k \to V^{k+1}$6 to $\phi_k: V^k \to V^{k+1}$7 MAE with HDSE ($\phi_k: V^k \to V^{k+1}$8 reduction).
• On classification datasets including MNIST, CIFAR10, and Peptides-func, Macro-F1 and AP scores improve, particularly at deeper levels or on rare label tasks.
• On large-scale node classification (e.g., ogbn-proteins, ogbn-products), GOAT+HDSE outperforms scalable GNNs and transformers (e.g., improvement from $\phi_k: V^k \to V^{k+1}$9 to $u, v \in V^0$0 ROC-AUC).
• Per-epoch runtime with HDSE-attached linear transformers remains competitive, e.g., GOAT+HDSE achieves $u, v \in V^0$1 ms on PubMed.

Ablation studies confirm that both hierarchy construction (e.g., METIS, Loukas, Spectral) and bias encoding are necessary for optimal performance (Luo et al., 2023).

PAMM-HiA-T5 for Text Classification

On RCV1-V2, NYT, and WOS datasets:

• PAMM-HiA-T5 yields Macro-F1 increases over prior state-of-the-art (e.g., $u, v \in V^0$2 on RCV1-V2),
• Ablation shows that BFS flattening and level-wise dependency account for substantial improvements,
• Gains are concentrated at deeper label levels, confirming the efficacy for complex hierarchical taxonomies (Huang et al., 2021).

6. Practical Implementation Considerations and Limitations

• Preprocessing requirements: HDSE requires computation of graph hierarchy and all-pair distances per hierarchy level. For large graphs, METIS provides $u, v \in V^0$3 scalability.
• Mask complexity: Path-adaptive masks in text require per-sample dynamic computation of ancestor sets; regularization ensures correct learning.
• Hyperparameter choice: Maximum clipped distance $u, v \in V^0$4 and hierarchy depth $u, v \in V^0$5 balance computational load, detail, and cross-graph generalization. Typically, $u, v \in V^0$6 or $u, v \in V^0$7 suffices unless deep community hierarchy exists.
• Additivity: Both HDSE and path-aware masks are modular, permitting their combination with Laplacian PE, random walk SE, or other inductive biases.
• Regularization: Hierarchy-aware features introduce additional model capacity and possible overfitting risk on small datasets, warranting regularization or early stopping.

Table: Summary of Hierarchical and Path-Aware Transformer Mechanisms

Mechanism	Structural Domain	Key Element
HDSE	Graphs	Multi-level GHD vectors, coarsened attention (Luo et al., 2023)
PAMM-HiA-T5	Hierarchical sequences	Path-adaptive mask, BFS flattening (Huang et al., 2021)
Implicit Causal Encoding	Hierarchical languages	BOS-triggered position/depth, no explicit PE (Hayakawa et al., 2024)

In summary, hierarchical and path-aware transformers utilize explicit or implicit architectural modifications—ranging from distance encoding in graph structures, path masking in hierarchical label generation, to internal inference of position/depth in sequence models—to achieve superior expressivity and generalization on data with intrinsic multilevel structure.