--- title: Graph-Infused Layers Transformer (GiLT) url: https://www.emergentmind.com/topics/graph-infused-layers-transformer-language-model-gilt type: topic --- # Graph-Infused Layers Transformer (GiLT) Searching arXiv for the named GiLT paper and closely related graph-infused transformer LM work. Searching arXiv for graph-infused transformer language model variants and adjacent architectures on text-attributed graphs and graph-language modeling. GiLT, the **Graph-Infused Layers Transformer Language Model**, is a structure-augmented autoregressive Transformer language model that incorporates **dependency graphs** into next-token prediction without inserting extra structural tokens into the sequence. Instead, it incrementally constructs a partial dependency graph alongside token generation and uses graph-derived features to modulate self-attention inside Transformer layers. The model is introduced in **“GiLT: Augmenting Transformer Language Models with Dependency Graphs”** [2605.15562]. In this formulation, linguistic structure is treated as a dynamic graph bias on attention rather than as bracket tokens, parsing actions, or a separate symbolic stream. ## 1. Definition, scope, and nomenclature GiLT is defined around two design commitments. First, it uses **dependency graphs rather than constituency trees**. Second, it does **not** insert extra structural tokens in language modeling; it injects structural information by modulating Transformer attention with features extracted from a dependency graph that is incrementally constructed during autoregressive decoding [2605.15562]. The paper positions GiLT against prior syntax-aware language models that emphasize constituency-tree structures or explicitly augment the sequence with parsing actions or structural markers. GiLT instead keeps the original tokenization and output space unchanged. A direct consequence is that it can be finetuned from a pretrained language model, because the symbol space is preserved. The method supports both **dependency parse trees** and **semantic dependency graphs**. The experiments use one syntactic setting, **DP**, and three semantic formalisms, **PSD**, **DM**, and **PAS**. The model is graph-general in the limited sense that dependency graphs subsume dependency trees, but the graph features used by GiLT are derived from **unlabeled dependencies** rather than explicit dependency-relation labels. A recurrent source of confusion is the acronym. GiLT in this sense refers specifically to **Graph-Infused Layers Transformer Language Model** [2605.15562]. This is distinct from **GILT**, the **Graph In-context Learning Transformer**, which is an **LLM-free, tuning-free** graph foundational model for few-shot node, edge, and graph classification rather than a language model [2510.04567]. ## 2. Graph-infused attention layers GiLT is built on top of **Transformer-XL (TXL)** for language modeling. The graph enters the model through the **self-attention score computation**, not through input-token augmentation. The operational pipeline is: generate a token prefix, predict dependencies over completed words, update a partial dependency graph, compute a **graph-based feature tape**, embed that tape, and inject the resulting vectors into graph-infused Transformer layers. The graph-based feature tape for token position \(x_k\) is \[ G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}. \] Each position stores three graph-derived features relating the current token’s word to each earlier token’s word. Because the graph is word-level while the LM is token-level, tokens within the same word share the same graph features. The three feature types are **degree**, **distance**, and **depth**. GiLT does not use plain degree; it uses a weighted directional degree: \[ \text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}}, \] with \[ m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10. \] Distance is the weighted shortest path in the current dependency graph, where traversing an edge along its direction costs \(m_{\mathrm{out}}\) and traversing against the direction costs \(m_{\mathrm{in}}\). Depth is defined relative to the root using the undirected backbone of the current rooted dependency graph; disconnected words receive depth \(0\). The feature tape is embedded as \[ e_k \in \mathbb{R}^{3 \times k \times d}, \] and each layer \(l\) applies a linear projection \[ e_k^l = f^l(e_k) \in \mathbb{R}^{k \times d}. \] The most compact attention modification given in the paper is \[ a^l_{kj} = [h^l_j + e^l_{kj}]^\top W_k^\top W_q h^l_k \tag{4} \] which adds the graph feature vector to the **key-side representation** before dot-product attention. In Transformer-XL form, the same mechanism is expressed as \[ \tilde{a}^l_{kj} = h^l_j W_{k,c}^{\top} W_q h^l_k + (r_{kj} + e^l_{kj}) W_{k,r}^{\top} W_q h^l_k + u W_{k,c}^{\top} h^l_k + v W_{k,r}^{\top}(r_{kj} + e^l_{kj}) \tag{5} \] where \(r_{kj}\) is the sinusoidal relative-position encoding. In effect, GiLT extends relative position bias into a **relative graph-position bias**. This is the core meaning of “graph-infused layers” in the model: a standard Transformer layer becomes graph-aware because its attention logits are modified by features extracted from the evolving dependency graph [2605.15562]. ## 3. Incremental dependency-graph construction GiLT is strictly autoregressive. At any decoding step, it can use only the generated prefix, the completed words so far, and the partial dependency graph over those words. The graph is updated **online**. When a word \(w_i\) is completed, GiLT predicts dependencies between \(w_i\) and previous words, including the root. The dependency predictor uses a **biaffine mechanism** over word representations. A word representation \(o_i \in \mathbb{R}^{3d}\) is formed by concatenating three components: the hidden state from the middle layer for the word’s first token, the hidden state from the penultimate layer for the word’s first token, and the input embedding of that first token. The paper’s rationale is that these representations contain information about the current word while avoiding final-layer states that are too specialized to next-token prediction. A naive graph update would add every edge whose probability exceeds \(0.5\), but the paper rejects this because beam search over graph hypotheses would become exponential in the number of possible edge subsets. GiLT therefore uses a restricted two-step update. First, it predicts the **number of new dependencies** \[ c_i \in \{0,1,\dots,C\}. \] The count distribution is \[ T_i := [T_i^0, T_i^1, \ldots, T_i^C] = \mathrm{softmax}\!\left( W_a\left( \frac{s_i}{\sqrt{2i-1}} \right) + b_a \right) \tag{3} \] where the score \(s_i\) aggregates candidate parent and child interactions and the normalization uses \(\sqrt{2i-1}\). Second, once \(c_i\) is determined, GiLT selects the top-\(c_i\) highest-scoring dependencies and adds them to the graph. This update scheme makes the graph search linear rather than exponential in the number of candidate edges. During training, teacher forcing is used for both token prediction and dependency prediction. During inference, graph structure is predicted online, and sentence probabilities are approximated with beam search over graph hypotheses. The training annotations on **BLLIP-LG** are not all gold: **PSD**, **PAS**, and **DM** semantic graphs are produced by **ACE**, and **DP** unlabeled projective dependency trees are produced by a **Biaffine-RoBERTa parser** [2605.15562]. ## 4. Training objective, model variants, and finetuning GiLT is not trained with language modeling loss alone. Its objective combines language modeling, binary dependency-edge prediction, and dependency-count prediction: \[ L = \alpha \frac{1}{N}\sum_{i=1}^{N} \mathrm{CE}(x_i,\hat{x}_i) + \beta \frac{1}{M^2}\sum_{j=1}^{M}\sum_{k=1}^{M} \mathrm{BCE}(P_{jk}, \hat{P}_{jk}) + \gamma \frac{1}{M}\sum_{k=1}^{M} \mathrm{CE}(T_k,\hat{T}_k) \tag{6} \] with \[ \alpha = 1,\qquad \beta = 0.2,\qquad \gamma = 0.2. \] The model also defines a joint probability over text \(x\) and graph \(y\): \[ p(x,y) = \prod_{k=1}^{N} p(x_k \mid x_{ \text{PAS} > \text{DP}. \] This suggests that the graph-oriented mechanism is particularly compatible with richer semantic dependency graphs. The paper also states that **TXL-Large** has more parameters than GiLT but only marginally improves syntactic scores, so the gains are not explained merely by scale. The feature ablation on **GiLT-PSD** indicates that syntactic generalization is sensitive to all three graph features. Starting from **PPL 14.9, BLIMP 75.7, SG 79.7**, removing degree yields **76.5 / 75.3**, removing depth yields **76.4 / 73.8**, and removing distance yields **74.5 / 75.1**. Removing the directional weighting of degree or distance also reduces SG. The paper’s interpretation is that the full combination yields the most robust structural behavior, while BLIMP and SG respond differently enough that neither should be treated as sufficient on its own. Efficiency is a separate empirical theme. Because GiLT does **not** insert extra structural tokens, it is substantially more practical than structure-token methods such as **DTG**. The reported generation speeds are **52.6 tokens/s** for TXL, **43.0** for GiLT, and **24.4** for DTG at beam size **1**. At beam size **300**, **DTG** could not run on an **NVIDIA A6000** due to memory, whereas GiLT still could. This supports the paper’s claim that the no-extra-token design is not only architecturally cleaner but materially more efficient. The pretrained-LM finetuning results show consistent downstream gains. Relative to **Post-GPT2**, **GiLT-GPT2** improves **RTE** from **64.1** to **65.3**, **SST2** from **94.84** to **95.11**, **MRPC** from **80.75/86.29** to **81.39/86.97**, and **STS-B** from **84.2/83.6** to **85.2/84.3**. On syntactic evaluation after finetuning, pretrained **GPT2** scores **BLIMP 82.8, SG 79.4**; **Post-GPT2** reaches **83.0, 84.6**; and **GiLT-GPT2** reaches **83.2, 85.5** [2605.15562]. ## 6. Position within the graph-language model design space GiLT belongs to a broader class of models that inject graph structure into Transformer computation, but it occupies a specific point in that design space. Its graph is **linguistic** rather than document-level or knowledge-graph structure, and its infusion mechanism is **attention modulation by dynamically predicted dependency features** rather than extra graph tokens or a separate graph encoder [2605.15562]. A useful comparison point is the **Graph Language Model (GLM)**, which converts a pretrained **T5** encoder into a graph transformer by adding graph-relative position biases and attention masks, allowing graph-only, text-only, and interleaved graph-text inputs [2401.07105]. GiLT is narrower in input domain but more tightly coupled to autoregressive next-token prediction and online graph construction. A second comparison is **GTLM**, which injects graph-aware attention biases directly into LLM attention modules for text-attributed graphs, adds only **0.015%** additional bias-related parameters relative to a **1B** base model, and proves prefix node permutation equivariance together with exact backward compatibility for single-node inputs [2605.10247]. Relative to GTLM, GiLT uses predicted dependency graphs and handcrafted graph features such as degree, distance, and depth, whereas GTLM broadcasts structural biases such as **SPD**, **RRWP**, and **Magnetic Laplacian** to token pairs. A third nearby design is **TAG-DLM**, which linearizes a sampled \(k\)-hop neighborhood of a text-attributed graph into a sequence and injects structure through a binary **topology attention mask** inside a masked diffusion language model [2606.31166]. TAG-DLM is graph-infused through visibility constraints across all bidirectional self-attention layers, but its infusion is explicitly described as **“purely through a binary attention mask.”** GiLT, by contrast, modulates attention with continuous graph-derived feature embeddings extracted from an incrementally predicted dependency graph. Other related systems use external graph modules rather than modifying attention directly. **DGTL** combines frozen LLMs with a disentangled multi-channel GNN and injects graph-derived embeddings into all LLM layers for text-attributed node classification [2310.18152]. **GILT** reframes few-shot graph tasks as numerical in-context learning with a linear GCN tokenizer and a two-stage Transformer, but it is explicitly **not really a “language model” in the conventional sense** [2510.04567]. These models underscore a central distinction: GiLT is a language model whose internal layers are graph-infused, not merely a graph learner that happens to use a Transformer. Two limitations define the current scope of GiLT. First, training depends on dependency annotations, and in the main experiments these are often **silver parses** rather than manual gold graphs. Second, the inference procedure still requires beam search over graph hypotheses, so exact marginalization over graphs is not performed. The paper also notes that GiLT uses **unlabeled dependencies**, which leaves finer-grained syntactic and semantic relation types unexploited. Within those constraints, GiLT’s contribution is to show that dependency structure can be made operational inside a Transformer LM by converting an incrementally predicted graph into a dynamic bias on self-attention, while avoiding the sequence-length and finetuning disadvantages of structural-token approaches [2605.15562]. Source: https://www.emergentmind.com/topics/graph-infused-layers-transformer-language-model-gilt