Graph-Infused Layers Transformer (GiLT)

• The paper introduces GiLT, which augments autoregressive Transformers with dynamic dependency graph biases in self-attention without extra tokens.
• It incrementally constructs dependency graphs using a biaffine predictor to extract features like degree, distance, and depth for next-token prediction.
• GiLT achieves competitive perplexity while improving syntactic generalization and efficiency compared to traditional syntax-aware language models.

Searching arXiv for the named GiLT paper and closely related graph-infused transformer LM work. Searching arXiv for graph-infused transformer LLM variants and adjacent architectures on text-attributed graphs and graph-language modeling. GiLT, the Graph-Infused Layers Transformer LLM, is a structure-augmented autoregressive Transformer LLM that incorporates dependency graphs into next-token prediction without inserting extra structural tokens into the sequence. Instead, it incrementally constructs a partial dependency graph alongside token generation and uses graph-derived features to modulate self-attention inside Transformer layers. The model is introduced in “GiLT: Augmenting Transformer LLMs with Dependency Graphs” (Huang et al., 15 May 2026). In this formulation, linguistic structure is treated as a dynamic graph bias on attention rather than as bracket tokens, parsing actions, or a separate symbolic stream.

1. Definition, scope, and nomenclature

GiLT is defined around two design commitments. First, it uses dependency graphs rather than constituency trees. Second, it does not insert extra structural tokens in language modeling; it injects structural information by modulating Transformer attention with features extracted from a dependency graph that is incrementally constructed during autoregressive decoding (Huang et al., 15 May 2026).

The paper positions GiLT against prior syntax-aware LLMs that emphasize constituency-tree structures or explicitly augment the sequence with parsing actions or structural markers. GiLT instead keeps the original tokenization and output space unchanged. A direct consequence is that it can be finetuned from a pretrained LLM, because the symbol space is preserved.

The method supports both dependency parse trees and semantic dependency graphs. The experiments use one syntactic setting, DP, and three semantic formalisms, PSD, DM, and PAS. The model is graph-general in the limited sense that dependency graphs subsume dependency trees, but the graph features used by GiLT are derived from unlabeled dependencies rather than explicit dependency-relation labels.

A recurrent source of confusion is the acronym. GiLT in this sense refers specifically to Graph-Infused Layers Transformer LLM (Huang et al., 15 May 2026). This is distinct from GILT, the Graph In-context Learning Transformer, which is an LLM-free, tuning-free graph foundational model for few-shot node, edge, and graph classification rather than a LLM (Ma et al., 6 Oct 2025).

2. Graph-infused attention layers

GiLT is built on top of Transformer-XL (TXL) for language modeling. The graph enters the model through the self-attention score computation, not through input-token augmentation. The operational pipeline is: generate a token prefix, predict dependencies over completed words, update a partial dependency graph, compute a graph-based feature tape, embed that tape, and inject the resulting vectors into graph-infused Transformer layers.

The graph-based feature tape for token position $x_k$ is

$$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$

Each position stores three graph-derived features relating the current token’s word to each earlier token’s word. Because the graph is word-level while the LM is token-level, tokens within the same word share the same graph features.

The three feature types are degree, distance, and depth. GiLT does not use plain degree; it uses a weighted directional degree: $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$ with

$m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10.$

Distance is the weighted shortest path in the current dependency graph, where traversing an edge along its direction costs $m_{\mathrm{out}}$ and traversing against the direction costs $m_{\mathrm{in}}$. Depth is defined relative to the root using the undirected backbone of the current rooted dependency graph; disconnected words receive depth $0$.

The feature tape is embedded as

$e_k \in \mathbb{R}^{3 \times k \times d},$

and each layer $l$ applies a linear projection

$e_k^l = f^l(e_k) \in \mathbb{R}^{k \times d}.$

The most compact attention modification given in the paper is

$$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$0

which adds the graph feature vector to the key-side representation before dot-product attention. In Transformer-XL form, the same mechanism is expressed as

$$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$1

where $$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$2 is the sinusoidal relative-position encoding. In effect, GiLT extends relative position bias into a relative graph-position bias. This is the core meaning of “graph-infused layers” in the model: a standard Transformer layer becomes graph-aware because its attention logits are modified by features extracted from the evolving dependency graph (Huang et al., 15 May 2026).

3. Incremental dependency-graph construction

GiLT is strictly autoregressive. At any decoding step, it can use only the generated prefix, the completed words so far, and the partial dependency graph over those words. The graph is updated online.

When a word $$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$3 is completed, GiLT predicts dependencies between $$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$4 and previous words, including the root. The dependency predictor uses a biaffine mechanism over word representations. A word representation $$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$5 is formed by concatenating three components: the hidden state from the middle layer for the word’s first token, the hidden state from the penultimate layer for the word’s first token, and the input embedding of that first token. The paper’s rationale is that these representations contain information about the current word while avoiding final-layer states that are too specialized to next-token prediction.

A naive graph update would add every edge whose probability exceeds $$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$6, but the paper rejects this because beam search over graph hypotheses would become exponential in the number of possible edge subsets. GiLT therefore uses a restricted two-step update.

First, it predicts the number of new dependencies

$$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$7

The count distribution is

$$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$8

where the score $$G_k = [g_{1k}, g_{2k}, \ldots, g_{kk}] \in \mathbb{N}^{3 \times k}.$$9 aggregates candidate parent and child interactions and the normalization uses $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$0. Second, once $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$1 is determined, GiLT selects the top-$$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$2 highest-scoring dependencies and adds them to the graph.

This update scheme makes the graph search linear rather than exponential in the number of candidate edges. During training, teacher forcing is used for both token prediction and dependency prediction. During inference, graph structure is predicted online, and sentence probabilities are approximated with beam search over graph hypotheses. The training annotations on BLLIP-LG are not all gold: PSD, PAS, and DM semantic graphs are produced by ACE, and DP unlabeled projective dependency trees are produced by a Biaffine-RoBERTa parser (Huang et al., 15 May 2026).

4. Training objective, model variants, and finetuning

GiLT is not trained with language modeling loss alone. Its objective combines language modeling, binary dependency-edge prediction, and dependency-count prediction: $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$3 with

$$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$4

The model also defines a joint probability over text $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$5 and graph $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$6: $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$7 so test-time sentence probability requires marginalizing over possible graph structures; in practice this is approximated with beam search.

In the main language-modeling experiments, GiLT uses a 16-layer TXL baseline with about 252M parameters. The paper separately reports about 268M parameters in the Transformer and about 54M parameters in the graph-related modules for GiLT. The language-modeling runs use one NVIDIA A6000 GPU for about 50 hours per run. Reported central hyperparameters include $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$8 and $$\text{degree}(w) = m_{\mathrm{out}} c_{\mathrm{out}} + m_{\mathrm{in}} c_{\mathrm{in}},$$9.

A second use case is pretrained-LM adaptation. The paper creates GiLT-GPT2 from GPT2-medium (355M) by replacing the last 12 vanilla Transformer layers with graph-infused layers, then finetuning on BLLIP-LG first. For this stage, the reported settings are batch size 64, 5000 warmup steps, cosine decay, maximum learning rate $m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10.$0 for the base model, and $m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10.$1 for newly initialized GiLT parameters; downstream-task finetuning uses learning rate $m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10.$2. Because newly initialized graph modules initially disturb hidden-state semantics, the model is first trained as an LM alone for 5000 steps before joint training with the biaffine graph module (Huang et al., 15 May 2026).

5. Empirical profile: perplexity, syntactic generalization, and efficiency

The main language-modeling evaluation uses BLLIP-LG, with syntactic generalization measured on BLIMP and the SG test suites. The reported results show that GiLT maintains competitive perplexity while improving syntactic generalization, especially with semantic dependency graphs (Huang et al., 15 May 2026).

Model	PPL	BLIMP / SG
TXL baseline	14.8	75.1 / 72.1
Pushdown-LM	14.6	74.0 / 74.6
GiLT-DP	15.5	76.1 / 72.3
GiLT-PAS	15.0	73.0 / 75.7
GiLT-DM	14.9	74.9 / 76.3
GiLT-PSD	14.9	75.7 / 79.7

The strongest overall variant is GiLT-PSD, which raises SG from 72.1 to 79.7 while remaining near the baseline in perplexity. The paper characterizes the relative ordering roughly as

$m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10.$3

This suggests that the graph-oriented mechanism is particularly compatible with richer semantic dependency graphs. The paper also states that TXL-Large has more parameters than GiLT but only marginally improves syntactic scores, so the gains are not explained merely by scale.

The feature ablation on GiLT-PSD indicates that syntactic generalization is sensitive to all three graph features. Starting from PPL 14.9, BLIMP 75.7, SG 79.7, removing degree yields 76.5 / 75.3, removing depth yields 76.4 / 73.8, and removing distance yields 74.5 / 75.1. Removing the directional weighting of degree or distance also reduces SG. The paper’s interpretation is that the full combination yields the most robust structural behavior, while BLIMP and SG respond differently enough that neither should be treated as sufficient on its own.

Efficiency is a separate empirical theme. Because GiLT does not insert extra structural tokens, it is substantially more practical than structure-token methods such as DTG. The reported generation speeds are 52.6 tokens/s for TXL, 43.0 for GiLT, and 24.4 for DTG at beam size 1. At beam size 300, DTG could not run on an NVIDIA A6000 due to memory, whereas GiLT still could. This supports the paper’s claim that the no-extra-token design is not only architecturally cleaner but materially more efficient.

The pretrained-LM finetuning results show consistent downstream gains. Relative to Post-GPT2, GiLT-GPT2 improves RTE from 64.1 to 65.3, SST2 from 94.84 to 95.11, MRPC from 80.75/86.29 to 81.39/86.97, and STS-B from 84.2/83.6 to 85.2/84.3. On syntactic evaluation after finetuning, pretrained GPT2 scores BLIMP 82.8, SG 79.4; Post-GPT2 reaches 83.0, 84.6; and GiLT-GPT2 reaches 83.2, 85.5 (Huang et al., 15 May 2026).

6. Position within the graph-LLM design space

GiLT belongs to a broader class of models that inject graph structure into Transformer computation, but it occupies a specific point in that design space. Its graph is linguistic rather than document-level or knowledge-graph structure, and its infusion mechanism is attention modulation by dynamically predicted dependency features rather than extra graph tokens or a separate graph encoder (Huang et al., 15 May 2026).

A useful comparison point is the Graph LLM (GLM), which converts a pretrained T5 encoder into a graph transformer by adding graph-relative position biases and attention masks, allowing graph-only, text-only, and interleaved graph-text inputs (&&&10&&&). GiLT is narrower in input domain but more tightly coupled to autoregressive next-token prediction and online graph construction.

A second comparison is GTLM, which injects graph-aware attention biases directly into LLM attention modules for text-attributed graphs, adds only 0.015% additional bias-related parameters relative to a 1B base model, and proves prefix node permutation equivariance together with exact backward compatibility for single-node inputs (Vajda, 11 May 2026). Relative to GTLM, GiLT uses predicted dependency graphs and handcrafted graph features such as degree, distance, and depth, whereas GTLM broadcasts structural biases such as SPD, RRWP, and Magnetic Laplacian to token pairs.

A third nearby design is TAG-DLM, which linearizes a sampled $m_{\mathrm{in}} = 1, \qquad m_{\mathrm{out}} = 10.$4-hop neighborhood of a text-attributed graph into a sequence and injects structure through a binary topology attention mask inside a masked diffusion LLM (Chen et al., 30 Jun 2026). TAG-DLM is graph-infused through visibility constraints across all bidirectional self-attention layers, but its infusion is explicitly described as “purely through a binary attention mask.” GiLT, by contrast, modulates attention with continuous graph-derived feature embeddings extracted from an incrementally predicted dependency graph.

Other related systems use external graph modules rather than modifying attention directly. DGTL combines frozen LLMs with a disentangled multi-channel GNN and injects graph-derived embeddings into all LLM layers for text-attributed node classification (Qin et al., 2023). GILT reframes few-shot graph tasks as numerical in-context learning with a linear GCN tokenizer and a two-stage Transformer, but it is explicitly not really a “LLM” in the conventional sense (Ma et al., 6 Oct 2025). These models underscore a central distinction: GiLT is a LLM whose internal layers are graph-infused, not merely a graph learner that happens to use a Transformer.

Two limitations define the current scope of GiLT. First, training depends on dependency annotations, and in the main experiments these are often silver parses rather than manual gold graphs. Second, the inference procedure still requires beam search over graph hypotheses, so exact marginalization over graphs is not performed. The paper also notes that GiLT uses unlabeled dependencies, which leaves finer-grained syntactic and semantic relation types unexploited. Within those constraints, GiLT’s contribution is to show that dependency structure can be made operational inside a Transformer LM by converting an incrementally predicted graph into a dynamic bias on self-attention, while avoiding the sequence-length and finetuning disadvantages of structural-token approaches (Huang et al., 15 May 2026).