Impact of extended Levi graph tokenization on GLM behavior

Determine the impact of removing whitespace between concepts and edges that occurs when tokenizing each node of the Levi graph individually in the Graph Language Model’s graph preprocessing pipeline, specifically on the resulting tokenization and representations produced when encoding Graphs of Triplets.

Background

To make LLMs operate natively on graphs, the paper converts Graphs of Triplets into extended Levi graphs and tokenizes each node individually. This procedure removes whitespace between concepts and relation tokens to ensure consistent tokenization for shared concepts across triplets.

The authors note that this modification may change the token sequence compared to a straightforward linearization of triplets, potentially affecting how pretrained LM weights interact with graph-structured inputs. They explicitly defer assessing the consequences of this change.