---
title: 'GS-Quant: Discrete Semantics in KGC'
url: https://www.emergentmind.com/topics/gs-quant
type: topic
---

# GS-Quant: Discrete Semantics in KGC

GS-Quant is a framework for knowledge graph completion (KGC) that quantizes entity representations into semantically organized discrete code sequences for use by large language models (LLMs). It is designed to address the modality gap between continuous knowledge-graph embeddings and discrete LLM tokens by replacing flat numerical compression with a coarse-to-fine code structure. The method combines residual quantization with a **Granular Semantic Enhancement** module and a **Generative Structural Reconstruction** module, then inserts the learned code tokens into an LLM vocabulary for KGC prompting and reranking. On WN18RR and FB15k-237, GS-Quant reports improvements over prior LLM baselines in both MRR and Hits@1, with the strongest gains concentrated in top-rank accuracy [2604.21649].

## 1. Problem formulation and conceptual basis

GS-Quant is formulated in the standard KGC setting. A knowledge graph is defined as
\[
\mathcal{G} = \{\mathcal{E}, \mathcal{R}, \mathcal{T}\},
\]
where \(\mathcal{E}\) is the entity set, \(\mathcal{R}\) is the relation set, and \(\mathcal{T}\) is the set of observed triples. Each triple has the form
\[
(h,r,t)\in\mathcal{T},
\]
with \(h,t\in\mathcal{E}\) and \(r\in\mathcal{R}\). The downstream task is mainly tail prediction, \((h,r,?)\), although head prediction is also possible. The paper’s central premise is that existing LLM-based KGC methods either linearize graph structure into text or inject dense embeddings into the model, and that both choices are mismatched to autoregressive token reasoning [2604.21649].

The framework is motivated by four contrasts between KG embeddings and LLM tokens: continuous versus discrete representation, holistic versus sequential structure, lookup-style versus autoregressive reasoning dynamics, and entangled latent semantics versus token-like compositional semantics. GS-Quant treats these contrasts not as an incidental implementation detail but as the main systems problem. Its argument is that prior quantization approaches such as SSQR produce code tuples that act as compressed numeric surrogates rather than as semantically ordered symbolic descriptors.

This motivates the method’s core design principle: entity codes should follow a linguistic coarse-to-fine logic. Earlier code positions are intended to encode broad semantic categories, while later positions refine specific attributes or instance-level distinctions. In the paper’s formulation, that principle is enforced by quantization-time supervision rather than being left to emerge implicitly.

## 2. Entity representation and residual quantization backbone

Before quantization, GS-Quant constructs an entity representation by combining structural and textual signals. Structural information is produced by a KG embedding backbone, exemplified by RotatE, yielding
\[
\mathbf{s}_x^{\mathcal{G}} \in \mathbb{R}^d.
\]
Textual information is produced by a pretrained language model over entity names and descriptions, yielding
\[
\mathbf{s}_x^T \in \mathbb{R}^d.
\]
The fused entity representation is
\[
\mathbf{s}_x = \rho\, \mathbf{s}_x^{\mathcal{G}} + (1-\rho)\,\mathbf{s}_x^T,
\]
with \(\rho\in[0,1]\) controlling the balance between relational and textual content [2604.21649].

Quantization is built on residual quantization. An entity representation \(\mathbf{s}\) is projected by an MLP,
\[
\mathbf{z} = \text{MLP}(\mathbf{s}),
\]
and the initial residual is
\[
\mathbf{r}_0 = \mathbf{z}.
\]
GS-Quant uses \(m\) residual quantization levels, with codebooks
\[
\mathbf{C}^{l} = \{\mathbf{v}_k^{l}\}_{k=1}^{K}.
\]
In the reported implementation, the framework uses **4 codebook layers** and **codebook size 1024**, so each entity receives a **4-token code tuple**. At level \(l\), the assigned code is
\[
c_l = \arg\min_k \|\mathbf{r}_l - \mathbf{v}_k^l\|_2,
\]
followed by the residual update
\[
\mathbf{r}_{l+1} = \mathbf{r}_l - \mathbf{v}_{c_l}^{l}.
\]
The final discrete representation is
\[
\mathcal{I} = \{c_i\}_{i=0}^{m-1}.
\]

The residual quantization loss is
\[
\mathcal{L}_{\text{Q}} = \sum_{l=0}^{m-1} \Big( \| \operatorname{sg}[\mathbf{r}_l] - \mathbf{v}^{l}_{c_l} \|_2^2 + \alpha \| \mathbf{r}_l - \operatorname{sg}[\mathbf{v}^{l}_{c_l}] \|_2^2 \Big),
\]
where \(\operatorname{sg}[\cdot]\) is stop-gradient and \(\alpha\) is the commitment weight. The paper is explicit that \(\mathcal{L}_{\text{Q}}\) alone is insufficient for semantically meaningful codebooks; it creates a numerical hierarchy, but not necessarily a semantic one.

## 3. Granular Semantic Enhancement

Granular Semantic Enhancement (GSE) is the component that aligns quantization depth with semantic granularity. The framework first constructs a hierarchy tree \(\mathcal{H}\) by hierarchical clustering over entity semantic representations. For a batch \(E\) of entities, each entity \(\mathbf{e}\in E\) is associated with a cluster centroid \(\boldsymbol{\mu}_{\mathbf{e}}\). To allow gradients to pass through discrete assignments, GS-Quant defines a differentiable surrogate sequence
\[
\mathcal{V}_e = \{\tilde{\mathbf{v}}_i\}_{i=0}^{m-1},
\qquad
\tilde{\mathbf{v}}_i = \mathbf{r}_i + \operatorname{sg}[\mathbf{v}_{c_i}^i - \mathbf{r}_i].
\]
This makes the forward pass use the selected code vector while the backward pass follows the residual path [2604.21649].

GSE contains two losses. The first is a coarse-to-fine alignment objective,
\[
\begin{split}
\mathcal{L}_1 &= -\frac{1}{|E|} \sum_{\mathbf{e} \in E} \sum_{i=0}^{m-1} \frac{\lambda_1^{i+1}}{m} \\
&\quad \times \log \frac{ \exp(\tilde{\mathbf{v}}_i \cdot \boldsymbol{\mu}_{\mathbf{e}} / \tau) }{ \sum_{\mathbf{e}' \in E} \exp(\tilde{\mathbf{v}}_i \cdot \boldsymbol{\mu}_{\mathbf{e}'} / \tau) }.
\end{split}
\]
Because the layer weight decays with depth, earlier quantization levels receive stronger pressure to align with broad semantics. The second is a hierarchical separability objective,
\[
\begin{split}
\mathcal{L}_2 &= \frac{1}{|E|} \sum_{\mathbf{e} \in E} \sum_{i=0}^{m-1} \frac{\lambda_2^{m-i}}{m \cdot |\mathcal{N}_{\mathbf{e}}|} \\
&\quad \times \sum_{\mathbf{n} \in \mathcal{N}_{\mathbf{e}}}
\log \frac{ \exp(\tilde{\mathbf{v}}_i \cdot \mathbf{n} / \tau) }{ \sum_{\mathbf{e}' \in E} \exp(\tilde{\mathbf{v}}_i \cdot \boldsymbol{\mu}_{\mathbf{e}'} / \tau) }.
\end{split}
\]
Here \(\mathcal{N}_{\mathbf{e}}\) is the set of neighbor centroids of entity \(\mathbf{e}\) in the hierarchy tree. The reverse decay in \(\lambda_2\) pushes deeper layers toward finer discrimination.

The total GSE objective is
\[
\mathcal{L}_{\text{GSE}} = \mathcal{L}_1 + \mathcal{L}_2.
\]
Operationally, this makes low-index codes coarse and high-index codes discriminative. The paper’s qualitative examples, including containment hierarchy and fine-grained degree distinctions, are consistent with this intended role separation.

## 4. Generative Structural Reconstruction and LLM integration

Even semantically stratified codebooks do not guarantee that the code tuple behaves like an ordered sequence. GS-Quant therefore adds Generative Structural Reconstruction (GSR), which uses a lightweight GPT-style Transformer decoder to impose causal dependencies over the code sequence. The decoder takes learnable queries
\[
\mathcal{Q} = \{\mathbf{q}_i\}_{i=0}^{L}
\]
concatenated with the surrogate code sequence \(\mathcal{V}_e\), and produces outputs \(\{\mathbf{o}_i\}_{i=0}^{L}\) through causal self-attention. These outputs are aligned with multi-granular targets: \(\mathbf{q}_0\) reconstructs the entity embedding \(\mathbf{s}\), while later query positions reconstruct hierarchy ancestors \(\{\mathbf{h}_i\}\) [2604.21649].

The GSR loss is
\[
\begin{split}
\mathcal{L}_{\text{GSR}} &= \|\tilde{\mathbf{o}}_0 - \mathbf{s}\|_2^2 + \lambda_s \|\tilde{\mathbf{o}}_1 - \mathbf{h}_0\|_2^2 \\
&\quad + \lambda_h \sum_{i=2}^{L} \|\tilde{\mathbf{o}}_i - \mathbf{h}_{i-1}\|_2^2.
\end{split}
\]
In the reported settings, reconstruction layers are 2, reconstruction heads are 4, and parent recon count is 5. For FB15k-237, \(\lambda_s=1\) and \(\lambda_h=1\); for WN18RR, \(\lambda_s=0.05\) and \(\lambda_h=1\). The full pretraining objective is
\[
\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{Q}} + \mathcal{L}_{\text{GSE}} + \mathcal{L}_{\text{GSR}}.
\]

After pretraining, codebook entries are added as new tokens in the LLM vocabulary. The original LLM parameters are frozen, and only the new token embeddings and LoRA adapter matrices in attention and feed-forward layers are updated. The reported fine-tuning setup uses **Llama3-8B** as the unified LLM backbone, with **LoRA rank 32**, **LoRA \(\alpha = 64\)**, learning rate \(2\times10^{-4}\), optimizer AdamW, bf16 precision, max training steps 3800, and DeepSpeed Zero-3. The prompt includes the query triple, textual description of the unknown entity, relational context triplets, and a candidate list in which each candidate is annotated with its 4-token code sequence. Inference is performed by having the fine-tuned LLM select the most plausible entity from the candidate set.

## 5. Empirical performance and ablation evidence

GS-Quant is evaluated on **FB15k-237** and **WN18RR**, using Hits@1, Hits@3, Hits@10, and MRR. The paper emphasizes that Hits@1 and MRR are the most practically informative because they measure whether the correct entity is ranked at the top [2604.21649].

| Dataset | GS-Quant | Best prior LLM baseline |
|---|---:|---:|
| WN18RR MRR | **0.635** | 0.617 |
| WN18RR Hits@1 | **0.594** | 0.569 |
| FB15k-237 MRR | **0.455** | 0.439 |
| FB15k-237 Hits@1 | **0.386** | 0.364 |

On **WN18RR**, GS-Quant reports **MRR 0.635**, **Hits@1 0.594**, **Hits@3 0.649**, and **Hits@10 0.712**. The strongest prior LLM baseline reported in the paper, DIFT, reaches **MRR 0.617** and **Hits@1 0.569**. On **FB15k-237**, GS-Quant reports **MRR 0.455**, **Hits@1 0.386**, **Hits@3 0.479**, and **Hits@10 0.592**, compared with DIFT at **MRR 0.439** and **Hits@1 0.364**. The gains are therefore **+0.018 MRR** and **+0.025 Hits@1** on WN18RR, and **+0.016 MRR** and **+0.022 Hits@1** on FB15k-237.

The ablations are consistent with the architecture claim. On FB15k-237, removing \(\mathcal{L}_1\) drops MRR from **0.455** to **0.450**, removing \(\mathcal{L}_2\) also yields **0.450**, removing \(\mathcal{L}_{\text{GSR}}\) yields **0.448**, and removing the code tokens entirely yields **0.404**. On WN18RR, the corresponding MRR values are **0.629**, **0.625**, **0.627**, and **0.607**, versus **0.635** for the full model. The largest degradation comes from removing code tokens, indicating that explicit discrete semantic units are the most consequential component.

The paper also reports codebook analyses and t-SNE visualizations indicating that coarse layers are sparse and globally uniform while deeper layers are more discriminative, and that GS-Quant tokens occupy a compact but integrated region of the LLM token space. This suggests that the framework is not only compressing entity representations but also reformatting them into a vocabulary structure more compatible with LLM sequence processing.

## 6. Implementation profile, limitations, and related usage context

GS-Quant’s quantization pretraining uses embedding dimension **512**, encoder layers **[512, 512, 512]**, codebook layers **4**, codebook size **1024**, learning rate \(1\times10^{-4}\), commitment loss weight **0.25**, \(\lambda_1=0.8\), and \(\lambda_2=0.4\). For FB15k-237, maximum pretraining steps are **500** with batch size **14,541**; for WN18RR, maximum steps are **1000** with batch size **16,348**. The framework also reports a checkpoint-selection criterion based on codebook entropy,
\[
\mathcal{Y} = - \frac{1}{M} \sum_{m=1}^{M} \sum_{k=1}^{K} p_k^m \log p_k^m,
\]
where \(p_k^m\) is the empirical activation frequency of code \(k\) in codebook \(m\). Higher entropy is reported to correlate positively with downstream KGC performance [2604.21649].

The paper explicitly notes several limitations. Performance and efficiency remain bounded by the pretrained LLM backbone. The learned code semantics are benchmark-specific, with transfer to substantially different graphs or domains left unclear. The evaluation scope is concentrated on link prediction rather than broader generative reasoning tasks. The text also implies additional failure modes: hierarchy quality depends on clustering quality; weak initial embeddings can degrade semantic supervision; entities with ambiguous or sparse descriptions may be harder to align; and a discrete bottleneck may still lose information for subtle distinctions.

A further point of clarification is terminological. In current arXiv usage, “GS-Quant” refers specifically to **Granular Semantic and Generative Structural Quantization for Knowledge Graph Completion**, and should be distinguished from unrelated methods with similar abbreviations, such as **GSQ** for Gumbel-Softmax weight quantization of LLMs [2604.18556] and **GSQ-Tuning** for group-shared-exponent fully quantized fine-tuning [2502.12913]. Within KGC, however, GS-Quant’s distinctive contribution is precise: it treats discrete codes not as flat compressed indices, but as semantically stratified and causally structured descriptors that can be consumed by an autoregressive language model.

Source: https://www.emergentmind.com/topics/gs-quant