---
title: 'GLiDRE: Lightweight Document Relation Extraction'
url: https://www.emergentmind.com/topics/glidre
type: topic
---

# GLiDRE: Lightweight Document Relation Extraction

Searching arXiv for the primary paper and closely related DocRE work to ground the article.
Searching arXiv for 2508.00757.
GLiDRE, short for **Generalist Lightweight model for Document-level Relation Extraction**, is a document-level relation extraction model introduced to address settings in which relations between entities span multiple sentences, require cross-sentence reasoning, and depend on long-range context and interactions among many entities. It reframes document-level relation extraction as representation matching between entity-pair contexts and relation labels in a shared latent space, combining a bi-encoder architecture, optional localized context pooling, and focal loss. In the reported experiments on Re-DocRED, GLiDRE attains state-of-the-art few-shot performance and competitive fully supervised and zero-shot performance relative to substantially larger instruction-tuned language models [2508.00757].

## 1. Problem formulation and task setting

Document-level relation extraction (DocRE) is defined over a document represented as a sequence of tokens $D$ and a set of $m$ pre-identified entities $E = \{e_1, e_2, \dots, e_m\}$, where each entity may have multiple mentions. The objective is to identify valid relational triplets $(e_h, r, e_t)$, where $e_h, e_t \in E$, $h$ may equal $t$, and $r$ is the relation holding between them. GLiDRE treats DocRE as a multi-label classification problem: for each candidate pair $(e_h, e_t)$, multiple relations may hold simultaneously [2508.00757].

The setting is difficult for both algorithmic and statistical reasons. In benchmarks such as DocRED and Re-DocRED, documents contain on average **19 entities** and roughly **390 candidate pairs per text**. This produces a large number of negative pairs and severe class imbalance. The paper emphasizes that in realistic information extraction scenarios, including biomedical and industrial text, the number of candidate entity pairs grows quadratically with the number of entities, while the evidence required to resolve a relation may be distributed across the document rather than confined to a single sentence [2508.00757].

GLiDRE is positioned against a line of DocRE systems largely based on the ATLOP architecture. Its specific motivation comes from GLiNER, which showed that compact bidirectional encoder-based models can outperform much larger large language models in zero-shot named entity recognition by matching label representations to textual spans in a latent space. GLiDRE transfers that representation-matching idea from NER to document-level relation extraction [2508.00757].

## 2. Architectural design and representation matching

GLiDRE extends the bi-encoder version of GLiNER by employing two separate transformer encoders: one for the document and another for relation labels. The document encoder is **DeBERTaV3 Large**, the label encoder is **BGE-Large V1.5**, and the final model has approximately **800 million parameters**. This separation avoids concatenating label text with document tokens, which bypasses context length limitations for labels and permits an effectively unlimited label set at inference time. The trade-off is that the model sacrifices label-document cross-attention and may find semantically similar labels harder to disambiguate [2508.00757].

Token and label representations follow a simple but specific design. Label embeddings are obtained by mean-pooling over the token representations that constitute label names. If the label encoder output dimension differs from the configured latent space dimension, the label embedding is passed through a two-layer feedforward network. On the document side, GLiDRE follows GLiNER and standard NER practice: when a word is split into subwords, the representation of the first subword is used. Entity representations are built from contextual embeddings $H \in \mathbb{R}^{L \times D}$ by pooling mention-token embeddings into mention-level vectors and then averaging across all mentions of the entity [2508.00757].

Candidate relation instances are formed over **all pairs of entities** in the document, without restricting to same-sentence pairs. The encoder contextualizes over the full document input window. For a candidate pair $(e_h, e_t)$, the relation representation is

$$
h_r = \text{FFN}(h_{e_h} \otimes h_{e_t}),
$$

where $\otimes$ denotes concatenation. GLiDRE then scores a candidate relation type $t$ by matching the pair representation against the corresponding label embedding $h_t$:

$$
s(e_h, e_t, t) = \sigma(h_r^\top h_t),
$$

with $\sigma(x) = (1 + e^{-x})^{-1}$. The model assigns relation $t$ to $(e_h, e_t)$ when

$$
s(e_h, e_t, t) > \tau,
$$

and the experiments set the global threshold $\tau$ to **0.5** [2508.00757].

The architecture optionally incorporates **Localized Context Pooling (LOP)**, inspired by ATLOP, to focus on context relevant to an entity pair. Let $A \in \mathbb{R}^{L \times L}$ be the attention matrix from the last document-encoder layer, averaged across heads. For entities $e_h$ and $e_t$, GLiDRE averages attention over mention tokens to obtain $A_h$ and $A_t$, then forms a joint attention vector

$$
\alpha = A_h \odot A_t.
$$

After normalization, it computes a localized context vector

$$
c_{\text{loc}} = \sum_{i=1}^{L} \frac{\alpha_i}{\sum_{j=1}^{L} \alpha_j} H_i.
$$

The head and tail entity representations are then refined as

$$
h'_{e_h} = \tanh(\text{FFN}_h(h_{e_h} \otimes c_{\text{loc}})),
$$

$$
h'_{e_t} = \tanh(\text{FFN}_t(h_{e_t} \otimes c_{\text{loc}})),
$$

after which the same concatenation-and-FFN step is used to form $h_r$. Compared with ATLOP, GLiDRE retains the basic idea of localized context pooling but replaces ATLOP’s bilinear classifier and adaptive threshold class with label embeddings, a dot-product score, focal loss, and a global threshold [2508.00757].

## 3. Training objective, synthetic pretraining, and inference

GLiDRE addresses class imbalance with **focal loss**,

$$
\text{FL}(p_t) = -\alpha_t (1 - p_t)^\gamma \log(p_t),
$$

where $p_t$ is the model probability for the ground-truth class, $\gamma$ is the focusing parameter, and $\alpha_t$ balances positive and negative examples. The paper reports an ablation in which an ATLOP-style adaptive thresholding mechanism was adapted to the bi-encoder by adding an MLP that predicts a threshold logit from the relation embedding and a global context over relation types. This degraded performance by about **1.4 F1 points** relative to focal loss, and the optimal global $\tau$ consistently converged near **0.5**, which the authors interpret as evidence that dynamic thresholds were unnecessary in this setup [2508.00757].

A central component of the training pipeline is synthetic DocRE pretraining. GLiDRE is pretrained on a corpus constructed semi-automatically, inspired by UniversalNER. Documents are sampled from **FineWeb**, and **Mistral-Small-24B-Instruct-2501** is prompted to generate structured JSON annotations of entities and relations. Malformed JSON and overly long documents are filtered out. The resulting corpus contains **136,404 documents** and **76,497 unique relation types**, with frequent types including **IS_LOCATED_IN**, **WORKS_FOR**, **PART_OF**, and **CONTAINS**. This pretraining regime is particularly compatible with the bi-encoder design, because relation labels are represented independently and can be embedded flexibly [2508.00757].

Inference is straightforward. For each entity pair, GLiDRE computes $s(e_h, e_t, t)$ for every candidate relation type embedding in the label set and selects all labels whose score exceeds $\tau$. Because label embeddings are computed by the label encoder independently of the document encoder, they can be precomputed for faster inference. The same property makes it easy to extend or change the label set without retraining the document encoder [2508.00757].

## 4. Evaluation protocol and empirical results on Re-DocRED

The main evaluation uses **Re-DocRED**, described as a human-revised version of DocRED that corrects false negatives, logical inconsistencies, and coreference errors across the **4,053 documents** in the original splits, while retaining the same relation schema and splits. GLiDRE is evaluated in three regimes: fully supervised fine-tuning on the full training set, few-shot fine-tuning on subsets with $N \in \{1, 5, 10, 50, 100, 500, 1000\}$, and zero-shot comparison against large instruction-tuned language models using a constrained triplet-generation prompt with vLLM’s regex-constrained decoding and temperature set to **0**. The evaluation follows Re-DocRED conventions, reporting micro-averaged **F1** over relation triples and **Ign\_F1**, which ignores triples appearing in both training and test sets [2508.00757].

In the fully supervised setting, GLiDRE achieves **Dev F1 77.76±0.35**, **Dev Ign F1 76.70±0.37**, **Test F1 77.83±0.23**, and **Test Ign F1 76.80±0.22**. These results are competitive with established DocRE systems. The reported comparison values are **ATLOP: Test F1 77.81±0.71, Test Ign F1 76.13±0.28**; **KD-DocRE: Test F1 78.28±0.72, Test Ign F1 77.60±0.25**; **DREEAM: Test F1 80.20±0.45, Test Ign F1 78.56±0.39**; and **TTM-RE: Test F1 79.95±0.13, Test Ign F1 78.20±0.34**. The paper also states that GLiDRE outperforms LLM-based methods fine-tuned or prompted for DocRE, such as **LMRC LLaMA2-13B-Chat** with **Test F1 74.63**, by over **3 F1 points**, despite being significantly smaller and more efficient [2508.00757].

The few-shot results are the paper’s strongest empirical claim. With only **10 training samples**, GLiDRE achieves **41.73±2.94 F1**, compared with **27.07±5.81** for DREEAM and **29.48±3.91** for ATLOP. The advantage remains through **N=100**, where GLiDRE reaches **60.04±0.50**, versus **57.17±0.37** for ATLOP and **58.14±0.86** for DREEAM. At **N=500** and **N=1000**, the gap narrows and the baselines become competitive, which the paper attributes to GLiDRE’s architectural bias being especially useful when labeled data is scarce [2508.00757].

In the zero-shot regime, GLiDRE records **Test F1 17.32** and **Ign F1 16.41**. The reported LLM comparison points are **Qwen 2.5 72B: Test F1 18.00, Ign F1 17.86**; **Mistral-Large 123B: Test F1 18.61, Ign F1 18.50**; **Llama 3.3 70B: Test F1 15.81, Ign F1 15.71**; and **Mistral-Small 24B: Test F1 14.38, Ign F1 14.27**. The paper further notes that GLiDRE dramatically outperforms earlier reported **GPT-3.5 Turbo** results with **Test F1 6.68**. Within the scope of the reported benchmark, this indicates that a specialized encoder-based architecture can remain competitive in zero-shot DocRE without relying on very large instruction-tuned decoders [2508.00757].

## 5. Ablations, efficiency, and reproducibility

The ablation study isolates several parts of the design. First, although ATLOP had reported small gains from **LogSumExp pooling** as a smooth max approximation,

$$
\text{LSE}(\mathbf{x_1,\ldots,x_n}) = \log \left( \sum_{i=1}^{n} \exp(x_i) \right),
$$

GLiDRE performs better with conventional mean pooling: **Test F1 76.86±0.15** for LogSumExp versus **77.83±0.23** for mean pooling. The paper suggests that the difference is likely related to the mismatch between GLiDRE’s bi-encoder scoring mechanism and ATLOP’s bilinear classifier [2508.00757].

Second, both synthetic pretraining and localized context pooling contribute positively. Removing synthetic pretraining reduces **Test F1** by about **0.7 points**, to **77.15±0.42**, whereas disabling LOP reduces it by about **0.2 points**, to **77.61±0.09**. Replacing focal loss with the adapted ATLOP-style adaptive thresholding mechanism yields **75.74±1.05** versus **77.15±0.42** in the no-pretraining comparison, reinforcing the conclusion that focal loss plus a global threshold is better suited to GLiDRE’s scoring mechanism and class imbalance [2508.00757].

The model is described as “lightweight” relative to the LLMs used for DocRE rather than small in absolute size. At inference it runs with **under 10 GB of GPU memory**, whereas instruction-tuned **70B–123B** models often require **more than 300 GB** and multiple high-memory GPUs, together with complex prompting and constrained decoding to enforce structured outputs. GLiDRE, by contrast, produces structured predictions natively through label matching. Its bi-encoder design also supports an effectively unbounded label set at inference time, because new relation labels can be embedded independently [2508.00757].

Implementation details are reported in a reproducible form. Experiments were run on a **single NVIDIA H100** with **batch size 16**. Pretraining takes approximately **24 hours for 50,000 steps**, and Re-DocRED fine-tuning takes approximately **3.5 hours for 10,000 steps**. The learning rates are **$1 \times 10^{-5}$** for the encoders and **$1 \times 10^{-4}$** for the remaining layers. The threshold $\tau$ is fixed at **0.5**, and the best dev-set checkpoint is used for test evaluation. The code is publicly available at **https://github.com/robinarmingaud/glidre**, and few-shot results are averaged over **5 seeds** per training-set size [2508.00757].

## 6. Limitations, scope, and prospective extensions

GLiDRE inherits several constraints typical of document-level relation extraction and encoder-based modeling. The first is combinatorial: candidate pairs scale quadratically with the number of entities, which increases memory usage for dense documents and can lead to out-of-memory failures. The second is contextual: although the bi-encoder can accommodate more labels, the document encoder’s maximum sequence length, exemplified in the paper by **512 tokens** for DeBERTa, limits the input span; truncation or chunking can break long-distance dependencies. The third is representational: because scoring is independent for each $(e_h, e_t, t)$ triple, the model does not explicitly capture inter-label interactions, which makes semantically similar relations harder to separate without cross-attention or some form of joint modeling [2508.00757].

The evaluation scope is also limited. GLiDRE assumes **gold entities and coreference chains** and does not perform named entity recognition or coreference resolution itself. Consequently, end-to-end information extraction performance depends on upstream modules. The paper also notes that it does not include explicit qualitative case studies of cross-sentence predictions, although it does provide the prompt templates used for zero-shot LLM inference and synthetic pretraining generation [2508.00757].

These constraints delimit what the reported results establish. They show that representation matching between entity-pair contexts and relation labels can be effective for DocRE, especially in few-shot and competitive zero-shot settings, but they do not establish an end-to-end document information extraction pipeline. The authors identify two future directions: improving synthetic relation generation to narrow remaining performance gaps, and investigating dynamic thresholding methods tailored specifically to the bi-encoder setup, with the aim of improving robustness across diverse relation schemas [2508.00757].

Source: https://www.emergentmind.com/topics/glidre