GLiDRE: Lightweight Document Relation Extraction

• GLiDRE is a lightweight, bi-encoder document-level relation extraction model that reframes relation extraction as representation matching between entity pairs and label embeddings.
• It employs separate DeBERTaV3 and BGE-Large V1.5 encoders to independently process documents and relation labels, effectively handling long-range context and bypassing input length limits.
• GLiDRE achieves state-of-the-art few-shot performance on Re-DocRED and competitive results in fully supervised and zero-shot settings, demonstrating efficiency compared to larger models.

Searching arXiv for the primary paper and closely related DocRE work to ground the article. Searching arXiv for (Armingaud et al., 1 Aug 2025). GLiDRE, short for Generalist Lightweight model for Document-level Relation Extraction, is a document-level relation extraction model introduced to address settings in which relations between entities span multiple sentences, require cross-sentence reasoning, and depend on long-range context and interactions among many entities. It reframes document-level relation extraction as representation matching between entity-pair contexts and relation labels in a shared latent space, combining a bi-encoder architecture, optional localized context pooling, and focal loss. In the reported experiments on Re-DocRED, GLiDRE attains state-of-the-art few-shot performance and competitive fully supervised and zero-shot performance relative to substantially larger instruction-tuned LLMs (Armingaud et al., 1 Aug 2025).

1. Problem formulation and task setting

Document-level relation extraction (DocRE) is defined over a document represented as a sequence of tokens $D$ and a set of $m$ pre-identified entities $E = \{e_1, e_2, \dots, e_m\}$, where each entity may have multiple mentions. The objective is to identify valid relational triplets $(e_h, r, e_t)$, where $e_h, e_t \in E$, $h$ may equal $t$, and $r$ is the relation holding between them. GLiDRE treats DocRE as a multi-label classification problem: for each candidate pair $(e_h, e_t)$, multiple relations may hold simultaneously (Armingaud et al., 1 Aug 2025).

The setting is difficult for both algorithmic and statistical reasons. In benchmarks such as DocRED and Re-DocRED, documents contain on average 19 entities and roughly 390 candidate pairs per text. This produces a large number of negative pairs and severe class imbalance. The paper emphasizes that in realistic information extraction scenarios, including biomedical and industrial text, the number of candidate entity pairs grows quadratically with the number of entities, while the evidence required to resolve a relation may be distributed across the document rather than confined to a single sentence (Armingaud et al., 1 Aug 2025).

GLiDRE is positioned against a line of DocRE systems largely based on the ATLOP architecture. Its specific motivation comes from GLiNER, which showed that compact bidirectional encoder-based models can outperform much larger LLMs in zero-shot named entity recognition by matching label representations to textual spans in a latent space. GLiDRE transfers that representation-matching idea from NER to document-level relation extraction (Armingaud et al., 1 Aug 2025).

2. Architectural design and representation matching

GLiDRE extends the bi-encoder version of GLiNER by employing two separate transformer encoders: one for the document and another for relation labels. The document encoder is DeBERTaV3 Large, the label encoder is BGE-Large V1.5, and the final model has approximately 800 million parameters. This separation avoids concatenating label text with document tokens, which bypasses context length limitations for labels and permits an effectively unlimited label set at inference time. The trade-off is that the model sacrifices label-document cross-attention and may find semantically similar labels harder to disambiguate (Armingaud et al., 1 Aug 2025).

Token and label representations follow a simple but specific design. Label embeddings are obtained by mean-pooling over the token representations that constitute label names. If the label encoder output dimension differs from the configured latent space dimension, the label embedding is passed through a two-layer feedforward network. On the document side, GLiDRE follows GLiNER and standard NER practice: when a word is split into subwords, the representation of the first subword is used. Entity representations are built from contextual embeddings $H \in \mathbb{R}^{L \times D}$ by pooling mention-token embeddings into mention-level vectors and then averaging across all mentions of the entity (Armingaud et al., 1 Aug 2025).

Candidate relation instances are formed over all pairs of entities in the document, without restricting to same-sentence pairs. The encoder contextualizes over the full document input window. For a candidate pair $m$0, the relation representation is

$m$1

where $m$2 denotes concatenation. GLiDRE then scores a candidate relation type $m$3 by matching the pair representation against the corresponding label embedding $m$4:

$m$5

with $m$6. The model assigns relation $m$7 to $m$8 when

$m$9

and the experiments set the global threshold $E = \{e_1, e_2, \dots, e_m\}$0 to 0.5 (Armingaud et al., 1 Aug 2025).

The architecture optionally incorporates Localized Context Pooling (LOP), inspired by ATLOP, to focus on context relevant to an entity pair. Let $E = \{e_1, e_2, \dots, e_m\}$1 be the attention matrix from the last document-encoder layer, averaged across heads. For entities $E = \{e_1, e_2, \dots, e_m\}$2 and $E = \{e_1, e_2, \dots, e_m\}$3, GLiDRE averages attention over mention tokens to obtain $E = \{e_1, e_2, \dots, e_m\}$4 and $E = \{e_1, e_2, \dots, e_m\}$5, then forms a joint attention vector

$E = \{e_1, e_2, \dots, e_m\}$6

After normalization, it computes a localized context vector

$E = \{e_1, e_2, \dots, e_m\}$7

The head and tail entity representations are then refined as

$E = \{e_1, e_2, \dots, e_m\}$8

$E = \{e_1, e_2, \dots, e_m\}$9

after which the same concatenation-and-FFN step is used to form $(e_h, r, e_t)$0. Compared with ATLOP, GLiDRE retains the basic idea of localized context pooling but replaces ATLOP’s bilinear classifier and adaptive threshold class with label embeddings, a dot-product score, focal loss, and a global threshold (Armingaud et al., 1 Aug 2025).

3. Training objective, synthetic pretraining, and inference

GLiDRE addresses class imbalance with focal loss,

$(e_h, r, e_t)$1

where $(e_h, r, e_t)$2 is the model probability for the ground-truth class, $(e_h, r, e_t)$3 is the focusing parameter, and $(e_h, r, e_t)$4 balances positive and negative examples. The paper reports an ablation in which an ATLOP-style adaptive thresholding mechanism was adapted to the bi-encoder by adding an MLP that predicts a threshold logit from the relation embedding and a global context over relation types. This degraded performance by about 1.4 F1 points relative to focal loss, and the optimal global $(e_h, r, e_t)$5 consistently converged near 0.5, which the authors interpret as evidence that dynamic thresholds were unnecessary in this setup (Armingaud et al., 1 Aug 2025).

A central component of the training pipeline is synthetic DocRE pretraining. GLiDRE is pretrained on a corpus constructed semi-automatically, inspired by UniversalNER. Documents are sampled from FineWeb, and Mistral-Small-24B-Instruct-2501 is prompted to generate structured JSON annotations of entities and relations. Malformed JSON and overly long documents are filtered out. The resulting corpus contains 136,404 documents and 76,497 unique relation types, with frequent types including IS_LOCATED_IN, WORKS_FOR, PART_OF, and CONTAINS. This pretraining regime is particularly compatible with the bi-encoder design, because relation labels are represented independently and can be embedded flexibly (Armingaud et al., 1 Aug 2025).

Inference is straightforward. For each entity pair, GLiDRE computes $(e_h, r, e_t)$6 for every candidate relation type embedding in the label set and selects all labels whose score exceeds $(e_h, r, e_t)$7. Because label embeddings are computed by the label encoder independently of the document encoder, they can be precomputed for faster inference. The same property makes it easy to extend or change the label set without retraining the document encoder (Armingaud et al., 1 Aug 2025).

4. Evaluation protocol and empirical results on Re-DocRED

The main evaluation uses Re-DocRED, described as a human-revised version of DocRED that corrects false negatives, logical inconsistencies, and coreference errors across the 4,053 documents in the original splits, while retaining the same relation schema and splits. GLiDRE is evaluated in three regimes: fully supervised fine-tuning on the full training set, few-shot fine-tuning on subsets with $(e_h, r, e_t)$8, and zero-shot comparison against large instruction-tuned LLMs using a constrained triplet-generation prompt with vLLM’s regex-constrained decoding and temperature set to 0. The evaluation follows Re-DocRED conventions, reporting micro-averaged F1 over relation triples and Ign_F1, which ignores triples appearing in both training and test sets (Armingaud et al., 1 Aug 2025).

In the fully supervised setting, GLiDRE achieves Dev F1 77.76±0.35, Dev Ign F1 76.70±0.37, Test F1 77.83±0.23, and Test Ign F1 76.80±0.22. These results are competitive with established DocRE systems. The reported comparison values are ATLOP: Test F1 77.81±0.71, Test Ign F1 76.13±0.28; KD-DocRE: Test F1 78.28±0.72, Test Ign F1 77.60±0.25; DREEAM: Test F1 80.20±0.45, Test Ign F1 78.56±0.39; and TTM-RE: Test F1 79.95±0.13, Test Ign F1 78.20±0.34. The paper also states that GLiDRE outperforms LLM-based methods fine-tuned or prompted for DocRE, such as LMRC LLaMA2-13B-Chat with Test F1 74.63, by over 3 F1 points, despite being significantly smaller and more efficient (Armingaud et al., 1 Aug 2025).

The few-shot results are the paper’s strongest empirical claim. With only 10 training samples, GLiDRE achieves 41.73±2.94 F1, compared with 27.07±5.81 for DREEAM and 29.48±3.91 for ATLOP. The advantage remains through N=100, where GLiDRE reaches 60.04±0.50, versus 57.17±0.37 for ATLOP and 58.14±0.86 for DREEAM. At N=500 and N=1000, the gap narrows and the baselines become competitive, which the paper attributes to GLiDRE’s architectural bias being especially useful when labeled data is scarce (Armingaud et al., 1 Aug 2025).

In the zero-shot regime, GLiDRE records Test F1 17.32 and Ign F1 16.41. The reported LLM comparison points are Qwen 2.5 72B: Test F1 18.00, Ign F1 17.86; Mistral-Large 123B: Test F1 18.61, Ign F1 18.50; Llama 3.3 70B: Test F1 15.81, Ign F1 15.71; and Mistral-Small 24B: Test F1 14.38, Ign F1 14.27. The paper further notes that GLiDRE dramatically outperforms earlier reported GPT-3.5 Turbo results with Test F1 6.68. Within the scope of the reported benchmark, this indicates that a specialized encoder-based architecture can remain competitive in zero-shot DocRE without relying on very large instruction-tuned decoders (Armingaud et al., 1 Aug 2025).

5. Ablations, efficiency, and reproducibility

The ablation study isolates several parts of the design. First, although ATLOP had reported small gains from LogSumExp pooling as a smooth max approximation,

$(e_h, r, e_t)$9

GLiDRE performs better with conventional mean pooling: Test F1 76.86±0.15 for LogSumExp versus 77.83±0.23 for mean pooling. The paper suggests that the difference is likely related to the mismatch between GLiDRE’s bi-encoder scoring mechanism and ATLOP’s bilinear classifier (Armingaud et al., 1 Aug 2025).

Second, both synthetic pretraining and localized context pooling contribute positively. Removing synthetic pretraining reduces Test F1 by about 0.7 points, to 77.15±0.42, whereas disabling LOP reduces it by about 0.2 points, to 77.61±0.09. Replacing focal loss with the adapted ATLOP-style adaptive thresholding mechanism yields 75.74±1.05 versus 77.15±0.42 in the no-pretraining comparison, reinforcing the conclusion that focal loss plus a global threshold is better suited to GLiDRE’s scoring mechanism and class imbalance (Armingaud et al., 1 Aug 2025).

The model is described as “lightweight” relative to the LLMs used for DocRE rather than small in absolute size. At inference it runs with under 10 GB of GPU memory, whereas instruction-tuned 70B–123B models often require more than 300 GB and multiple high-memory GPUs, together with complex prompting and constrained decoding to enforce structured outputs. GLiDRE, by contrast, produces structured predictions natively through label matching. Its bi-encoder design also supports an effectively unbounded label set at inference time, because new relation labels can be embedded independently (Armingaud et al., 1 Aug 2025).

Implementation details are reported in a reproducible form. Experiments were run on a single NVIDIA H100 with batch size 16. Pretraining takes approximately 24 hours for 50,000 steps, and Re-DocRED fine-tuning takes approximately 3.5 hours for 10,000 steps. The learning rates are $e_h, e_t \in E$0 for the encoders and $e_h, e_t \in E$1 for the remaining layers. The threshold $e_h, e_t \in E$2 is fixed at 0.5, and the best dev-set checkpoint is used for test evaluation. The code is publicly available at https://github.com/robinarmingaud/glidre, and few-shot results are averaged over 5 seeds per training-set size (Armingaud et al., 1 Aug 2025).

6. Limitations, scope, and prospective extensions

GLiDRE inherits several constraints typical of document-level relation extraction and encoder-based modeling. The first is combinatorial: candidate pairs scale quadratically with the number of entities, which increases memory usage for dense documents and can lead to out-of-memory failures. The second is contextual: although the bi-encoder can accommodate more labels, the document encoder’s maximum sequence length, exemplified in the paper by 512 tokens for DeBERTa, limits the input span; truncation or chunking can break long-distance dependencies. The third is representational: because scoring is independent for each $e_h, e_t \in E$3 triple, the model does not explicitly capture inter-label interactions, which makes semantically similar relations harder to separate without cross-attention or some form of joint modeling (Armingaud et al., 1 Aug 2025).

The evaluation scope is also limited. GLiDRE assumes gold entities and coreference chains and does not perform named entity recognition or coreference resolution itself. Consequently, end-to-end information extraction performance depends on upstream modules. The paper also notes that it does not include explicit qualitative case studies of cross-sentence predictions, although it does provide the prompt templates used for zero-shot LLM inference and synthetic pretraining generation (Armingaud et al., 1 Aug 2025).

These constraints delimit what the reported results establish. They show that representation matching between entity-pair contexts and relation labels can be effective for DocRE, especially in few-shot and competitive zero-shot settings, but they do not establish an end-to-end document information extraction pipeline. The authors identify two future directions: improving synthetic relation generation to narrow remaining performance gaps, and investigating dynamic thresholding methods tailored specifically to the bi-encoder setup, with the aim of improving robustness across diverse relation schemas (Armingaud et al., 1 Aug 2025).