GLiREL: Zero-Shot Relation Extraction

• GLiREL is a lightweight model for zero-shot relation extraction that integrates joint encoding with span-pair extraction to streamline processing.
• It leverages synthetic pretraining from large-scale data to address annotated data scarcity and boost relation classification accuracy.
• The architecture achieves high computational efficiency by processing all entity pairs and relations in a single forward pass.

GLiREL (Generalist Lightweight Model for Zero-Shot Relation Extraction) is a neural architecture and training paradigm designed to efficiently perform zero-shot relation classification between multiple entities in a single forward pass. Optimized for both accuracy and computational speed, GLiREL integrates transformer-based joint encoding, a lightweight span-and-pair extractor, and specifically designed synthetic pretraining protocols to advance state-of-the-art zero-shot relation extraction benchmarks (Boylan et al., 6 Jan 2025).

1. Architecture and Joint Encoding Strategy

GLiREL employs a tri-module design centered around a joint encoder paradigm. Its architecture consists of:

• A bidirectional DeBERTa V3-large transformer encoder that simultaneously processes the $M$ relation label prompts and the $N$ input text tokens.
• A lightweight extractor that utilizes hidden representations to construct embeddings for all $E$ extracted entity spans and their $E\cdot(E-1)$ ordered pairs.
• A final scoring layer computing the sigmoid of the dot-product between each entity-pair embedding and each relation label embedding.

The input is formatted as $$[\text{REL}]~t_0~[\text{REL}]~t_1~\dots~[\text{REL}]~t_{M-1}~[\text{SEP}]~x_0~x_1~\dots~x_{N-1}$$, paired with a list of entity spans. The transformer outputs pooled label representations $p_0,\dots,p_{M-1}\in\mathbb{R}^D$ and token representations $h_0,\dots,h_{N-1}\in\mathbb{R}^D$. These are projected through two-layer feedforward networks (FFN) to generate label embeddings $q_t$ and entity span embeddings $e_{ij} = \mathrm{FFN}(h_i \oplus h_j)$, which are subsequently paired: $\kappa_{uv} = \mathrm{FFN}(e_u \oplus e_v)$. Optional refinement layers apply cross- and self-attention between $N$0 and $N$1. Scoring for each entity pair and relation is:

$N$2

interpreted as the probability that entity pair $N$3 expresses relation $N$4. This architecture allows for parallel evaluation of all entity pairs and relations in a single transformer pass, unlike earlier pairwise or sequential approaches (Boylan et al., 6 Jan 2025).

2. Zero-Shot Training Paradigm

GLiREL is explicitly structured for zero-shot generalization. Relation types are split into “seen” ($N$5) and “unseen” ($N$6) categories, with $N$7. Training instances comprise the input text, entity spans, the set $N$8, and a binary matrix of gold labels $N$9. The model is trained via pairwise, per-label binary cross-entropy:

$E$0

To stabilize learning, the input sequence is prepended with up to $E$1 labels (fake negatives sampled if $E$2), and random shuffling/dropping of labels regularizes the joint encoding space, enhancing label-set robustness.

3. Synthetic Data Generation Protocol

To overcome the scarcity and domain-bias of annotated relations, GLiREL leverages large-scale synthetic pretraining:

1. Unannotated samples from the FineWeb corpus are selected.
2. Named entity recognition (NER) is applied to identify mention sets $E$3 per document $E$4; all ordered pairs $E$5 are constructed.
3. An open LLM (Mistral 7B-Instruct) is prompted: for each entity pair, determine the relation or “NO RELATION”.
4. The LLM’s output is parsed and compiled into a synthetic dataset.
5. Overlaps with benchmark relation sets are filtered out to maintain zero-shot integrity.

This protocol yields approximately 25 million labeled pairs over roughly 63,000 texts, forming the backbone for pretraining before task-specific fine-tuning (Boylan et al., 6 Jan 2025).

4. Inference and Computational Efficiency

For inference, the architecture accepts arbitrary sets of zero-shot relation prompts and entity spans, enabling processing of all pairs and relations in one transformer call. For $E$6 relations and $E$7 entities, the transformer output directly constructs the tensor $E$8 with $E$9. This reduces inference complexity to $E\cdot(E-1)$0 as compared to $E\cdot(E-1)$1 in earlier one-pair-per-forward architectures.

GLiREL’s parameter count (DeBERTa V3-large backbone at 467M parameters) exceeds some baselines (TMC-BERT: 109M), yet its speed is markedly higher. On GPU inference (for $E\cdot(E-1)$2 relations) over the Wiki-ZSL benchmark, GLiREL attains 47.6 sentences/sec, compared to TMC-BERT’s 1.41 and RelationPrompt’s 2.06. CPU throughput evidences similar scaling (Boylan et al., 6 Jan 2025).

5. Evaluation and Performance Metrics

GLiREL’s experimental regime spans multiple datasets:

• Wiki-ZSL: 94,383 sentences, 113 relations, $E\cdot(E-1)$3183K annotated pairs.
• FewRel: 56,000 sentences, 80 relations.

Zero-shot splits select $E\cdot(E-1)$4 relations as unseen, report macro-averaged precision, recall, and F1 over five random seeds per split.

$E\cdot(E-1)$5	Model	Wiki-ZSL F1	FewRel F1
5	TMC-BERT	88.92	93.62
	GLiREL	62.80	81.21
	GLiREL + synthetic pretrain	83.28	94.20
10	TMC-BERT	81.23	85.68
	GLiREL	68.89	80.14
	GLiREL + synthetic pretrain	83.67	87.60
15	TMC-BERT	73.77	81.00
	GLiREL	60.91	70.40
	GLiREL + synthetic pretrain	73.91	84.48

Pretraining on synthetic data uniformly reduces the performance gap with strong baselines and delivers state-of-the-art F1 for zero-shot setups (Boylan et al., 6 Jan 2025).

6. Integration with Atomic Proposition Extraction

GLiREL benefits from upstream atomic proposition extraction as an intermediate preprocessing step (Pommeret et al., 3 Apr 2026). The propositioner (MPropositionneur-V2) decomposes sentences into minimal, semantically autonomous units, which are then fed to GLiREL for entity-centric relation extraction. Quantitative ablation shows that “Prop” mode (using atomized inputs) increases relation recall (58.5% vs. 57.6%) at a cost to entity recall, while a combination pipeline (“Comb”—direct extraction with fallback to atomized) recovers entity recall and boosts relation recall to 59.3%. These effects are consistent across English, French, German, Spanish, Portuguese, and Italian, evidencing GLiREL’s adaptability within multilingual, proposition-augmented pipelines.

7. Applications and Impact

GLiREL’s architecture is primed for knowledge graph construction, information extraction, and large-scale automated knowledge base population in zero-shot regimes. Its efficiency, combined with the accuracy gains enabled by synthetic pretraining and atomic proposition integration, address key limitations in scaling relation extraction to novel domains and label sets (Boylan et al., 6 Jan 2025, Pommeret et al., 3 Apr 2026). The joint-encoding approach and pretraining protocol constitute a template for further advances in generalist, relation-centric NLP models.

---

GLiREL exemplifies the state of the art in scalable, efficient, and accurate zero-shot relation extraction, uniting transformer-based joint encoding, synthetic LLM-derived pretraining, and proposition-aware enhancement strategies. Its design and empirical profile inform both theoretical and applied research in relation extraction and entity-centric information extraction systems.