---
title: 'FarsEval-PKBETS: Persian NLP Benchmark'
url: https://www.emergentmind.com/topics/farseval-pkbets-benchmark
type: topic
---

# FarsEval-PKBETS: Persian NLP Benchmark

FarsEval-PKBETS denotes a suite of rigorously curated benchmarks targeting two fundamental tasks in Persian natural language processing: knowledge base population (KBP) for the FarsBase Persian knowledge graph, and comprehensive evaluation of large language models (LLMs) on a culturally, linguistically, and domain-diverse set of reasoning tasks. The term delineates both (1) the first large-scale, human-verified corpus for Persian KBP ([2005.01879]) and (2) a diverse, expert-driven challenge set for assessing LLMs on a spectrum of domains critical for Persian applications ([2504.14690]).

## 1. Motivation and Scope

The FarsEval-PKBETS benchmarks arose from the acute scarcity of representative, high-fidelity resources for evaluating language technologies in Persian. While languages such as English benefit from mature KBP corpora and a surfeit of LLM evaluation benchmarks spanning diverse knowledge, reasoning, and generative tasks, Persian has historically lacked both KB-scale gold datasets and challenge sets reflecting its unique cultural, linguistic, and regulatory contexts. Existing evaluation resources for Persian tend to derive from translated English datasets, focus narrowly on single domains (e.g., standardized exams, medicine), and rarely probe domain-transfer, generation skills, or culturally situated reasoning ([2504.14690]). FarsEval-PKBETS addresses these gaps by offering:

- A human-verified KBP gold standard (22,015 triples) supporting system-level KBP evaluation ([2005.01879]).
- An LLM benchmark (4,000 Q–A pairs) spanning medicine, law, religion, language, encyclopedic knowledge, social and human preferences, ethics, bias, and generative text tasks, with explicit emphasis on Persian-specific content ([2504.14690]).

## 2. FarsEval-PKBETS for Knowledge Base Population

The KBP component is built on the FarsBase-KBP gold corpus—the first fully human-verified evaluation set for Persian knowledge graph construction ([2005.01879]). Corpus creation entailed crawling all Persian Wikipedia articles containing both subject and object of an existing FarsBase triple, followed by expert annotation selecting the correct predicate from 406 predicate classes. The annotation schema encompasses:

- Normalization of subjects and objects to FarsBase entity IDs (via ParsEL).
- Use of FarsBase ontology URIs (e.g., fkgo:birthPlace) as predicates.
- Enrichment of each instance with tokens, POS tags, NER tags, dependency parse, linked entity types, and FarsBase classes.

Key corpus statistics:

| Metric             | Value            |
|--------------------|-----------------|
| Sentences/triples  | 22,015          |
| Unique predicates  | 406             |
| Unique entities    | 11,024          |
| Avg. examples/pred.| 54.2 (σ≈102.1)  |

Entity types (subjects + objects): 45.2% Person, 20.1% Location, 14.8% Organization, 9.3% Work, 10.6% Miscellaneous.

Predicate coverage includes both dense and rare relations, with the top-10 predicates accounting for significant frequency diversity.

The corpus is randomly stratified by predicate: 70% train, 15% dev, 15% test.

## 3. FarsEval-PKBETS for Large Language Model Evaluation

The LLM evaluation benchmark comprises 4,000 Persian question–answer items, designed to probe factual knowledge, reasoning, and text generation in a variety of formats:

| Format        | Count | Percentage |
|---------------|-------|------------|
| Multiple-choice (MCQ) | 2,000 | 50% |
| Short-answer  (SAQ)   | 1,200 | 30% |
| Descriptive   (DQ)    | 800   | 20% |

Twelve primary domains are covered, including medicine (500), law (350), religion (250), Persian language (250), encyclopedic knowledge (300), social knowledge (500), text generation (450), and others. Sourcing draws from Iranian residency exams, official legal texts, religious jurisprudence, and real-world/social media scenarios. Annotation emphasizes exploitation of Persian-specific ambiguity, idiomatic expressions, and sub-dialectal variation.

A web-based annotation and review platform (“Saba”) is used, supporting live LLM evaluation, chain-of-thought labeling, metadata tracking, and a public leaderboard.

## 4. Benchmark Evaluation Methodology

For KBP, model performance is assessed at the triple (subject, predicate, object) level. Metrics incorporate:

- Precision ($P$), recall ($R$), and F₁ at strict triple match level:
  $$
  P = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP}}, \quad
  R = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FN}}, \quad
  F_1 = 2\,\frac{P\,R}{P + R}
  $$
- Micro- and macro-averages over predicates:
  $$
  P_{\text{micro}} = \frac{\sum_c \mathrm{TP}_c}{\sum_c (\mathrm{TP}_c + \mathrm{FP}_c)}, \quad
  P_{\text{macro}} = \frac{1}{|C|}\sum_c P_c
  $$
  Analogous formulas apply for $R$ and $F_1$.
- Area under the precision-recall curve (AUC-PR).
- Extraction throughput (triples/sentence).

For LLMs, two accuracy metrics are used ([2504.14690]):

- Strict accuracy: only fully correct answers are counted.
- Lenient accuracy: includes “semi-correct” responses (e.g., minor errors or correct answer/incorrect label).

The task setup is zero-shot; models are prompted exclusively in Persian and tested across all formats and domains without few-shot examples.

## 5. System Architectures and Module Performance

### KBP System

The FarsBase-KBP system ([2005.01879]) operates in five stages:

1. Web crawling and preprocessing
2. Entity linking (ParsEL: unsupervised, mixing contextual and surface features)
3. Relation/information extraction via an ensemble of six modules: PredPatt (Universal Dependencies-based Open IE), DependencyPattern (frequent syntactic templates), PSIE (combined DEP/CONSTIT parsing), RePersian (regex from Dadegan Treebank), TokensRegex (handcrafted rules), and Distant Supervision (PCNN, multi-instance learning)
4. Canonicalization via infobox alignment and lexicon matching
5. Knowledge fusion (ensemble: accept if found by ≥2 modules, or confidence ≥0.90 in one module)

Human experts verify all candidate triples before addition to FarsBase.

| Module           | Precision | Recall  | F₁      | Triples/Sentence |
|------------------|-----------|---------|---------|------------------|
| DependencyPattern| 0.7474    | 0.0032  | 0.0064  | 0.019            |
| DistantSup       | 0.2610    | 0.2104  | 0.2330  | 0.806            |
| PredPatt         | 0.1368    | 0.0006  | 0.0012  | 3.015            |
| RePersian        | 0.1747    | 0.0023  | 0.0046  | 0.357            |
| TokensRegex      | 0.7829    | 0.1502  | 0.2520  | 1.697            |
| PSIE             | 0.1626    | 0.0043  | 0.0083  | 2.035            |
| Fusion (θ=0.90)  | 0.7313    | 0.1779  | 0.2862  | 1.812            |

AUC-PR for the fusion system on TEST is 0.412. Micro and macro F₁ on TEST are 0.285 and 0.231, respectively. Ablation experiments demonstrate the precision-recall trade-off as the fusion confidence threshold θ varies.

### LLM Results

Three LLMs were evaluated ([2504.14690]):

| Model        | Strict Acc. | Lenient Acc. |
|--------------|-------------|--------------|
| Llama 3-70B  | 0.47        | 0.54         |
| PersianMind  | 0.19        | 0.23         |
| Dorna (8B)   | 0.30        | 0.39         |

Performance varies across domains: best results on NLP paraphrase (0.70 accuracy), and the highest generative successes in personality-centric and style-centric tasks. The weakest LLM performance is in emergency medicine (0.14 accuracy), alternative medicine (0.27), and Persian grammar/proverbs (0.24). Even Llama 3-70B remains below 0.60 strict accuracy in core domain knowledge tasks.

## 6. Error Analysis and Challenges

Systematic limitations identified in KBP include low recall and overgeneration in certain extractors, noisy distant supervision, entity linking errors, shortcomings in parsing Persian’s morphosyntax, and failures in canonicalization especially for idiomatic verbs (~12% error rate). LLM evaluation exposes deficiencies in justification generation (with 43% of multiple-choice chain-of-thought justifications inconsistent), fluency and factuality in free-form answers, and pronounced struggles with genuinely local or specialized knowledge (medical, legal), as well as cultural reasoning (ethics, bias, respecting rights).

Common KBP failure cases: nested and coordinative structures (e.g., “A and B founded C”), implicit relations (requiring world knowledge), and non-standard Persian orthography in Wikipedia source texts.

Recommendations for the KBP track include expanding data sources, integrating multilingual embeddings (mBERT), crowd-based validation for ambiguous cases, semi-automatic rule induction for extraction, more robust canonicalization (e.g., via ELMo clustering), and second-tier negative example releases.

For LLMs, the data highlights persistent bottlenecks in Persian-centric domain adaptation and reasoning, indicating the need for deeper, domain-specific fine-tuning and more deliberate integration of local factuality and reasoning priors.

## 7. Impact and Prospects

FarsEval-PKBETS sets new standards for rigor, coverage, and challenge in Persian-language NLP evaluation ([2005.01879], [2504.14690]). Its KBP corpus has enabled the first micro–macro comparative evaluation of Persian KBP systems at scale, with established methodology for per-predicate metrics and detailed ablation of fusion rules. The LLM benchmark provides a uniquely demanding set of generation, reasoning, and cultural adaptation tasks, sharply exposing the sub-50% accuracy ceiling of current state-of-the-art models.

Potential extensions include dynamic updates to reflect legal and regulatory change, new modules for low-resource Persian dialects, and more rigorous generative and interactive evaluation (multi-turn legal/medical consultation scenarios). The released leaderboards and annotation APIs further support transparent, ongoing benchmarking.

FarsEval-PKBETS, by combining exhaustive expert-driven annotation, diverse domain/task distribution, and public infrastructure for evaluation, establishes a replicable framework for benchmarking in other under-resourced languages and offers empirical guidance for subsequent model and data development in Persian.

Source: https://www.emergentmind.com/topics/farseval-pkbets-benchmark