PsiloQA: Multilingual Hallucination Benchmark

• PsiloQA is a multilingual QA benchmark that provides fine-grained, token-level hallucination detection across 14 diverse languages.
• It utilizes a fully automated three-stage pipeline to generate and annotate hallucinated spans with high cost efficiency.
• Empirical results show that multilingual mmBERT achieves state-of-the-art cross-lingual performance in detecting hallucinations.

PsiloQA is a large-scale, multilingual question answering (QA) benchmark explicitly designed for span-level hallucination detection in LLMs. Departing from the limitations of previous datasets that are English-only and primarily sequence-level, PsiloQA delivers fine-grained, word-level supervision across 14 typologically diverse languages by exhaustively annotating hallucinated spans within LLM-generated answers. The benchmark is constructed via a fully automated, three-stage pipeline (Wikipedia-based QA pair generation, no-context hallucination induction, and GPT-4o-driven inconsistent span annotation) and supports rigorous evaluation of diverse hallucination detection methodologies. Empirical findings establish encoder-based detectors, particularly multilingual mmBERT, as state-of-the-art in both accuracy and cross-lingual generalization. PsiloQA’s scale and construction methodology confer significant advantages in cost, generalizability, and transferability to related factuality benchmarks (Rykov et al., 6 Oct 2025).

1. Dataset Definition and Composition

PsiloQA addresses the need for granular factuality supervision in QA by pinpointing contiguous word spans, rather than labeling entire generations, that introduce unsupported or contradictory content. The dataset encompasses 70,044 samples partitioned into train (63,792), validation (3,355), and test (2,897) splits. Coverage spans 14 languages: Arabic, Basque, Catalan, Czech, German, English, Spanish, Farsi, Finnish, French, Hindi, Italian, Swedish, and Chinese. Training sample size varies from approximately 1,500 (German) to 23,000 (English), with the remaining languages containing 2,000–7,000 instances each.

A span-level hallucination, for the purposes of PsiloQA, is any sequence of tokens in an LLM answer that either contradicts the gold answer or context, or introduces unsupported facts. Such regions are demarcated using [HAL]…[/HAL] tags.

Language	Training Samples	Validation Samples	Test Samples
English	~23,000	–	–
German	~1,500	–	–
Other (typical)	2,000–7,000	–	–

The dataset structure enables detailed assessment of both boundary-localization and generalization across typologically divergent languages.

2. Automated Three-Stage Construction Pipeline

PsiloQA’s construction leverages automation at every stage, which enables scale and cost efficiency while minimizing human intervention. The pipeline proceeds as follows:

1. Question-Answer Generation: For each target language, Wikipedia passages are sampled. A single GPT-4o prompt generates three QA pairs (covering easy, medium, and hard difficulty) with answers strictly entailed by the passage; one-third yield longer, more elaborate answers, ensuring dataset variety and complexity.
2. Hallucinated Answer Elicitation: To provoke nontrivial hallucinations, questions are re-issued to 24 diverse LLMs (open and proprietary, ranging from 135M to 120B parameters) in a zero-context setting—i.e., without document context, forcing reliance on parametric knowledge. For each LLM-question pair, one answer is retained, resulting in a balanced, multilingual collection of hallucinated responses.
3. Automatic Span-Level Annotation: GPT-4o is prompted to compare the retrieved Wikipedia passage, the gold-standard answer, and the LLM-generated answer. Inconsistent word-level spans are annotated with [HAL]…[/HAL]. Several heuristics enforce precision (limited over-tagging, minimum span size), and samples with model refusals or empty outputs are filtered. Rule-based and prompt-based classifiers further prune malformed or noisy samples, with 6,575 such instances filtered.

3. Evaluation Frameworks and Metrics

PsiloQA supports benchmarking of multiple hallucination detection paradigms:

• Uncertainty Quantification (UQ):
  • Maximum Token Probability (MSP)
  • Claim-Conditioned Probability (CCP)
  • Focus (attention-reweighted token uncertainty)
• LLM-Based Tagging:
  • FActScore (GPT-4o): Fact decomposition and verification at token level via automatic context checking
  • Qwen2.5-32B-it (3-shot): Few-shot tagger for hallucinated spans
• Fine-Tuned Encoders:
  • LettuceDetect-base: English-centric model leveraging local-global attention, pre-trained on RAGTruth
  • ModernBERT-base: English-only baseline, fine-tuned on PsiloQA
  • mmBERT-base: Fully multilingual encoder, trained per-language and jointly across PsiloQA

Character-level binary labels derived from span annotations define two metrics:

• Intersection over Union (IoU): $|Ĥ \cap H| / |Ĥ \cup H|$, assessing boundary-localization accuracy
• Average Precision (AP): $\sum_{n} p(n) \cdot \Delta r(n)$, computed over ranked character predictions

AP is robust to class imbalance and does not require thresholding; IoU directly evaluates boundary accuracy.

4. Core Findings: Performance, Cross-Linguality, and Transfer

4.1 Encoder-Based Model Leadership

Multilingual mmBERT, fine-tuned on all of PsiloQA, achieves state-of-the-art performance: average AP ≈ 78.3% and IoU ≈ 63.5% (test split, 2,897 samples, 14 languages). ModernBERT (English-only) trails but still outperforms UQ and LLM-based detection methods. The top UQ method (Focus) reaches AP ≈ 63.6% for English, with IoU ≈ 42%. LLM-based taggers yield moderate APs (≤76%) but IoU ≤ 45%, signifying suboptimal precision in hallucination boundary localization.

4.2 Cross-Lingual Generalization

Jointly fine-tuned multilingual mmBERT outperforms per-language variants and delivers absolute IoU gains of 5–15 points on PsiloQA and the Mu-SHROOM shared task. Gains are observed in even distant languages such as Arabic, Hindi, and Chinese, indicating successful parameter sharing across distinct scripts and families.

4.3 Knowledge Transfer and Comparison to Human-Labeled Data

mmBERT models trained only on PsiloQA_en outperform those trained solely on RAGTruth_QA across external benchmarks (FAVA-Bench, HalluEntity, Mu-SHROOM_en). On Mu-SHROOM_en, PsiloQA_en yields IoU = 58.5% compared to RAGTruth’s 40.3%, a >45% relative improvement. Additional gains from joint training are marginal, suggesting that dataset scale drives most improvements.

4.4 Cost Efficiency

Automated GPT-4o annotation costs are approximately $535 (pricing:$4 per 1M input tokens, $16 per 1M output tokens), versus$3,000 for span-level human annotation of RAGTruth_QA. Given the larger size, PsiloQA achieves at least 5×, and in practice >17×, greater cost-effectiveness.

5. Implications for Multilingual Hallucination Detection

PsiloQA establishes that fully automated pipelines can generate high-quality, annotated hallucination data at scale and for multiple languages, a capability previously unattained with manual approaches. Empirical results demonstrate that synthetic, large-scale data can exceed smaller, human-annotated corpora for fine-grained token-level detector training. Multilingual pre-training with joint fine-tuning underpins robust cross-lingual parameter sharing, supporting universal (vs. per-language) model architectures.

This suggests a feasible path for deploying accurate, unified hallucination detectors in multilingual, real-world LLM deployments and raises the potential for automated expansion to additional domains.

6. Open Challenges and Future Directions

Several avenues for advancing hallucination detection emerge from PsiloQA:

• Mitigating annotation source bias through ensemble annotation techniques, such as span averaging across multiple LLM annotators
• Extending the automated pipeline’s scope from QA to other factuality-critical tasks, e.g., summarization or data-to-text generation
• Developing a more granular taxonomy of hallucination types, distinguishing, for example, between entity errors and causal inconsistencies
• Designing hybrid architectures that fuse fine-tuned encoders with lightweight UQ or specialized LLM-based prompts, aiming to further enhance span boundary localization

The dataset’s release under CC-BY-4.0 is positioned to facilitate widespread research into fine-grained, multilingual hallucination detection and to advance strategies for safer, more reliable LLM deployments (Rykov et al., 6 Oct 2025).