Turk-LettuceDetect: Turkish Hallucination Detector

• Turk-LettuceDetect adapts the English LettuceDetect framework to Turkish, detecting hallucinations at the token level in Retrieval-Augmented Generation systems.
• It leverages Turkish-specific and multilingual encoders (ModernBERT-base-tr, TurkEmbed4STS, EuroBERT) fine-tuned on a machine-translated RAGTruth benchmark for robust performance.
• Evaluation shows ModernBERT-base-tr achieving a token-level F1 near 0.73, demonstrating efficient filtering of unsupported spans in Turkish RAG pipelines.

Turk-LettuceDetect is a family of encoder-based models for automatically detecting hallucinations in Turkish Retrieval-Augmented Generation (RAG) systems. It adapts the English-centric LettuceDetect framework to Turkish and to a broader multilingual setting by treating hallucination detection as a token-level classification problem over generated answers conditioned on a question and retrieved context. The system was developed to address a specific gap: most hallucination detection frameworks and benchmarks are English-centric, while Turkish is both low-resource and morphologically complex, making token-level grounding comparatively difficult. The work introduces three Turkish-capable detectors—based on ModernBERT-base-tr, TurkEmbed4STS, and EuroBERT—and trains them on a machine-translated Turkish version of the RAGTruth benchmark (Taş et al., 22 Sep 2025).

1. Research context and scope

Turk-LettuceDetect was proposed in response to a persistent limitation of LLMs in grounded generation: even when a RAG pipeline supplies external evidence, generated answers may still contain unsupported or contradictory content. The paper frames this as especially consequential for Turkish because the language is low-resource in terms of hallucination-specific supervision and morphologically complex and agglutinative in ways that complicate token-level grounding (Taş et al., 22 Sep 2025).

The system is explicitly positioned as an extension of LettuceDetect, an encoder-based token-level hallucination detector for RAG originally developed for English. The original LettuceDetect formulation used a long-context ModernBERT architecture and achieved an example-level F1 score of 79.22% on RAGTruth, improving over Luna while remaining substantially smaller than large LLM-based judges (Kovács et al., 24 Feb 2025). Turk-LettuceDetect preserves the same token classification formulation and general architectural style, but replaces the English-centric backbone choice with Turkish-specific or Turkish-supported encoders and trains them on a Turkish machine translation of RAGTruth (Taş et al., 22 Sep 2025).

Within the broader literature, Turk-LettuceDetect belongs to a line of work that treats hallucination detection as a post-generation verification problem over $(q,c,a)$ triples rather than as a generation problem itself. Later work extended the LettuceDetect paradigm beyond document-grounded text to code, tool output, and structured evidence, showing that document-only detectors may not generalize well under modality shift (Kovács et al., 1 Jul 2026). A plausible implication is that Turk-LettuceDetect should be understood as a document-grounded Turkish detector rather than a universal verifier for all grounded generation settings.

2. Task formulation and labeling scheme

Turk-LettuceDetect formulates hallucination detection as binary token-level classification over the generated answer. Given question tokens $q = (q_1, \dots, q_{T_q})$, context tokens $c = (c_1, \dots, c_{T_c})$, and answer tokens $a = (a_1, \dots, a_{T_a})$, the concatenated input is

$$x = [\text{[CLS]}, q, \text{sep}, c, \text{sep}, a, \text{[SEP]}].$$

The encoder produces contextual representations

$$h_1, \dots, h_T = \text{Encoder}(x), \quad h_i \in \mathbb{R}^d,$$

and a token classification head maps each $h_i$ to binary logits,

$z_i = W h_i + b \in \mathbb{R}^2,$

with class probabilities

$$p(y_i = k \mid x) = \frac{\exp(z_{i,k})}{\sum_{k'=0}^{1} \exp(z_{i,k'})}, \quad k \in \{0,1\}.$$

The labels are defined only for answer tokens. If an answer token is supported by the context it receives $y_i=0$; if it is hallucinated it receives $q = (q_1, \dots, q_{T_q})$0; question and context tokens are masked with label $q = (q_1, \dots, q_{T_q})$1 so that they are ignored in the loss. Formally,

$q = (q_1, \dots, q_{T_q})$2

Training uses token-wise cross-entropy over answer tokens only:

$q = (q_1, \dots, q_{T_q})$3

This formulation makes Turk-LettuceDetect a sequence-labeling system rather than an answer-level classifier. Example-level decisions are obtained only after aggregation, for example by determining whether any answer token has been labeled hallucinated. The token-level formulation is central to the framework because it permits span localization, not merely binary pass/fail scoring (Taş et al., 22 Sep 2025).

3. Model family and training configuration

The paper fine-tunes three encoder-based architectures under the same supervision and hyperparameter regime: ModernBERT-base-tr, TurkEmbed4STS, and EuroBERT. All three are implemented as standard token classification pipelines in a HuggingFace-style sequence-labeling setting, with concatenated question, context, and answer as input and per-token binary logits as output (Taş et al., 22 Sep 2025).

Model	Backbone characterization	Notable property
ModernBERT-base-tr	Turkish-specific ModernBERT	Supports up to 8,192 tokens
TurkEmbed4STS	Turkish embedding model based on GTE-multilingual-base	Adapted for token-level outputs
EuroBERT	Multilingual encoder	Variant named lettucedect-210m-eurobert-tr-v1

ModernBERT-base-tr is described as a Turkish-specific ModernBERT variant built on a BERT-like transformer architecture with Rotary Positional Embeddings and local-global attention. Its context window extends to 8,192 tokens, which is important for long RAG contexts. TurkEmbed4STS is a Turkish embedding model originally optimized for semantic textual similarity and based on GTE-multilingual-base; in Turk-LettuceDetect it is adapted from sentence-level use to token-level hallucination detection. EuroBERT is a multilingual encoder that had already been used in the multilingual extension of the original LettuceDetect and is here fine-tuned on Turkish-translated RAGTruth (Taş et al., 22 Sep 2025).

Training is conducted on the Turkish machine-translated RAGTruth dataset with 17,790 training examples and 2,700 test examples. The reported setup uses 6 epochs, learning rate $q = (q_1, \dots, q_{T_q})$4, and batch size 4 on a single NVIDIA A100 40GB GPU. Each epoch takes about 20 minutes, for a total of roughly 2 hours per model. No complex curriculum learning or class balancing strategy is described; masking of non-answer tokens is the main structural constraint (Taş et al., 22 Sep 2025).

The original LettuceDetect paper emphasized the efficiency advantages of long-context encoders over LLM-based judges, reporting 30 to 60 examples per second on a single GPU for the English ModernBERT system (Kovács et al., 24 Feb 2025). Turk-LettuceDetect inherits that encoder-centric design philosophy, although the Turkish paper does not give an explicit tokens-per-second throughput figure.

4. Turkish RAGTruth and machine-translation adaptation

The dataset foundation is RAGTruth, a benchmark containing 17,790 training and 2,700 test instances across question answering, data-to-text generation, and news summarization. Each example includes a prompt or question, context, multiple model-generated answers, and human annotations marking hallucinated spans with rationale. The original hallucination taxonomy distinguishes Evident Conflict, Subtle Conflict, Evident Introduction of Baseless Information, and Subtle Introduction of Baseless Information; Turk-LettuceDetect collapses these into a binary supported-versus-hallucinated labeling scheme for training (Taş et al., 22 Sep 2025).

To create Turkish supervision, the authors translate RAGTruth into multiple languages, including Turkish, using google/gemma-3-27b-it with vLLM on a single A100 GPU. Throughput is reported as about 30 examples processed in parallel, and the full translation for one target language takes approximately 12 hours. Two translation protocols are described. For answers containing hallucination markup, <HAL> tags are preserved exactly in both number and placement, while the tagged content is translated. For prompts and metadata, the goal is functional equivalence rather than literal translation, with instructional text and JSON keys and values translated where relevant (Taş et al., 22 Sep 2025).

This translation procedure is methodologically consequential because hallucination span labels must survive translation despite changes in tokenization and character offsets. The paper notes several difficulties: label preservation, alignment shifts, error propagation from machine translation itself, and tokenization effects induced by Turkish agglutination. A single Turkish token may correspond to multiple English tokens or vice versa, which changes the granularity of span supervision. The authors therefore recalculate start and end indices and labels for Turkish text after translation (Taş et al., 22 Sep 2025).

The reliance on machine-translated supervision distinguishes Turk-LettuceDetect from later multilingual hallucination benchmarks that generate synthetic hallucinations directly over multilingual QA data. MultiWikiQHalluA, for example, constructs synthetic token-level hallucination datasets for 306 languages and trains token-level classifiers for 30 European languages using the LettuceDetect framework (Thoresen et al., 4 May 2026). This suggests a broader methodological landscape in which machine-translated human annotations and synthetic multilingual generation are alternative routes to non-English hallucination supervision.

5. Evaluation results and comparison with LLM judges

The paper evaluates precision, recall, F1, and AUROC at both example level and token level, but the core reported comparison for Turk-LettuceDetect is token-level performance on Turkish RAGTruth (Taş et al., 22 Sep 2025).

Model	Whole-test Precision	Whole-test Recall	Whole-test F1
ModernBERT-base-tr	0.7583	0.7024	0.7266
TurkEmbed4STS	0.7268	0.7014	0.7132
EuroBERT (210m)	0.7511	0.6908	0.7163

ModernBERT-base-tr achieves the best overall token-level result on the complete Turkish RAGTruth test set, with precision 0.7583, recall 0.7024, and F1 0.7266. It also performs strongly on QA with F1 0.7588 and data-to-text with F1 0.7391. TurkEmbed4STS reaches an overall F1 of 0.7132 and is described as comparatively balanced across tasks, with QA precision and recall nearly identical at 0.7378 and 0.7382. EuroBERT reaches overall F1 0.7163 and performs best on data-to-text, where it attains precision 0.7866 and F1 0.7496 (Taş et al., 22 Sep 2025).

Across all three models, summarization is the hardest task. ModernBERT-base-tr records F1 0.6007 on summary data, TurkEmbed4STS 0.5862, and EuroBERT 0.5771. The paper attributes this pattern to the more open-ended and implicitly inferential nature of abstractive summarization, whereas structured tasks such as QA and data-to-text permit stronger grounding signals (Taş et al., 22 Sep 2025).

The paper also compares the specialized encoders with decoder-only LLMs used directly as hallucination judges, including GPT-4.1, Mistral, and Qwen variants. The reported pattern is that LLM judges exhibit very high recall—up to about 0.9938—but low precision because they over-generate hallucination labels. Qwen3-14B achieves the best example-level F1 among the evaluated LLMs at about 0.7429, yet the paper concludes that these judge-style approaches remain less attractive for high-precision, real-time deployment than the specialized encoders (Taş et al., 22 Sep 2025).

This comparison aligns with the original English LettuceDetect result: encoder-based token classifiers can approach the performance of larger judge models while remaining much smaller and operationally cheaper (Kovács et al., 24 Feb 2025).

6. Deployment, limitations, and significance

Turk-LettuceDetect is intended for integration into Turkish RAG pipelines as a post-generation token-level filter, a scoring or quality-control component, or a training-time monitoring tool. Given a triple $q = (q_1, \dots, q_{T_q})$5, the detector assigns a label to each answer token, enabling downstream actions such as highlighting unsupported spans, suppressing or editing suspicious segments, or triggering another round of retrieval and regeneration. The paper also notes that the fraction of hallucinated tokens can function as a hallucination score for threshold-based rejection or reprocessing (Taş et al., 22 Sep 2025).

The system’s long-context capability is operationally important. ModernBERT-based Turk-LettuceDetect supports contexts up to 8,192 tokens. The paper notes that RAGTruth inputs have mean length about 801 tokens, median 741, and maximum 2,632 tokens, while real RAG systems may combine multiple long documents. The suite is described as computationally efficient, with model sizes in the approximately 135M to 305M range and training feasible on a single A100 40GB GPU (Taş et al., 22 Sep 2025).

Several limitations are identified. The most immediate is reliance on machine-translated training data, which may diverge from native Turkish and may itself introduce artifacts. Domain coverage is restricted to QA, Yelp data-to-text, and CNN/Daily Mail summarization. All models are weaker on summarization than on structured tasks. Turkish morphology and syntax introduce sub-token phenomena that token-level binary labeling may not capture well, especially when unsupported content is fused inside agglutinative forms. The framework is also sensitive to noisy retrieval: if relevant evidence is absent from the retrieved context, correct content derived from parametric model knowledge may still be labeled hallucinated under the benchmark’s grounding criterion (Taş et al., 22 Sep 2025).

The paper proposes several directions for future work: human-annotated Turkish hallucination datasets, extension to other low-resource and morphologically complex languages, architectures with stronger Turkish morphological modeling, and task-specific strategies for different grounded generation settings. These directions are consistent with broader developments in the hallucination detection literature, where later work has pushed the LettuceDetect line toward multilingual synthetic benchmarks and toward structured evidence settings beyond documents (Thoresen et al., 4 May 2026, Kovács et al., 1 Jul 2026).

Turk-LettuceDetect’s central significance is that it establishes the first dedicated Turkish hallucination detection suite for RAG and provides both models and a translated benchmark resource. In practical terms, it shows that specialized long-context encoders can deliver token-level hallucination localization for Turkish with overall F1 near 0.73 on Turkish RAGTruth, while maintaining a precision-efficiency profile more suitable for real-time deployment than prompt-based LLM judges (Taş et al., 22 Sep 2025).