---
title: 'TurkEmbed4STS: Turkish STS & Hallucination Model'
url: https://www.emergentmind.com/topics/turkembed4sts
type: topic
---

# TurkEmbed4STS: Turkish STS & Hallucination Model

TurkEmbed4STS is a Turkish-specific sentence and token encoder derived from the TurkEmbed family, optimized for Semantic Textual Similarity (STS) and further adapted for token-level hallucination detection in Retrieval-Augmented Generation (RAG) systems for Turkish. Developed on top of the gte-multilingual-base architecture, it integrates specialized sequential training and matryoshka representation learning, providing both high-accuracy embeddings and efficient, interpretable, long-context processing. TurkEmbed4STS achieves state-of-the-art performance on Turkish STS benchmarks and supports robust hallucination detection for real-world Turkish RAG applications [2511.08376][2509.17671].

## 1. Model Architecture and Representation Learning

TurkEmbed4STS is based on the gte-multilingual-base encoder (12-layer transformer, hidden size 768, 305 M parameters), supporting sequences up to 8,192 tokens via rotary embeddings and local-global attention mechanisms [2511.08376]. The standard use involves mean-pooling the final hidden states to obtain a fixed 768-dimensional vector for each sentence. For scalability, TurkEmbed4STS introduces *matryoshka representation learning*, which allows dynamic projection into lower-dimensional subspaces (64, 128, 256, 512, or 768 dimensions) by optimizing a set of linear projections $z_d = P_d(h_L)$ for $d \in \{64, 128, 256, 512, 768\}$, where $h_L$ is the pooled top-layer hidden state. All projections are trained jointly, enabling inference-time truncation with graceful degradation in retrieval and similarity tasks.

For hallucination detection, TurkEmbed4STS diverges from sentence-level regression models by (i) retaining all token-level representations and (ii) appending a lightweight linear classification head over the output of each token to assign a binary label (supported/hallucinated). This design enables token granularity in downstream detection, allowing interpretable span-level outputs adaptable to various structured and generative Turkish NLP tasks [2509.17671].

## 2. Training Procedure and Datasets

The training pipeline is two-stage for embedding applications [2511.08376]:

1. **Natural Language Inference (NLI) Pretraining:** Uses Turkish translations of SNLI and MultiNLI (All-NLI-TR, 482,091 train, 6,802 dev, 6,827 test), with Multiple Negatives Ranking Loss (MNRL) applied per embedding dimension.
2. **Semantic Textual Similarity (STS) Fine-Tuning:** Uses the Turkish STS-Benchmark (STSB-TR, 5,749 train, 1,500 val, 1,379 test), optimizing the CoSENT loss and matryoshka losses per subspace.

Training parameters include AdamW optimizer (learning rate $2 \times 10^{-5}$), batch sizes 64 (NLI) and 32 (STS), and maximum sequence length 128 for embedding tasks.

For hallucination detection, TurkEmbed4STS is fine-tuned from the STS-specialized checkpoint on a machine-translated Turkish RAGTruth dataset (17,790 train, 2,700 test instances; average input 801 tokens, up to 2,632) [2509.17671]. The input concatenates “[CLS] question [SEP] context documents [SEP] generated answer [SEP],” with only answer tokens labeled (supported=0, hallucinated=1, background=-100). Training uses AdamW (learning rate $1 \times 10^{-5}$), batch size 4, 6 epochs, on an A100-40GB GPU.

## 3. Token-Level Hallucination Detection Mechanism

Hallucination detection is framed as a token-level classification problem. Each generated answer token receives a label:
- 0: “supported” if the token can be grounded in the retrieved context.
- 1: “hallucinated” if not supported by any context.

The loss for each answer token $i$ is standard cross-entropy:
$$
L_i = -[y_i \log p_i + (1-y_i) \log(1 - p_i)]
$$
with total loss over $N$ answer tokens:
$$
\text{Loss} = \frac{1}{N} \sum_i L_i
$$

Input masking ensures no loss is computed over question or context tokens. This enables focused learning on generative outputs, supporting accurate span identification for hallucination analysis.

## 4. Empirical Results and Comparative Performance

TurkEmbed4STS exhibits strong and balanced performance across multiple Turkish generation and retrieval tasks. On token-level hallucination detection (RAGTruth test set), performance metrics are as follows [2509.17671]:

| Task Type   | Precision | Recall  | F1-Score | AUROC  |
|-------------|-----------|---------|----------|--------|
| Summary     | 0.6325    | 0.5656  | 0.5862   | 0.5656 |
| Data2Text   | 0.7397    | 0.7333  | 0.7365   | 0.7333 |
| QA          | 0.7378    | 0.7382  | 0.7380   | 0.7382 |
| Whole set   | 0.7268    | 0.7014  | 0.7132   | 0.7014 |

Compared to alternative Turkish hallucination detectors:
- **ModernBERT:** F1 = 0.7266 (slightly higher on the full set, especially in QA precision: 0.7583).
- **EuroBERT:** F1 = 0.7163 (competes in data-to-text, lower overall recall).

For STS evaluation on STSB-TR [2511.08376]:

| Model                      | Emb. Dim | Pearson | Spearman |
|----------------------------|----------|---------|----------|
| Emrecan’s Model            | 768      | 0.834   | 0.830    |
| nomic-embed-text-v2-moe    | 768      | 0.828   | 0.834    |
| multilingual-E5-large      | 1024     | 0.846   | 0.854    |
| TurkEmbed4STS              | 64–768   | 0.845   | 0.853    |

Relative to Emrecan’s model, TurkEmbed4STS yields a +1.1% absolute Pearson and +2.3% Spearman improvement for STS-b-TR. On cross-lingual STS22-TR, TurkEmbed4STS attains 0.646 Pearson/0.668 Spearman (Emrecan: 0.540/0.563).

## 5. Computational Efficiency and Scalability

TurkEmbed4STS is designed for efficient inference on contemporary hardware and long-context scenarios. For hallucination detection and STS:
- **Model size:** 210–305 M parameters (slight size difference due to classification head integration).
- **Inference memory:** ~10 GB for 8,192-token max-length on GPU.
- **Latency:** Tens of milliseconds per 800-token sequence at batch size 4 (hallucination detection, A100-40GB), ~2 ms/sentence (embedding, FP16, batch=1, T4 GPU) [2509.17671][2511.08376].
- **Matryoshka mechanism:** Embedding dimension can be dynamically reduced at inference for improved speed, with modest accuracy loss.

This architecture is more resource-efficient than prompt-based hallucination detection with 70B+ parameter LLMs, especially for real-time or large-batch Turkish RAG applications.

## 6. Strengths, Limitations, and Error Characterization

**Strengths:**
- Precise token-level supervision enables interpretable hallucination span detection.
- Consistent, well-balanced precision and recall in QA and data-to-text; robust across various generative Turkish tasks.
- Maintains competitive or superior STS correlation compared to prior art.
- Supports extremely long contexts (up to 8,192 tokens).
- Embedding output adaptable to application resource constraints.

**Limitations:**
- Summarization remains the hardest detection challenge (F1 ≈ 0.59).
- Slightly lower performance than ModernBERT in some QA cases and EuroBERT in data-to-text.
- Embedding inference speed and GPU memory usage higher than smaller models (e.g., Emrecan); this suggests potential for further distillation.

**Error Analysis:** Missed hallucinations are concentrated in subtle baseless introductions, where micro-level context drift is difficult for the model to detect. False positives typically appear for technical terms absent from the retrieved context in data-to-text. There is no dedicated ablation for TurkEmbed4STS within hallucination detection, but its balanced performance-tradeoff profile is notable.

## 7. Applications, Impact, and Future Directions

TurkEmbed4STS supports a broad suite of Turkish NLP tasks, including:
- Token-level hallucination detection in RAG pipelines [2509.17671].
- Semantic search, clustering, passage retrieval, paraphrase mining, and metrics for generative evaluation [2511.08376].
- Downstream adaptation for real-time, resource-constrained, or large-scale Turkish language applications.

The design demonstrates that sequential NLI → STS training, matryoshka embedding, and explicit token-level objectives provide a stable foundation for Turkish-specific reliability in both generative evaluation and embedding retrieval contexts. Further improvement opportunities include model distillation for speed, integration of human-annotated parallel data for robustness, and more dynamic dimension selection in deployment [2511.08376].

Source: https://www.emergentmind.com/topics/turkembed4sts