Translation Tangles: Evaluating Quality & Fairness

• Translation Tangles is a unified framework that standardizes translation quality and fairness assessment through multi-domain benchmarks, hybrid bias detection, and human verification.
• It evaluates multilingual translations by analyzing disparities across language families and domains, highlighting issues like typological distance and resource imbalance.
• A hybrid bias detection pipeline combining heuristic rules, semantic similarity filtering, and LLM-based judgment quantifies biases in cultural, sociocultural, and gender categories.

Translation Tangles is a unified, end-to-end framework and dataset for evaluating both translation quality and fairness in multilingual LLMs. It was created to systematize three intertwined issues in machine translation with LLMs: uneven quality across language families and directions, encoded and amplified biases in translated content, and fairness concerns for low-resource languages. The framework combines a multi-domain benchmarking suite, a hybrid bias detection pipeline, and a human-annotated subset for bias verification, thereby linking conventional MT evaluation with bias-aware analysis in a single experimental substrate (Sayeedi et al., 9 Oct 2025).

1. Problem setting and analytical scope

Translation Tangles targets a regime in which multilingual LLMs are fluent and often context-aware, yet their behavior remains uneven across typological distance, translation direction, and domain. The framework specifically emphasizes that smaller and mid-sized LLMs struggle more with cross-family pairs than intra-family pairs, that domain performance varies sharply, and that models can introduce or erase sensitive entities or terms such as gendered roles, racial identifiers, and religious markers. The reported bias counts and quality errors are most frequent when translating from Gujarati, Kazakh, Finnish, and Lithuanian, which the paper presents as evidence of resource and representation disparities (Sayeedi et al., 9 Oct 2025).

Its stated goals are correspondingly threefold. First, it provides a reproducible evaluation suite covering language family distance, directionality, and domain shift. Second, it detects, categorizes, and validates potential biases via interpretable heuristics, semantic similarity, and LLM-as-a-judge, and quantifies their prevalence across models, languages, and domains. Third, it releases a human-verified, bias-annotated subset of 1,439 reference–translation pairs for detector calibration and fairness-aware MT research.

A central conceptual point is that Translation Tangles treats fairness in translation as a content-preservation problem rather than as a standard binary-decision fairness problem. Accordingly, the study quantifies bias via counts, agreement, and human validation rather than demographic-parity or equalized-odds style metrics. The paper explicitly notes that demographic parity difference is not applied here because translation bias manifests as content alterations rather than binary decisions.

2. Benchmark composition, language coverage, and corpora

The framework evaluates 12 bidirectional language pairs, or 24 directions, drawn from multiple language families and grouped so that family distance can be studied directly (Sayeedi et al., 9 Oct 2025).

Dimension	Coverage	Sources
Language pairs	de↔en, cs↔en, ru↔en, fr↔de, lt↔en, gu↔en, bn↔en, fi↔en, et↔en, tr↔en, kk↔en, zh↔en	Multi-family setup
Domains	General/news, Legal, Medical, Literature	WMT18/19, BanglaNMT, MultiEURLEX, ELRC-Medical-V2, Kazakh–Russian–English corpus
Annotation subset	1,439 reference–translation pairs	Human bias verification

Within Indo-European, the benchmark includes Germanic de↔en, Slavic cs↔en and ru↔en, Romance↔Germanic fr↔de, Baltic lt↔en, and Indo-Iranian (Indic) gu↔en and bn↔en. Outside Indo-European, it covers Uralic (Finnic) fi↔en and et↔en, Turkic tr↔en and kk↔en, and Sino-Tibetan (Sinitic) zh↔en. This construction supports direct comparison between intra-family and cross-family directions.

The domain layer is equally explicit. General/news data come from WMT18 and WMT19, with BanglaNMT used for Bengali–English. Legal evaluation uses MultiEURLEX, described as 23 EU languages and 65K documents with train 55K, dev 5K, and test 5K. Medical evaluation uses ELRC-Medical-V2 for English↔21 EU languages with approximately 13K aligned sentences per pair in CSV format and no predefined splits. Literature evaluation uses a Kazakh–Russian–English corpus with 71K parallel pairs in Parquet format and no splits.

Several corpus statistics are given because resource imbalance is part of the framework’s explanatory logic. WMT19 varies substantially by pair, with ru–en at approximately 37.5M training examples and gu–en at approximately 13.7K. WMT18 uses standardized splits including approximately 3K test per pair. BanglaNMT provides 2.38M train, 597 validation, and 1K test examples. The study emphasizes that no training is done on these datasets; they serve purely for evaluation.

Preprocessing and curation are standardized. Sources are pulled via Hugging Face or WMT shared task repositories, and parallel data are standardized into fields such as doc_id, language codes, source_text, and target_text. Generation uses standardized translation prompts with controlled temperature $= 0.1$, together with truncation to preserve 500 tokens for prompt and response.

3. Evaluation protocol, model set, and quality metrics

Translation Tangles evaluates 15 open-source LLMs spanning small, medium, and large scales: Gemma-7B, Gemma-2-9B, Llama-3.1-8B, Llama-3.1-70B, Llama-3.2-1B, Llama-3.2-70B, Llama-3.2-90B, Mixtral-8x7B, OLMo-1B, Phi-3.5-mini, Qwen-2.5-0.5B, Qwen-2.5-1.5B, Qwen-2.5-3B, deepseek-r1-distill-32b, and deepseek-r1-distill-70b (Sayeedi et al., 9 Oct 2025). Model families are analyzed by scale using the thresholds small $\leq 7$B, medium $7$–$30$B, and large $>30$B.

The decoding configuration is deliberately controlled. The paper specifies temperature $= 0.1$, batch size 16, and truncation to reserve context window capacity. It also states that top-k, top-p, beam search, and length penalty are not specified, and that models are used as-is. This matters because Translation Tangles aims to reduce confounding from decoding variability and to attribute differences to model capability, language family distance, and domain.

Quality is measured with BLEU, chrF, TER, BERTScore, COMET, WER, CER, and ROUGE. The paper gives the standard BLEU expression

$$\mathrm{BLEU} = \mathrm{BP}\cdot \exp\left(\sum_{n=1}^{N} w_n \cdot \log p_n\right),$$

with brevity penalty

$\mathrm{BP} = \exp(\min(0, 1-r/c)),$

where $r$ is reference length and $c$ is candidate length. For chrF it gives

$\leq 7$0

where $\leq 7$1 and $\leq 7$2 are precision and recall of character n-grams computed between candidate and reference.

The study reports aggregated results as means and standard deviations across models and domains. Significance testing and confidence intervals are not reported. It also notes that experiments benefited from an academic HPC cluster.

4. Hybrid bias detection pipeline

The bias analysis pipeline is hybrid in a strict sense: it combines heuristic detection, semantic similarity gating, and LLM-based validation (Sayeedi et al., 9 Oct 2025). The first stage applies two heuristic detectors independently and then unions them. One detector uses category-specific keyword lexicons for gender, religious, cultural, social, and racial terms. The other uses named-entity-recognition deltas with spaCy plus extended tags.

The keyword lists are concrete. Gender examples include “he, she, man, woman, husband, wife, housewife, nurse, doctor.” Religious examples include “allah, god, jesus, hindu, muslim, islam, christian, jewish, mosque, church.” Cultural examples include “sari, kimono, turban, hijab, diwali, ramen, sushi, taco, chopstick, yoga.” Social examples include “rich, poor, slum, elite, working class, laborer, billionaire, beggar.” Racial examples include “white, black, asian, african, latino, caucasian, arab, indigenous.”

The NER mapping defines PERSON → gender, NORP → cultural/religious/racial, GPE → sociocultural, ORG → social, and LANGUAGE → cultural, with augmented tags RELIGION → religious and ETHNICITY → racial. A term is flagged if it appears in the translation $\leq 7$3 but not in the reference $\leq 7$4 as an insertion, or in $\leq 7$5 but not in $\leq 7$6 as an erasure, and belongs to a sensitive lexicon. An entity is flagged if a new sensitive entity appears in $\leq 7$7 relative to $\leq 7$8.

The second stage gates heuristic outputs by semantic similarity. Sentence embeddings are computed with gemini-embedding-001, and cosine similarity is defined as

$\leq 7$9

The threshold is $7$0, selected via grid search and knee-point analysis. The final rule is explicit: a pair is marked biased only if at least one heuristic fires and $7$1. The paper reports that per-bias curves saturate near $7$2, and that normalized sensitivity confirms elbow behavior across categories.

The third stage uses Gemini-2.5-Flash as an LLM-as-a-judge with temperature $7$3. The model consumes the reference and translation and returns structured JSON fields: "bias_detected", "detected_biases", and "reasons". Retries are used to ensure well-formed JSON. In the experimental logic of the paper, these judgments serve as a second pass and as a pseudo-gold for ablations, while human annotations remain the final gold.

Ablation results quantify both utility and limitation. Heuristic versus LLM-judge agreement is approximately 48.79% overall across categories. Agreement is higher for social at 100%, though very rare, religion at 66.7%, and gender at 61.1%; it is lower for racial at 13.6%, sociocultural at 45.8%, and cultural at 49.5%. Throughput differs sharply: heuristics plus similarity process approximately 1,902 samples in less than 9 minutes on CPU, whereas LLM-as-a-judge takes more than 30 minutes for the same sample size.

5. Bias taxonomy and human verification

Translation Tangles annotates six bias categories: gender, religious, cultural, social, racial, and sociocultural (Sayeedi et al., 9 Oct 2025). These are described as grounded in prior NLP and social-science taxonomies, including gender stereotypes, religious identity markers, cultural prioritization, social-class associations, race and ethnicity descriptors, and broader sociocultural framing.

The human-annotated subset contains 1,439 sampled reference–translation pairs. Its composition is deliberately stratified. Agreement cases, where both heuristic and LLM judge flag bias, contribute 851 samples drawn from 928. Disagreement cases, where the heuristic flagged bias and the LLM judge did not, contribute 294 sampled from 974 remaining. Undetected cases, where neither flagged bias, contribute 294 samples. This mixture is intended to calibrate both automatic detectors and the judge model.

The annotation protocol uses two independent annotators, with a third adjudicator resolving disagreements. Annotators are described as blinded and not exposed to system outputs or other annotators’ decisions. Human review is reported as independent with ethical oversight, and explanations were recorded for some cases. Translation-only failures or “safety refusals” were excluded from metric aggregation.

The paper explicitly does not report a numeric inter-annotator agreement such as Cohen’s $7$4. It nonetheless states the formula

$7$5

where $7$6 is observed proportion agreement and $7$7 is expected agreement by chance given each annotator’s label distribution. The absence of measured $7$8 is one of the study’s disclosed methodological omissions.

A recurrent misconception addressed by this design is that automatic evaluation alone suffices for fairness analysis. Translation Tangles formalizes the opposite position: heuristics provide a fast and interpretable first-pass filter, LLM-as-a-judge offers scalable validation and explanations, but human annotations remain essential for calibration.

6. Empirical results on quality and fairness

The quality results show persistent family-distance effects that shrink with scale but do not disappear (Sayeedi et al., 9 Oct 2025). For large models, BLEU is approximately 29.11 for intra-family directions versus 25.13 for cross-family directions; BERTScore is approximately 0.707 versus 0.646; COMET is approximately 0.812 versus 0.763. For medium models, BLEU is approximately 20.99 versus 15.00, BERTScore approximately 0.510 versus 0.419, and COMET approximately 0.662 versus 0.574. For small models, BLEU is approximately 10.37 versus 6.18, BERTScore approximately 0.346 versus 0.207, and COMET approximately 0.512 versus 0.407. The best overall system is reported as llama-3.2-90B with BLEU approximately 44.16, BERTScore approximately 0.798, and COMET approximately 0.873. The paper also notes notable failures, including near-zero BLEU in some low-resource or typologically distant directions such as en→tr despite large model capacity.

Domain robustness is uneven and follows a stable ordering. Law is highest, Literature is lowest, and Medical lies in between with higher variance. The reported aggregates are: Law with BLEU approximately $7$9, BERTScore approximately $30$0, ROUGE-L approximately $30$1, chrF approximately $30$2, and WER approximately $30$3; Medical with BLEU approximately $30$4, BERTScore approximately $30$5, ROUGE-L approximately $30$6, chrF approximately $30$7, and WER approximately $30$8; Literature with BLEU approximately $30$9, BERTScore approximately $>30$0, ROUGE-L approximately $>30$1, chrF approximately $>30$2, and WER approximately $>30$3. The study interprets this as diminishing returns from scaling in domain-specific tasks, with improvements that are inconsistent and smaller than in general news.

Bias incidence is dominated by cultural and sociocultural categories. Framework counts, with judge-confirmed counts in parentheses where relevant, are: cultural 798 (395 confirmed), sociocultural 744 (341), gender 265 (162), racial 66 (9), religious 24 (16), and social 5 (5). The paper states that cultural and sociocultural biases account for more than 75% of detected instances.

Variation across models is substantial. gemma-2-9B shows the highest overall bias, notably sociocultural bias with $>30$4. llama-3.2-8B has the highest cultural bias with $>30$5. Larger models such as llama-3.2-90B with $>30$6 and llama-3.1-70B with $>30$7 show lower counts, which the paper presents as suggesting that scale and safety alignment reduce bias, though not uniformly; Mixtral-8x7B is noted as a counterexample with high cultural bias.

Variation across language pairs is equally pronounced. The highest total bias counts occur for gu→en with $>30$8, including cultural $>30$9, followed by kk→en with $= 0.1$0, fi→en with $= 0.1$1, and lt→en with $= 0.1$2. Lower counts are reported for de→en with $= 0.1$3 and zh→en with $= 0.1$4, consistent with what the paper describes as resource exposure and alignment in pretraining corpora.

The qualitative analysis identifies several recurrent phenomena: sensitive entity substitutions or shifts, such as “church”→“temple” flagged as religious bias; insertions of culturally loaded terms not present in the reference; sociocultural reframing, such as “win”→“be successful”; safety refusals on politically sensitive inputs; non-preservation of pronoun number and human referents; and template-based bias explanations from LLM judges. Taken together, these results support one of the study’s central claims: accurate translations can still be biased, and standard metrics may remain strong while sensitive entities or terms are inserted or erased.

7. Position within MT evaluation, reproducibility, and limitations

Translation Tangles is positioned as an extension of conventional MT benchmarking rather than a replacement for it (Sayeedi et al., 9 Oct 2025). General MT benchmarks primarily focus on translation quality, and cross-lingual task suites primarily target multilingual NLP performance; Translation Tangles adds bias-aware evaluation with a hybrid pipeline and human-verified labels, together with cross-family and directional analyses across Law, Medical, and Literature. A plausible implication is that it supplies a bridge between adequacy-oriented evaluation and fairness diagnostics that standard score tables do not expose.

The framework is designed for reuse. The paper provides a public repository at github.com/faiyazabdullah/TranslationTangles. The end-to-end workflow is explicit: generate translations with standardized prompts and decoding settings; compute BLEU, chrF, TER, BERTScore, COMET, WER, CER, and ROUGE; run heuristic bias detection and the sentence-embedding similarity filter with $= 0.1$5; validate flagged samples via Gemini-2.5-Flash; and optionally evaluate against the human-annotated subset of 1,439 pairs. Expected outputs include metric score tables per language pair, domain, and model, as well as bias counts, category breakdowns, and JSON judge outputs with explanations.

The limitations section is unusually direct. Bias detection is applied only in $= 0.1$6, so reverse directions and intra-regional pairs remain outside the present scope. The six-category taxonomy excludes LGBTQ+, disability, political ideology, and other harm types, while subtle harms such as sarcasm, omission bias, and normative framing remain difficult. The human study is described as high-quality but modest relative to the overall dataset scale. Standard MT metrics do not capture fairness, and fairness metrics adapted to translation require careful formulation around insertion and erasure rates of sensitive content. The study also notes that LLM-as-a-judge systems can themselves be biased or inconsistent and that reliance on a single judge model should be tempered with human review and multi-judge consensus.

The paper does not experimentally evaluate mitigation. It states instead that the framework is aimed at measurement and can be reused to test mitigation strategies such as prompt engineering, debiasing, post-editing, or constrained decoding. In that sense, Translation Tangles functions as an evaluation infrastructure: it provides a reproducible substrate for studying how translation quality, language-family distance, domain shift, and bias interact in multilingual LLMs, and it makes explicit that improvements in fluency or adequacy do not by themselves guarantee representational fairness.