Separate Cultural Adaptation Effects from Translation Quality

Determine whether the larger unsafe-compliance rates observed for culturally adapted prompts, relative to literally translated prompts, are attributable to cultural adaptation itself rather than to differences in translation quality or harm preservation, for example by using reference-free quality estimation.

Background

The paper compares culturally adapted prompts with prompts produced by literal machine translation and finds that culturally adapted prompts generally elicit more unsafe compliance in the non-English settings. The authors measure the degree of textual change using chrF and observe that greater adaptation is associated with larger compliance differences in Bengali and Levantine Arabic.

The authors acknowledge that low chrF may reflect not only cultural adaptation but also poorer translation quality or imperfect transfer of harmfulness. Because these factors could independently affect refusal and compliance rates, the paper leaves unresolved how to disentangle the causal effect of cultural adaptation from translation-quality effects.

References

One alternative explanation remains: low chrF may partly reflect low harmfulness transfer and poor translation quality, which can itself induce refusals. We leave separating adaptation from translation quality ( e.g.\ via reference-free quality estimation) to future work.

— In the Guise of Correction: How Framing and Cultural Adaptation Erode LLM Safety on Real-World Criminal Cases  (2610.21789 - Kabir et al., 15 Oct 2026) in Section 6, “Disentangling Culture from Language” (final paragraph)