- The paper introduces a 10,000-plus-example benchmark across five Chinese-to-foreign-language directions, covering narrative posts and short social snippets with expert translations and safety filtering.
- The paper finds that conventional metrics can miss slang failures: GPT-5 leads slang accuracy with a 77.70% Slang Success Rate, while GPT-4o achieves the highest Fun Posts XCOMET score at 85.50.
- The paper shows that reminder prompts improve slang translation but have little effect on tone preservation, while embedding similarity and LLM judging provide complementary tools for evaluating informal social-media style.
Motivation and problem statement
Existing large-scale MT benchmarks such as WMT and FLORES are dominated by formal, edited text (news, Wikipedia), which differs substantially from the informal, creative, and context-dependent language of Chinese social media. The authors identify two obstacles to evaluating MT in this domain: data scarcity, since high-quality parallel data requires bilingual annotators fluent in platform-specific slang and stylistic conventions in both languages; and metric limitations, since surface-overlap metrics (BLEU, chrF) and even neural metrics like COMET/XCOMET are not designed for culturally grounded or stylistically variable content. A motivating example shows Qwen3-8B translating a formal Chinese sentence into Japanese correctly while failing to preserve slang and tone in a semantically equivalent social-media variant.
The benchmark: two complementary subsets
The benchmark is built from real user-generated content collected from a Chinese social platform, filtered through LLM-based screening plus internal safety APIs (removing personal information, violent, political, hateful, and pornographic content), then translated by bilingual expert annotators into Spanish, French, Japanese, Korean, and Russian, with peer review and adjudication. It comprises over 10,000 instances across five zh→X directions:
- Fun Posts (1,183 source sentences per language pair, average 41.1 characters): longer, narrative-style posts rich in neologisms and slang. 619 of the 1,183 samples contain detected slang or neologisms.
- Social Snippets (1,000 samples per pair, average 10.1 characters): short, emotional, reactive comments where tone and stylized expression carry meaning beyond literal content.
Targeted evaluation metrics
For Fun Posts, the paper proposes the Slang Success Rate (SSR). Slang–translation pairs are extracted from the human references, GPT-5 augments each with candidate target-language expressions (6.6 candidates per term on average), human annotators filter invalid candidates, and fuzzy matching (threshold 0.8) determines whether any acceptable translation appears in the model output. SSR is the fraction of slang items correctly rendered.
For Social Snippets, an Embedding Similarity (ES) score averages cosine similarities between source and translation under three embedding models: mStyleDistance style embeddings, XLM-EMO emotion embeddings, and multilingual sentiment embeddings. This is complemented by an adapted GEMBA-stars prompt that asks GPT-4o to judge whether social-media-specific tone and style are preserved. Notably, ES has limited absolute discriminability — even human translations score only 70.32 on average versus 68.27 for the best model (GPT-5) — but its rankings align closely with GEMBA-stars, making it a computationally cheap proxy for LLM-as-a-judge evaluation.
Main findings across 22 models
The evaluation covers closed-source APIs (GPT-4o, GPT-5, Claude-Sonnet-4), translation-specialized systems (NLLB-3.3B, Aya-101, Google Translate, GemmaX2-9B, Hunyuan-MT-7B), and open-source general-purpose LLMs (DeepSeek-V3, GPT-OSS-120B, Aya-Expanse-8B, eight Qwen3 variants, three Gemma3 variants). Representative average results:
| Model |
Fun Posts XCOMET |
Fun Posts SSR |
Social Snippets XCOMET |
Social Snippets ES |
GEMBA |
| GPT-4o |
85.50 |
64.91 |
76.50 |
68.20 |
3.65 |
| GPT-5 |
85.26 |
77.70 |
76.55 |
68.27 |
3.69 |
| Claude-Sonnet-4 |
85.19 |
71.31 |
75.14 |
68.10 |
3.58 |
| DeepSeek-V3 |
84.10 |
68.46 |
76.98 |
68.08 |
3.64 |
| Hunyuan-MT-7B |
83.71 |
51.60 |
75.93 |
66.85 |
3.28 |
| Google Translate |
81.93 |
53.57 |
73.81 |
66.01 |
3.17 |
| NLLB-3.3B |
65.49 |
23.52 |
58.37 |
61.74 |
2.13 |
Several results stand out:
- A divergence between overall quality and slang handling: GPT-4o achieves the highest XCOMET on Fun Posts, yet GPT-5 leads SSR by a wide margin (77.70 vs. 64.91). This indicates that standard adequacy-oriented metrics and slang fidelity can rank models differently, validating the need for targeted metrics. The authors attribute GPT-4o's XCOMET lead to its multilingual optimization, noting that GPT-5 was run in minimal thinking mode, which may understate its reasoning-driven translation behavior.
- Translation-specialized systems lag on informal language: all specialized models trail closed-source APIs on SSR by more than 11 points. Surprisingly, Google Translate attains the highest SSR among them (53.57), plausibly due to web-scale training data containing user-generated content, yet it underperforms on tone preservation in Social Snippets, suggesting it is optimized for longer standard-form text. Hunyuan-MT-7B is the strongest specialized system on XCOMET and GEMBA for Social Snippets.
- Scaling and post-training effects: the Qwen3 and Gemma3 series show clear scaling trends, and instruction-tuned variants consistently outperform their base counterparts, indicating that recent post-training improves translation robustness. DeepSeek-V3 leads open-source models on both subsets, rivaling GPT-4o's SSR.
- Social Snippets are harder: XCOMET scores drop noticeably relative to Fun Posts across all systems, confirming that short, emotion-laden, fragmented text remains difficult. On this subset GPT-5 overtakes GPT-4o on both XCOMET and ES, consistent with its tendency to adopt user-like tones including emojis.
Prompting interventions
Simple reminder prompts — alerting the model to possible slang/neologisms (Fun Posts) or asking it to preserve tone and style (Social Snippets) — yield asymmetric gains. For Fun Posts, reminders improve both XCOMET and SSR consistently (e.g., GPT-4o: +0.44 XCOMET, +2.17 SSR; Qwen3-8B: +0.70 XCOMET, +1.13 SSR). For Social Snippets, tone/style reminders produce negligible changes, indicating that subtle emotional and stylistic cues cannot be recovered by prompting alone. This contrast implies that slang errors are partly attentional (models fail to recognize non-standard tokens as requiring special treatment), whereas stylistic fidelity failures reflect deeper capability gaps.
Limitations and open questions
The authors acknowledge that their improvement experiments are restricted to simple general prompts; slang-aware pretraining and targeted fine-tuning remain unexplored here. Coverage is limited to five Chinese-foreign directions due to annotation cost, and the benchmark's slang dictionary depends on GPT-5 candidate generation followed by human filtering, so recall of acceptable translations is bounded by that pipeline. Additionally, ES exhibits a narrow dynamic range (roughly 62–68 across models against 70.32 for human translations), limiting its use as an absolute quality measure. Open questions include whether reasoning-enhanced decoding would change GPT-5's standing on semantic fidelity, and how SSR-style evaluation extends to languages beyond the five covered.
Conclusion
This work contributes a curated, expert-translated multilingual benchmark for Chinese social media MT, together with subset-specific metrics — SSR for slang and neologism fidelity, and embedding-based plus LLM-judged measures for tone and style preservation. Evaluation of 22 models shows that high scores on conventional metrics do not guarantee correct handling of informal expressions, that translation-specialized systems trail frontier LLMs on slang, and that simple prompting helps with lexical novelty but not with stylistic nuance. The benchmark provides a diagnostic instrument for measuring progress on these specific failure modes.
