Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus

Abstract: We investigate the extent to which cosine similarity between paragraph embeddings is invariant under machine translation, using the Manifesto Corpus of over 2,800 political party platforms in 28 languages translated to English via the EU eTranslation service. Rather than measuring translation-induced semantic shift directly we measure the stability of pairwise similarity relationships across embedding models, and use inter-model disagreement on original-language text as a calibrated invariance threshold. This yields a per-language non-inferiority test for four hypotheses about how translation interacts with embedding choice, with verdicts that distinguish languages where translation demonstrably preserves semantic structure from those where it demonstrably degrades it and from those where the available evidence does not resolve the question. The framework is corpus- and pipeline-agnostic and extends naturally to downstream tasks. Applied to our data, it identifies ten languages with translation invariance and four with detectable distortion.

• The paper quantifies translation-induced distortions by comparing original and machine-translated embeddings across a six-stage pipeline.
• It employs robust statistical tests and Pearson correlations to assess invariance across multilingual and monolingual encoding strategies.
• Results reveal that while multilingual models maintain robustness, translation-induced distortions vary by language and impact downstream tasks differently.

Invariance of Textual Similarity Under Machine Translation: An Analysis Using the Political Manifesto Corpus

Introduction

The central research question of "Is Textual Similarity Invariant under Machine Translation? Evidence Based on the Political Manifesto Corpus" (2605.00618) is whether similarity relationships among textual embeddings—critical for applications such as IR, semantic clustering, or ideological scaling—are preserved when non-English texts are machine-translated prior to vectorization. Unlike prior evaluations that focus on direct semantic preservation or linguistic attributes post-translation, this work quantifies stability in the relative geometry of embeddings across a multilingual corpus using the EU eTranslation system and high-variance NLP pipelines. The methodology is notable for a corpus- and pipeline-agnostic design, enabling robust, model-calibrated quantification of translation-induced distortions in embedding spaces.

Methodological Framework

The study uses a 2,878-document subset of the Manifesto Corpus, covering 28 EU languages, and introduces a six-stage pipeline:

1. Translation: Each non-English document is translated into English using eTranslation, producing parallel corpora.
2. Segmentation and Alignment: Documents are tokenized to sentences and pooled into semantically coherent "pseudo-paragraphs" using PELT segmentation, with monotone alignment ensuring consistent chunking between source and translation.
3. Feature Extraction: For each language, several top-performing SentenceBERT and contextual transformer models from HuggingFace are used to generate embeddings, including monolingual, multilingual, and English-only variants.
4. Pooling: Sentence embeddings are aggregated by centrality-weighted pooling (PageRank-weighted averages) to form robust paragraph representations, emphasizing semantically central content.
5. Similarity Matrix Construction: Cosine similarity matrices are computed for each language-model pair, capturing the corpus' intrinsic semantic geometry within each pipeline.
6. Comparative Evaluation: Correlations (Pearson) between flattened similarity matrices (upper triangle) are calculated across all model-pipeline pairs.

To address model-induced noise, they calibrate expected invariance using inter-model variance on original texts, adjusting per language and embedding family. Four explicit invariance hypotheses are formally tested using a one-sided non-inferiority framework, rather than a conventional significance test.

Main Hypotheses and Statistical Testing

The four hypotheses assessed are:

• Baseline invariance: Translation+monolingual encoding (OT) does not degrade similarity structure relative to model-model variance in original language (OO).
• Best-model invariance: The optimal post-translation English model matches or surpasses original-LLM variance.
• Multilingual invariance: Multilingual encoders’ similarity structure is preserved under translation (same model, original vs. translated).
• Performance equivalence: Translation+monolingual embedding is at least as effective as direct multilingual embedding (OT vs. OM).

An invariance margin is set per-language as $\delta_L = \kappa \cdot \sigma_L$, where $\sigma_L$ is derived from inter-model variability; verdicts are classified as invariant, distorted, or indeterminate.

Core Results

Corpus-Level and Language-Level Effects

Figure 1: Average correlations by language and by the type of pipeline used to generate the embeddings.

Figure 1 highlights pronounced heterogeneity across both languages and pipeline type. On average, O–T (original-translation) correlations are lower than O–O or O–M, with several exceptions (notably Swedish).

Rigorous statistical analysis partitions the languages into:

• Translation Invariant (All tests passed): German, Italian, Portuguese, Spanish, Ukrainian.
• Substantially Invariant (Three of four tests): Bulgarian, Greek, Hungarian, Icelandic, Swedish.
• Translation Distorted: French, Japanese, Latvian, Lithuanian; these demonstrate statistically significant semantic distortion even with optimal post-translation models.
• Indeterminate: The remainder, primarily reflecting low data or inter-model noise rather than detected translation degradation.

Particularly, no direct resource or typological correlates explain translation distortion: both high-resource (French, Japanese) and low-resource (Latvian, Lithuanian) languages fall into the distorted category.

Multilingual Encoders and Downstream Stability

Contrary to the translation+monolingual pipeline, multilingual encoders applied to both original and translated text are highly robust: in 24 of 28 languages, translations do not induce significant geometric distortions in similarity structure compared to baseline inter-model variability. Critically, no language exhibits statistically significant translation-induced degradation using this approach, emphasizing its resilience.

The study further evaluates propagation to downstream tasks—classification (ideological family), clustering, and dimensionality reduction (UMAP):

• Classification: No language exhibits translation-distortion at $\kappa=1$; minor numerical performance differences among pipelines are not statistically significant, and adjusted Rand indices remain high.
• Clustering: Only French and Japanese show persistent distortion in cluster structures; others are invariant or indeterminate.
• Dimensionality Reduction: Distortion concentrated in Latvian (also Czech for best-model), suggesting UMAP increases sensitivity to translation effects in intrinsically unstable languages.

Translation-induced perturbations are attenuated or disappear entirely for discrete or robust downstream tasks, with only high-precision low-dimensional representations (such as UMAP projections) sometimes exposing latent instability.

Key Numerical Results and Contradictory Claims

• Translation Invariance: Ten languages meet at least three invariance criteria at $\kappa=1$; for five, all four are satisfied, supporting use of translation+monolingual pipelines in these cases.
• Translation-Induced Distortion: Four languages (French, Japanese, Latvian, Lithuanian) show significant O–T geometric degradation (as high as mean $r \sim 0.4$ vs. baseline $r \sim 0.8$).
• Robustness of Multilingual Pipelines: Multilingual embedding approaches yield no significant translation-induced geometric distortion in any language tested.
• Downstream Task Sensitivity: Translation-induced geometric noise does not uniformly propagate to classification or clustering tasks, indicating that, in many practical contexts, machine translation is “good enough” if evaluated solely on discrete output accuracy.

These results contradict the common assumption that translation plus monolingual encoding is a universally reliable shortcut for cross-lingual similarity, especially in critical, fine-grained semantic tasks.

Theoretical and Practical Implications

The study’s findings have notable implications for both NLP research and practice:

• Empirical Calibration Is Essential: Any operational pipeline for cross-lingual analysis must empirically calibrate translation invariance for the affected language(s) and tasks.
• Preference for Multilingual Encoders: Where computational budget allows, direct use of multilingual encoders is the only empirically confirmed cross-lingual semantic normalization strategy.
• Task-Dependence: The risk of semantically significant translation distortion is highly task- and language-specific, with greatest impacts for applications leveraging geometric properties (e.g., representational similarity, ranking), and reduced for hard decision tasks (classification).
• Future Directions: The paper recommends expanding coverage to non-English pivots, other MT systems, and non-political corpora to more fully map translation invariance boundaries and typological drivers.

Conclusion

This paper delivers a rigorous, model-calibrated methodology for quantifying translation-induced semantic similarity drift at a level that supports both architectural decision-making and theoretical generalization. The methodology transcends corpus and model specifics, and its findings emphasize the necessity of per-language empirical validation before deploying translation-based cross-lingual semantic pipelines. Widespread robustness in multilingual encoding approaches, as quantified here, empowers practitioners working in computational humanities and IR with strong evidence for pipeline selection, while the uncovered translation distortions in specific languages constitute a salient warning for LLM-centric research pipelines that assume translation invariance.